Related papers: Hessian Spectral Analysis at Foundation Model Scal…
Whilst there have been major advancements in the field of first order optimisation of deep learning models, where state of the art open source mixture of expert models go into the hundreds of billions of parameters, methods that rely on…
High-fidelity spectroscopy presents challenges for both observations and in designing instruments. High-resolution and high-accuracy spectra are required for verifying hydrodynamic stellar atmospheres and for resolving intergalactic…
Using the replica method we calculate the mean spectral density of the Hessian matrix at the global minimum of a random $N \gg 1$ dimensional isotropic, translationally invariant Gaussian random landscape confined by a parabolic potential…
Given an optimization problem, the Hessian matrix and its eigenspectrum can be used in many ways, ranging from designing more efficient second-order algorithms to performing model analysis and regression diagnostics. When nonlinear models…
A central feature of quantum metrology is the possibility of Heisenberg scaling, a quadratic improvement over the limits of classical statistics. This scaling, however, is notoriously fragile to noise. While for some noise types it can be…
The lack of theoretical results for Layer Normalization and feedforward Hessians has left a gap in the study of Transformer optimization landscapes. We address this by deriving explicit second-order expressions for these components, thereby…
L-BFGS is the state-of-the-art optimization method for many large scale inverse problems. It has a small memory footprint and achieves superlinear convergence. The method approximates Hessian based on an initial approximation and an update…
In conventional spectral/finite element methods, the triangulation/quadrilateralization of the domain produces many interior edges which require additional DOF. What if we could directly use the original hull without going to…
Hessians of neural network (NN) contain essential information about the curvature of NN loss landscapes which can be used to estimate NN generalization capabilities. We have previously proposed generalization criteria that rely on the…
Persistent homology is constrained to purely topological persistence while multiscale graphs account only for geometric information. This work introduces persistent spectral theory to create a unified low-dimensional multiscale paradigm for…
Clustering short text embeddings is a foundational task in natural language processing, yet remains challenging due to the need to specify the number of clusters in advance. We introduce a scalable spectral method that estimates the number…
Helium atom scattering (HAS) is a well established technique for examining the surface structure and dynamics of materials at atomic sized resolution. The HAS technique Helium spin-echo spectroscopy opens up the possibility of compressing…
We address the question whether the super-Heisenberg scaling for quantum estimation is realizable. We unify the results of two approaches. In the first one, the original system is compared with its copy rotated by the parameter dependent…
We show that all the bands of the Hofstadter model on the torus have an exactly flat dispersion and Berry curvature when a special system size is chosen. This result holds for any hopping and Chern number. Our analysis therefore provides a…
We combine the finite size scaling method with the meshfree spectral method to calculate quantum critical parameters for a given Hamiltonian. The basic idea is to expand the exact wave function in a finite exponential basis set and…
We develop an analysis pipeline for characterizing the topology of large scale structure and extracting cosmological constraints based on persistent homology. Persistent homology is a technique from topological data analysis that quantifies…
We analyze the scaling behavior of the fidelity, and the corresponding susceptibility, emerging in finite-size many-body systems whenever a given control parameter $\lambda$ is varied across a quantum phase transition. For this purpose we…
Simulating a Gaussian process requires sampling from a high-dimensional Gaussian distribution, which scales cubically with the number of sample locations. Spectral methods address this challenge by exploiting the Fourier representation,…
We investigate the universal property of curvatures in surface models which display a flat phase and a rough phase whose criticality is described by the Gaussian model. Earlier we derived a relation between the Hessian of the free energy…
We provide a detailed description of the Herschel-SPIRE Fourier Transform Spectrometer (FTS) Spectral Feature Finder (FF). The FF is an automated process designed to extract significant spectral features from SPIRE FTS data products.…