Related papers: Gradient extremals, talwegs, valleys, and directio…

In this book chapter, we briefly describe the main components that constitute the gradient descent method and its accelerated and stochastic variants. We aim at explaining these components from a mathematical point of view, including…

Cohen et al. (2021) empirically study the evolution of the largest eigenvalue of the loss Hessian, also known as sharpness, along the gradient descent (GD) trajectory and observe the Edge of Stability (EoS) phenomenon. The sharpness…

The displacement and deviation vectors in spaces (manifolds), the tangent bundle of which is endowed with a transport along paths, are introduced. In case these spaces are equipped with a linear connection, the deviation equations (between…

Herein the topics of (natural) gradient descent, data decorrelation, and approximate methods for backpropagation are brought into a common discussion. Natural gradient descent illuminates how gradient vectors, pointing at directions of…

We study the convergence of the Riemannian steepest descent algorithm on the Grassmann manifold for minimizing the block version of the Rayleigh quotient of a symmetric matrix. Even though this problem is non-convex in the Euclidean sense…

In this paper, we characterize the noise of stochastic gradients and analyze the noise-induced dynamics during training deep neural networks by gradient-based optimizers. Specifically, we firstly show that the stochastic gradient noise…

Extremal graphical models encode the conditional independence structure of multivariate extremes and provide a powerful tool for quantifying the risk of rare events. Prior work on learning these graphs from data has focused on the setting…

The article considers an inhomogeneous Erd\H{o}s-R\"enyi random graph on $\{1,\ldots, N\}$, where an edge is placed between vertices $i$ and $j$ with probability $\varepsilon_N f(i/N,j/N)$, for $i\le j$, the choice being made independent…

We consider the (sub-Riemannian type) control problem of finding a path going from an initial point $x$ to a target point $y$, by only moving in certain admissible directions. We assume that the corresponding vector fields satisfy the…

Motivated by results from recent particle tracking experiments in turbulence (Xu et al., Nat. Phys. 7, 709 (2011)), we study the Lagrangian time correlations of vorticity alignments with the three eigenvectors of the deformation-rate…

Natural gradients have been widely used in optimization of loss functionals over probability space, with important examples such as Fisher-Rao gradient descent for Kullback-Leibler divergence, Wasserstein gradient descent for…

Gradient sensing underlies important biological processes including morphogenesis, polarization, and cell migration. The precision of gradient sensing increases with the length of a detector (a cell or group of cells) in the gradient…

Discretizing variational principles, as opposed to discretizing differential equations, leads to discrete-time analogues of mechanics, and, systematically, to geometric numerical integrators. The phase space of such variational…

The largest eigenvalue of the Hessian, or sharpness, of neural networks is a key quantity to understand their optimization dynamics. In this paper, we study the sharpness of deep linear networks for univariate regression. Minimizers can…

A generalized Wigner matrix perturbed by a finite-rank deterministic matrix is considered. The fluctuations of the largest eigenvalues, which emerge outside the bulk of the spectrum, and the corresponding eigenvectors, are studied. Under…

In dually flat manifolds, there is a deep connection between gradient flows and pregeodesics. This was one of the many important contributions of Amari to information geometry. In this paper, we extend the study of this relationship to…

We present a unified description of extremal metrics for the Laplace and Steklov eigenvalues on manifolds of arbitrary dimension using the notion of $n$-harmonic maps. Our approach extends the well-known results linking extremal metrics for…

The deviations of a graded algebra are a sequence of integers that determine the Poincare series of its residue field and arise as the number of generators of certain DG algebras. In a sense, deviations measure how far a ring is from being…

In this paper we give a description of the asymptotic behavior, as $\epsilon\to 0$, of the $\epsilon$-gradient flow in the finite dimensional case. Under very general assumptions we prove that it converges to an evolution obtained by…

Deep equilibrium models (DEQs) have recently emerged as a powerful paradigm for training infinitely deep weight-tied neural networks that achieve state of the art performance across many modern machine learning tasks. Despite their…