Related papers: Gradient extremals, talwegs, valleys, and directio…
Traditional analyses of gradient descent optimization show that, when the largest eigenvalue of the loss Hessian - often referred to as the sharpness - is below a critical learning-rate threshold, then training is 'stable' and training loss…
Differential geometry may be generalized to allow infinitesimals to any order. The purpose of the present contribution is to show that the theory so developed expands received geometrical ideas in an interesting way, rich in potential for…
In this work, we consider smooth unconstrained optimization problems and we deal with the class of gradient methods with momentum, i.e., descent algorithms where the search direction is defined as a linear combination of the current…
We introduce the notion of extremal basis of tangent vector fields at a boundary point of finite type of a pseudo-convex domain in $\mathbb{C}^n$. Then we define the class of geometrically separated domains at a boundary point, and give a…
We study first order methods to compute the barycenter of a probability distribution $P$ over the space of probability measures with finite second moment. We develop a framework to derive global rates of convergence for both gradient…
This paper establishes risk convergence and asymptotic weight matrix alignment --- a form of implicit regularization --- of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail,…
Gradient schemes is a framework which enables the unified convergence analysis of many different methods -- such as finite elements (conforming, non-conforming and mixed) and finite volumes methods -- for $2^{\rm nd}$ order diffusion…
In this paper we prove, using a refinement of Terracini's Lemma, a sharp lower bound for the degree of (higher) secant varieties to a given projective variety, which extends the well known lower bound for the degree of a variety in terms of…
We show that the spatially homogeneous Boltzmann equation evolves as the gradient flow of the entropy with respect to a suitable geometry on the space of probability measures which takes the collision process into account. This gradient…
The {\L}ojasiewicz inequality characterizes objective-value convergence along gradient flows and, in special cases, yields exponential decay of the cost. However, such results do not directly give rates of convergence in the state. In this…
We derive the system of differential equations for the gradient flow characterizing the training process of linear in-context learning in full generality. Next, we explore the geometric structure of the gradient flows in two instances,…
An appealing property of the natural gradient is that it is invariant to arbitrary differentiable reparameterizations of the model. However, this invariance property requires infinitesimal steps and is lost in practical implementations with…
Various spatial-gradient extensions of standard viscoelastic rheologies of the Kelvin-Voigt, Maxwell's, and Jeffreys' types are analyzed in linear one-dimensional situations as far as the propagation of waves and their dispersion and…
Gradient descent (GD) is a collection of continuous optimization methods that have achieved immeasurable success in practice. Owing to data science applications, GD with diminishing step sizes has become a prominent variant. While this…
Many problems in high-dimensional statistics and optimization involve minimization over nonconvex constraints-for instance, a rank constraint for a matrix estimation problem-but little is known about the theoretical properties of such…
We study the geometry of flag manifolds under different embeddings into a product of Grassmannians. We show that differential geometric objects and operations -- tangent vector, metric, normal vector, exponential map, geodesic, parallel…
We prove full convergence of gradient-flows of the arc-length restricted tangent point energies in the Hilbert-case towards critical points. This is done through a {\L}ojasiewicz-Simon gradient inequality for these energies. In order to do…
Eigenvector continuation is a computational method that finds the extremal eigenvalues and eigenvectors of a Hamiltonian matrix with one or more control parameters. It does this by projection onto a subspace of eigenvectors corresponding to…
Gradient vector fields are fundamental objects from both theoretical and practical perspectives, since various phenomena can be modeled within this framework. The ``moduli space'' of such vector fields provides the foundation for describing…
We propose to use the {\L}ojasiewicz inequality as a general tool for analyzing the convergence rate of gradient descent on a Hilbert manifold, without resorting to the continuous gradient flow. Using this tool, we show that a Sobolev…