Related papers: Gradient extremals, talwegs, valleys, and directio…

This paper develops a general approach to characterize the long-time trajectory behavior of nonconvex gradient descent in generalized single-index models in the large aspect ratio regime. In this regime, we show that for each iteration the…

The local geometry of high dimensional neural network loss landscapes can both challenge our cherished theoretical intuitions as well as dramatically impact the practical success of neural network training. Indeed recent works have observed…

We study the properties of alignment, a form of implicit regularization, in linear neural networks under gradient descent. We define alignment for fully connected networks with multidimensional outputs and show that it is a natural…

We present a manifestly covariant formulation of the gradient descent method, ensuring consistency across arbitrary coordinate systems and general curved trainable spaces. The optimization dynamics is defined using a covariant force vector…

Turbulent fluid flows exhibit a complex small-scale structure with frequently occurring extreme velocity gradients. Particles probing such swirling and straining regions respond with an intricate shape-dependent orientational dynamics,…

We explicitly construct parameter transformations between gradient flows in metric spaces, called curves of maximal slope, having different exponents when the associated function satisfies a suitable convexity condition. These…

A recent line of work has shown remarkable behaviors of the generalization error curves in simple learning models. Even the least-squares regression has shown atypical features such as the model-wise double descent, and further works have…

A Finsler geometry may be understood as a homogeneous variational problem, where the Finsler function is the Lagrangian. The extremals in Finsler geometry are curves, but in more general variational problems we might consider extremal…

A geometric setup for constrained variational calculus is presented. The analysis deals with the study of the extremals of an action functional defined on piecewise differentiable curves, subject to differentiable, non-holonomic…

The paper surveys recent progresses in understanding the dynamics and loss landscape of the gradient flow equations associated to deep linear neural networks, i.e., the gradient descent training dynamics (in the limit when the step size…

In optimization, the negative gradient of a function denotes the direction of steepest descent. Furthermore, traveling in any direction orthogonal to the gradient maintains the value of the function. In this work, we show that these…

Gradient, chemically modified, flat surfaces enable directed transport of droplets. Calculation of apparent contact angles inherent for gradient surfaces is challenging even for atomically flat ones. Wetting of gradient, flat solid surfaces…

We analyse the eigenvectors of the adjacency matrix of a random inhomogeneous graph constructed from a specified degree sequence. We assume that the empirical degree sequence has bounded mean and variance. We show that near the edges of the…

This paper establishes a theory of nonlinear spectral decompositions by considering the eigenvalue problem related to an absolutely one-homogeneous functional in an infinite-dimensional Hilbert space. This approach is both motivated by…

This paper considers the analysis of continuous time gradient-based optimization algorithms through the lens of nonlinear contraction theory. It demonstrates that in the case of a time-invariant objective, most elementary results on…

The rapid progress in machine learning in recent years has been based on a highly productive connection to gradient-based optimization. Further progress hinges in part on a shift in focus from pattern recognition to decision-making and…

In this paper, we show that although the minimizers of cross-entropy and related classification losses are off at infinity, network weights learned by gradient flow converge in direction, with an immediate corollary that network…

We study the Wasserstein gradient flow of the Sinkhorn divergence when both the source and the target are Gaussian distributions. We prove the existence of a flow that stays in the class of Gaussian distributions, and is unique in the…

We consider extremal eigenvalues of sparse random matrices, a class of random matrices including the adjacency matrices of Erd\H{o}s-R\'{e}nyi graphs $\mathcal{G}(N,p)$. Recently, it was shown that the leading order fluctuations of extremal…

Sampling a probability distribution with an unknown normalization constant is a fundamental problem in computational science and engineering. This task may be cast as an optimization problem over all probability measures, and an initial…