Related papers: Understanding high-index saddle dynamics via numer…
We showcase important features of the dynamics of the Stochastic Gradient Descent (SGD) in the training of neural networks. We present empirical observations that commonly used large step sizes (i) lead the iterates to jump from one side of…
Setting regularization parameters for Lasso-type estimators is notoriously difficult, though crucial in practice. The most popular hyperparameter optimization approach is grid-search using held-out validation data. Grid-search however…
Stochastic gradient descent (SGD) is a ubiquitous algorithm for a variety of machine learning problems. Researchers and industry have developed several techniques to optimize SGD's runtime performance, including asynchronous execution and…
Automatic segmentation of an image to identify all meaningful parts is one of the most challenging as well as useful tasks in a number of application areas. This is widely studied. Selective segmentation, less studied, aims to use limited…
In this article, we investigate the artificial viscosity and hourglass control algorithms for high-order staggered Lagrangian hydrodynamics(SGH), as proposed in~\cite[Sun et al., 2025]{Sun2025High}. Inspired by the subzonal pressure method…
We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance.…
Along with the practical success of the discovery of dynamics using deep learning, the theoretical analysis of this approach has attracted increasing attention. Prior works have established the grid error estimation with auxiliary…
In this paper, we employ the concept of quasi-relative interior to analyze the method of Lagrange multipliers and establish strong Lagrangian duality for nonsmooth convex optimization problems in Hilbert spaces. Then, we generalize the…
In this paper, we propose a new primal-dual algorithmic framework for a class of convex-concave saddle point problems frequently arising from image processing and machine learning. Our algorithmic framework updates the primal variable…
In this paper, we unroll the dynamics of the dual ascent (DA) algorithm in two coupled graph neural networks (GNNs) to solve constrained optimization problems. The two networks interact with each other at the layer level to find a saddle…
In recent years, stochastic variance reduction algorithms have attracted considerable attention for minimizing the average of a large but finite number of loss functions. This paper proposes a novel Riemannian extension of the Euclidean…
We study a fixed step-size noisy distributed gradient descent algorithm for solving optimization problems in which the objective is a finite sum of smooth but possibly non-convex functions. Random perturbations are introduced to the…
The inverse problem of the calculus of variations consists in determining if the solutions of a given system of second order differential equations correspond with the solutions of the Euler-Lagrange equations for some regular Lagrangian.…
Gradient dominance property is a condition weaker than strong convexity, yet sufficiently ensures global convergence even in non-convex optimization. This property finds wide applications in machine learning, reinforcement learning (RL),…
The topic of this article is the numerical search of codimension 2 Normally Hyperbolic Invariant Manifolds (NHIM) in Hamiltonian systems with 3 degrees of freedom and their internal dynamics. We point out relations between different…
Representing images and videos with Symmetric Positive Definite (SPD) matrices and considering the Riemannian geometry of the resulting space has proven beneficial for many recognition tasks. Unfortunately, computation on the Riemannian…
A computational procedure that allows the detection of a new type of high-dimensional chaotic saddle in Hamiltonian systems with three degrees of freedom is presented. The chaotic saddle is associated with a so-called normally hyperbolic…
This work presents a numerical analysis of computing transition states of semilinear elliptic partial differential equations (PDEs) via the index-1 saddle dynamics, or equivalently, the gentlest ascent dynamics. To establish clear…
We develop a new method for solving minimization problems on the Stiefel Manifold using damped dynamical systems. The constraints are satisfied in the limit by an additional damped dynamical system. The method is illustrated by numerical…
Stochastic gradient method (SGM) has been popularly applied to solve optimization problems with objective that is stochastic or an average of many functions. Most existing works on SGMs assume that the underlying problem is unconstrained or…