Related papers: From Adam to Adam-Like Lagrangians: Second-Order N…

Motivated by robotic trajectory optimization problems we consider the Augmented Lagrangian approach to constrained optimization. We first propose an alternative augmentation of the Lagrangian to handle the inequality case (not based on…

A conservative invariant domain preserving Arbitrary Lagrangian Eulerian method for solving nonlinear hyperbolic systems is introduced. The method is explicit in time, works with continuous finite elements and is first-order accurate in…

We introduce AlphaGrad, a memory-efficient, conditionally stateless optimizer addressing the memory overhead and hyperparameter complexity of adaptive methods like Adam. AlphaGrad enforces scale invariance via tensor-wise L2 gradient…

This paper proposes a novel nonlinear sliding mode state feedback controller for perturbed second-order systems. In analogy to a linear proportional-derivative (PD) feedback control, the proposed nonlinear scheme uses the output of interest…

Adaptive optimization algorithms, particularly Adam and its variant AdamW, are fundamental components of modern deep learning. However, their training dynamics lack comprehensive theoretical understanding, with limited insight into why…

Time and again, non-conventional forms of Lagrangians with non-quadratic velocity dependence have found attention in the literature. For one thing, such Lagrangians have deep connections with several aspects of nonlinear dynamics including…

The Adaptive Momentum Estimation (Adam) algorithm is highly effective in training various deep learning tasks. Despite this, there's limited theoretical understanding for Adam, especially when focusing on its vanilla form in non-convex…

This paper provides the first tight convergence analyses for RMSProp and Adam in non-convex optimization under the most relaxed assumptions of coordinate-wise generalized smoothness and affine noise variance. We first analyze RMSProp, which…

We propose a continuous-time scheme for large-scale optimization that introduces individual, adaptive momentum coefficients regulated by the kinetic energy of each model parameter. This approach automatically adjusts to local landscape…

Historically speaking, it is hard to balance the global and local efficiency of second-order optimization algorithms. For instance, the classical Newton's method possesses excellent local convergence but lacks global guarantees, often…

We showcase the utility of the Lagrangian descriptors method in qualitatively understanding the underlying dynamical behavior of dynamical systems governed by fractional-order differential equations. In particular, we use the Lagrangian…

Any given system of ordinary differential equations in $n$-dimensional configuration space can be obtained from a peculiar variational problem with one local symmetry. The obtained action functional leads to the Hamiltonian formulation in…

We study the mathematical theory of second order systems with two species, arising in the dynamics of interacting particles subject to linear damping, to nonlocal forces and to external ones, and resulting into a nonlocal version of the…

A broad class of optimization problems can be cast in composite form, that is, considering the minimization of the composition of a lower semicontinuous function with a differentiable mapping. This paper investigates the versatile template…

This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates. We demonstrate that Adam achieves a faster convergence compared to SGDM under the condition of…

Lagrangian multiform theory is a variational framework for integrable systems. In this article we introduce a new formulation which is based on symplectic geometry and which treats position, momentum and time coordinates of a…

This paper proposes a theory that bridges classical analytical mechanics and nonequilibrium thermodynamics. Its intent is to derive the evolution equations of a system from a stationarity principle for a suitably augmented Lagrangian…

Textbook treatments of classical mechanics typically assume that the Lagrangian is nonsingular. That is, the matrix of second derivatives of the Lagrangian with respect to the velocities is invertible. This assumption insures that (i)…

We show that the equations which describe irreversible evolution of a system can be derived from a variational principle. We suggest a Lagrangian, which depends on the properties of the normal and the so-called "mirror-image" system. The…

In this paper we show how the well-know local symmetries of Lagrangeans systems, and in particular the diffeomorphism invariance, emerge in the Hamiltonian formulation. We show that only the constraints which are linear in the momenta…