Related papers: Continuous-Time Analysis of Accelerated Gradient M…
In this paper, we establish the ordinary differential equation (ODE) that underlies the training dynamics of Model-Agnostic Meta-Learning (MAML). Our continuous-time limit view of the process eliminates the influence of the manually chosen…
We introduce a methodology for seeking conservation laws within a Hamiltonian dynamical system, which we term ``neural deflation''. Inspired by deflation methods for steady states of dynamical systems, we propose to {iteratively} train a…
In the first part of the paper we consider accelerated first order optimization method for convex functions with $L$-Lipschitz-continuous gradient, that is able to automatically adapts to problems which satisfies Polyak-{\L}ojasiewicz…
Momentum methods play a significant role in optimization. Examples include Nesterov's accelerated gradient method and the conditional gradient algorithm. Several momentum methods are provably optimal under standard oracle models, and all…
Stochastic gradient methods (SGMs) are predominant approaches for solving stochastic optimization. On smooth nonconvex problems, a few acceleration techniques have been applied to improve the convergence rate of SGMs. However, little…
This paper studies delayed stochastic algorithms for weakly convex optimization in a distributed network with workers connected to a master node. Recently, Xu et al. 2022 showed that an inertial stochastic subgradient method converges at a…
We present a theoretical analysis of some popular adaptive Stochastic Gradient Descent (SGD) methods in the small learning rate regime. Using the stochastic modified equations framework introduced by Li et al., we derive effective…
In this thesis we develop a novel framework to study smooth and strongly convex optimization algorithms, both deterministic and stochastic. Focusing on quadratic functions we are able to examine optimization algorithms as a recursive…
For gradient flows, the existing structure-preserving schemes are difficult to achieve arbitrary high-order accuracy in time while preserving maximum-principle (MBP) and energy dissipating simultaneously. In this paper, we develop a new…
We study single-loop gradient-flow dynamics for nested optimization, where the outer variable evolves while auxiliary variables track the inner solution map. While existing analyses typically rely on problem- and condition-specific Lyapunov…
We present maximally-fast numerical algorithms for conserved coarsening systems that are stable and accurate with a growing natural time-step $\Delta t=A t_s^{2/3}$. For non-conserved systems, only effectively finite timesteps are…
Nesterov's accelerated gradient method for minimizing a smooth strongly convex function $f$ is known to reduce $f(\x_k)-f(\x^*)$ by a factor of $\eps\in(0,1)$ after $k\ge O(\sqrt{L/\ell}\log(1/\eps))$ iterations, where $\ell,L$ are the two…
This article derives lower bounds on the convergence rate of continuous-time gradient-based optimization algorithms. The algorithms are subjected to a time-normalization constraint that avoids a reparametrization of time in order to make…
Large-scale constrained optimization problems are at the core of many tasks in control, signal processing, and machine learning. Notably, problems with functional constraints arise when, beyond a performance{\nobreakdash-}centric goal…
We extend slow manifolds near a transcritical singularity in a fast-slow system given by the explicit Euler discretization of the corresponding continuous-time normal form. The analysis uses the blow-up method and direct trajectory-based…
The derivation of second-order ordinary differential equations (ODEs) as continuous-time limits of optimization algorithms has been shown to be an effective tool for the analysis of these algorithms. Additionally, discretizing…
Stability and error estimate for the Oseen equations in a projection based variational setup has been derived in this paper. The use of Geometric Conservation Law (GCL) provides unconditional stability whereas without using GCL we have a…
We present a novel framework based on semi-bounded spatial operators for analyzing and discretizing initial boundary value problems on moving and deforming domains. This development extends an existing framework for well-posed problems and…
Understanding the dynamics of neural network parameters during training is one of the key challenges in building a theoretical foundation for deep learning. A central obstacle is that the motion of a network in high-dimensional parameter…
We propose computationally tractable accelerated first-order methods for Riemannian optimization, extending the Nesterov accelerated gradient (NAG) method. For both geodesically convex and geodesically strongly convex objective functions,…