Related papers: On the Instability of Nesterov's ODE under Non-Con…
We study the connections between ordinary differential equations and optimization algorithms in a non-Euclidean setting. We propose a novel accelerated algorithm for minimising convex functions over a convex constrained set. This algorithm…
In this paper we propose new numerical algorithms in the setting of unconstrained optimization problems and we study the rate of convergence in the iterates of the objective function. Furthermore, our algorithms are based upon splitting and…
We present a coupled system of ODEs which, when discretized with a constant time step/learning rate, recovers Nesterov's accelerated gradient descent algorithm. The same ODEs, when discretized with a decreasing learning rate, leads to novel…
The continuous-time model of Nesterov's momentum provides a thought-provoking perspective for understanding the nature of the acceleration phenomenon in convex optimization. One of the main ideas in this line of research comes from the…
In a Hilbert setting, we develop fast methods for convex unconstrained optimization. We rely on the asymptotic behavior of an inertial system combining geometric damping with temporal scaling. The convex function to minimize enters the…
Recent work has established that the trajectory of the Nesterov ODE, a the continuous-time model of Nesterov's accelerated gradient method, exhibits point convergence towards a minimizer of a convex potential. A natural next question is…
We study necessary and sufficient conditions for contraction and incremental stability of dynamical systems with respect to non-Euclidean norms. First, we introduce weak pairings as a framework to study contractivity with respect to…
Motivated by stochastic models of climate phenomena, the steady-state of a linear stochastic model with additive Gaussian white noise is studied. Fluctuation theorems for nonequilibrium steady-states provide a constraint on the character of…
Incompressible fluid equations are studied with UV cut-off and in periodic boundary conditions. Properties of the resulting ODEs holding uniformly in the cut-off are considered and, in particular, are conjectured to be equivalent to…
We give a development of the ODE method for the analysis of recursive algorithms described by a stochastic recursion. With variability modelled via an underlying Markov process, and under general assumptions, the following results are…
In nonsmooth optimization, a negative subgradient is not necessarily a descent direction, making the design of convergent descent methods based on zeroth-order and first-order information a challenging task. The well-studied bundle methods…
This paper considers a class of reinforcement-based learning (namely, perturbed learning automata) and provides a stochastic-stability analysis in repeatedly-played, positive-utility, finite strategic-form games. Prior work in this class of…
The overall performance or expected excess risk of an iterative machine learning algorithm can be decomposed into training error and generalization error. While the former is controlled by its convergence analysis, the latter can be tightly…
We consider a dissipative vector field which is represented by a nearly-integrable Hamiltonian flow to which a non symplectic force is added, so that the phase space volume is not preserved. The vector field depends upon two parameters,…
We show that the complex-valued ODE \begin{equation*} \dot z_t = a_{n+1} z^{n+1} + a_n z^n+\cdots+a_0, \end{equation*} which necessarily has trajectories along which the dynamics blows up in finite time, can be stabilized by the addition of…
The present article considers stability of the solutions to nonlinear and nonautonomous compartmental systems governed by ordinary differential equations (ODEs). In particular, compartmental systems with a right-hand side that can be…
In this paper, we study the behavior of solutions of the ODE associated to Nesterov acceleration. It is well-known since the pioneering work of Nesterov that the rate of convergence $O(1/t^2)$ is optimal for the class of convex functions…
Adam is a popular variant of stochastic gradient descent for finding a local minimizer of a function. In the constant stepsize regime, assuming that the objective function is differentiable and non-convex, we establish the convergence in…
Adaptive optimizers can reduce to normalized steepest descent (NSD) when only adapting to the current gradient, suggesting a close connection between the two algorithmic families. A key distinction between their analyses, however, lies in…
A novel method for stability and instability study of autonomous dynamical systems using the flow and divergence of the vector field is proposed. A relation between the method of Lyapunov functions and the proposed method is established.…