Related papers: Variable stepsize SDIMSIMs for ordinary differenti…
We show that existing Runge-Kutta methods for ordinary differential equations (odes) can be modified to solve stochastic differential equations (sdes) with strong solutions provided that appropriate changes are made to the way stepsizes are…
We present a theoretical analysis of stochastic optimization methods in terms of their sensitivity with respect to the step size. We identify a key quantity that, for each method, describes how the performance degrades as the step size…
To overcome the tradeoff of the conventional normalized least mean square (NLMS) algorithm between fast convergence rate and low steady-state misalignment, this paper proposes a variable step size (VSS) NLMS algorithm by devising a new…
Stochastic gradient descent (SGD) is a promising method for solving large-scale inverse problems, due to its excellent scalability with respect to data size. In this work, we analyze a new data-driven regularized stochastic gradient descent…
SGD with momentum (SGDM) has been widely applied in many machine learning tasks, and it is often applied with dynamic stepsizes and momentum weights tuned in a stagewise manner. Despite of its empirical advantage over SGD, the role of…
This paper focuses on systems of nonlinear second-order stochastic differential equations with multi-scales. The motivation for our study stems from mathematical physics and statistical mechanics, for examples, Langevin dynamics and…
We study the use of inverse harmonic Rayleigh quotients with target for the stepsize selection in gradient methods for nonlinear unconstrained optimization problems. This provides not only an elegant and flexible framework to parametrize…
We introduce an efficient and accurate staggered-grid finite-difference (SGFD) method to solve the two-dimensional elastic wave equation. We use a coupled first-order stress-velocity formulation. In the standard implementation of SGFD…
In this work, we propose new adaptive step size strategies that improve several stochastic gradient methods. Our first method (StoPS) is based on the classical Polyak step size (Polyak, 1987) and is an extension of the recent development of…
Many relevant problems in the area of systems and control, such as controller synthesis, observer design and model reduction, can be viewed as optimization problems involving dynamical systems: for instance, maximizing performance in the…
This paper introduces a family of stochastic extragradient-type algorithms for a class of nonconvex-nonconcave problems characterized by the weak Minty variational inequality (MVI). Unlike existing results on extragradient methods in the…
We generalize the theory of underlying one-step methods to strictly stable general linear methods (GLMs) solving nonautonomous ordinary differential equations (ODEs) that satisfy a global Lipschitz condition. We combine this theory with the…
We study the stochastic optimization problem from a continuous-time perspective, with a focus on the Stochastic Gradient Descent with Momentum (SGDM) method. We show that the trajectory of SGDM, despite its \emph{stochastic} nature,…
Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several…
In this paper, we deal with multiobjective composite optimization problems, where each objective function is a combination of smooth and possibly non-smooth functions. We first propose a parameter-dependent conditional gradient method to…
In this paper, we present a novel explicit second order scheme with one step for solving the forward backward stochastic differential equations, with the Crank-Nicolson method as a specific instance within our proposed framework. We first…
We study the scaling limits of stochastic gradient descent (SGD) with constant step-size in the high-dimensional regime. We prove limit theorems for the trajectories of summary statistics (i.e., finite-dimensional functions) of SGD as the…
Large-scale nonconvex optimization problems are ubiquitous in modern machine learning, and among practitioners interested in solving them, Stochastic Gradient Descent (SGD) reigns supreme. We revisit the analysis of SGD in the nonconvex…
In this paper, we propose and analyse a family of generalised stochastic composite mirror descent algorithms. With adaptive step sizes, the proposed algorithms converge without requiring prior knowledge of the problem. Combined with an…
A variable stepsize exponential multistep integrator, with contour integral approximation of the operator-valued exponential functions, is proposed for solving semilinear parabolic equations with nonsmooth initial data. By this approach,…