Related papers: Multiscale analysis of accelerated gradient method…
In this paper, we describe a stochastic adaptive fast gradient descent method based on the mirror variant of similar triangles method. To our knowledge, this is the first attempt to use adaptivity in stochastic method. Additionally, a main…
Distributed machine learning has recently become a critical paradigm for training large models on vast datasets. We examine the stochastic optimization problem for deep learning within synchronous parallel computing environments under…
One of the most widely used methods for solving large-scale stochastic optimization problems is distributed asynchronous stochastic gradient descent (DASGD), a family of algorithms that result from parallelizing stochastic gradient descent…
Adaptive gradient methods have attracted much attention of machine learning communities due to the high efficiency. However their acceleration effect in practice, especially in neural network training, is hard to analyze, theoretically. The…
A simple model to handle the flow of people in emergency evacuation situations is considered: at every point x, the velocity U(x) that individuals at x would like to realize is given. Yet, the incompressibility constraint prevents this…
In a variety of problems originating in supervised, unsupervised, and reinforcement learning, the loss function is defined by an expectation over a collection of random variables, which might be part of a probabilistic model or the external…
In simulation-based inferences for partially observed Markov process models (POMP), the by-product of the Monte Carlo filtering is an approximation of the log likelihood function. Recently, iterated filtering [14, 13] has originally been…
Optimization techniques are at the core of many scientific and engineering disciplines. The steepest descent methods play a foundational role in this area. In this paper we studied a generalized steepest descent method on Riemannian…
We consider generalized gradient systems in Banach spaces whose evolutions are generated by the interplay between an energy functional and a dissipation potential. We focus on the case in which the dual dissipation potential is given by a…
Stochastic gradient descent is an optimisation method that combines classical gradient descent with random subsampling within the target functional. In this work, we introduce the stochastic gradient process as a continuous-time…
We study accelerated optimization methods in the Gaussian phase retrieval problem. In this setting, we prove that gradient methods with Polyak or Nesterov momentum have similar implicit regularization to gradient descent. This implicit…
In this work, we introduce an asynchronous decentralized accelerated stochastic gradient descent type of method for decentralized stochastic optimization, considering communication and synchronization are the major bottlenecks. We establish…
We further develop a new framework, called PDE Acceleration, by applying it to calculus of variations problems defined for general functions on $\mathbb{R}^n$, obtaining efficient numerical algorithms to solve the resulting class of…
We study the gradient flow of the potential energy on the infinite-dimensional Riemannian manifold of spatial curves parametrized by the arc length, which models overdamped motion of a falling inextensible string. We prove existence of…
Gradient descent and stochastic gradient descent are central to modern machine learning, yet their behavior under large step sizes remains theoretically unclear. Recent work suggests that acceleration often arises near the edge of…
This paper reviews the gradient sampling methodology for solving nonsmooth, nonconvex optimization problems. An intuitively straightforward gradient sampling algorithm is stated and its convergence properties are summarized. Throughout this…
We design and analyze a novel accelerated gradient-based algorithm for a class of bilevel optimization problems. These problems have various applications arising from machine learning and image processing, where optimal solutions of the two…
We study the gradient descent (GD) dynamics of a depth-2 linear neural network with a single input and output. We show that GD converges at an explicit linear rate to a global minimum of the training loss, even with a large stepsize --…
Composite convex optimization models arise in several applications, and are especially prevalent in inverse problems with a sparsity inducing norm and in general convex optimization with simple constraints. The most widely used algorithms…
This paper deals with subsampled spectral gradient methods for minimizing finite sum. Subsample function and gradient approximations are employed in order to reduce the overall computational cost of the classical spectral gradient methods.…