Related papers: On adaptive stochastic heavy ball momentum for sol…
In this paper, we consider constrained optimization problems with convex, smooth objective and constraints. We propose a new stochastic gradient algorithm, called the Stochastic Moving Ball Approximation (SMBA) method, to solve this class…
Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to…
We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together…
In this work, we apply the Stochastic Grid Bundling Method (SGBM) to numerically solve backward stochastic differential equations (BSDEs). The SGBM algorithm is based on conditional expectations approximation by means of bundling of Monte…
Many solid mechanics problems on complex geometries are conventionally solved using discrete boundary methods. However, such an approach can be cumbersome for problems involving evolving domain boundaries due to the need to track boundaries…
Several variants of stochastic gradient descent (SGD) have been proposed to improve the learning effectiveness and efficiency when training deep neural networks, among which some recent influential attempts would like to adaptively control…
Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the…
The recently proposed Sampling Kaczmarz Motzkin (SKM) algorithm performs well in comparison with the state-of-the-art methods in solving large-scale Linear Feasibility (LF) problems. To explore the concept of momentum in the context of…
We analyze a class of stochastic gradient algorithms with momentum on a high-dimensional random least squares problem. Our framework, inspired by random matrix theory, provides an exact (deterministic) characterization for the sequence of…
This paper proposes a semidefinite programming based method for estimating moments of a stochastic hybrid system (SHS). For polynomial SHSs -- which consist of polynomial continuous vector fields, reset maps, and transition intensities --…
This article deals with the adaptive and approximative computation of the Lam\'e equations. The equations of linear elasticity are considered as boundary integral equations and solved in the setting of the boundary element method (BEM).…
We propose a stochastic optimization method for minimizing loss functions, expressed as an expected value, that adaptively controls the batch size used in the computation of gradient approximations and the step size used to move along such…
We study the convergence behavior of the stochastic heavy-ball method with a small stepsize. Under a change of time scale, we approximate the discrete method by a stochastic differential equation that models small random perturbations of a…
We present a stochastic variance-reduced heavy ball power iteration algorithm for solving PCA and provide a convergence analysis for it. The algorithm is an extension of heavy ball power iteration, incorporating a step size so that progress…
This paper presents two stochastic model predictive control methods for linear time-invariant systems subject to unbounded additive uncertainties. The new methods are developed by formulating the chance constraints into deterministic form,…
A framework previously introduced in [3] for solving a sequence of stochastic optimization problems with bounded changes in the minimizers is extended and applied to machine learning problems such as regression and classification. The…
Recently, many variance reduced stochastic alternating direction method of multipliers (ADMM) methods (e.g.\ SAG-ADMM, SDCA-ADMM and SVRG-ADMM) have made exciting progress such as linear convergence rates for strongly convex problems.…
The Random Batch Method (RBM) is an effective technique to reduce the computational complexity when solving certain stochastic differential problems (SDEs) involving interacting particles. It can transform the computational complexity from…
Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD)…
In this work, a nonlinear momentum method is introduced to enhance the convergence performance of momentum-based gradient optimization algorithms. Classical momentum methods, such as the Heavy Ball method, can be viewed as a dynamical…