Related papers: SRKCD: a stabilized Runge-Kutta method for stochas…
A very popular approach for solving stochastic optimization problems is the stochastic gradient descent method (SGD). Although the SGD iteration is computationally cheap and the practical performance of this method may be satisfactory under…
In this paper we propose stochastic gradient-free methods and accelerated methods with momentum for solving stochastic optimization problems. All these methods rely on stochastic directions rather than stochastic gradients. We analyze the…
The class of stochastic Runge-Kutta methods for stochastic differential equations due to R\"o{\ss}ler is considered. Coefficient families of diagonally drift-implicit stochastic Runge-Kutta (DDISRK) methods of weak order one and two are…
Relative smoothness - a notion introduced by Birnbaum et al. (2011) and rediscovered by Bauschke et al. (2016) and Lu et al. (2016) - generalizes the standard notion of smoothness typically used in the analysis of gradient type methods. In…
Many control, optimization, and learning algorithms rely on discretizations of continuous-time contracting systems, where preservation of contractivity under numerical integration is key for stability, robustness, and reliable fixed-point…
A large class of machine learning techniques requires the solution of optimization problems involving spectral functions of parametric matrices, e.g. log-determinant and nuclear norm. Unfortunately, computing the gradient of a spectral…
We study the A-stability and accuracy characteristics of Clenshaw-Curtis collocation. We present closed-form expressions to evaluate the Runge-Kutta coefficients of these methods. From the A-stability study, Clenshaw-Curtis methods are…
Recently there is a large amount of work devoted to the study of Markov chain stochastic gradient methods (MC-SGMs) which mainly focus on their convergence analysis for solving minimization problems. In this paper, we provide a…
We construct a family of embedded pairs for optimal strong stability preserving explicit Runge-Kutta methods of order $2 \leq p \leq 4$ to be used to obtain numerical solution of spatially discretized hyperbolic PDEs. In this construction,…
In this paper, we study the performance of a large family of SGD variants in the smooth nonconvex regime. To this end, we propose a generic and flexible assumption capable of accurate modeling of the second moment of the stochastic…
In the present paper, a class of stochastic Runge-Kutta methods containing the second order stochastic Runge-Kutta scheme due to E. Platen for the weak approximation of It\^o stochastic differential equation systems with a multi-dimensional…
Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a…
Variance reduction (VR) methods employ stochastic gradients with decreasing variance, and they have been widely applied to solve large-scale optimization problems in machine learning because of their efficiency. Existing theoretical studies…
We apply the concept of effective order to strong stability preserving (SSP) explicit Runge-Kutta methods. Relative to classical Runge-Kutta methods, methods with an effective order of accuracy are designed to satisfy a relaxed set of order…
This study focuses on the development and analysis of a group of high-order implicit-explicit (IMEX) Runge--Kutta (RK) methods that are suitable for discretizing gradient flows with nonlinearity that is Lipschitz continuous. We demonstrate…
Many time-dependent differential equations are equipped with invariants. Preserving such invariants under discretization can be important, e.g., to improve the qualitative and quantitative properties of numerical solutions. Recently,…
The simulation of chemical kinetics involving multiple scales constitutes a modeling challenge (from ordinary differential equations to Markov chain) and a computational challenge (multiple scales, large dynamical systems, time step…
We develop a framework for convexifying a fairly general class of optimization problems. Under additional assumptions, we analyze the suboptimality of the solution to the convexified problem relative to the original nonconvex problem and…
We develop a class of algorithms, as variants of the stochastically controlled stochastic gradient (SCSG) methods (Lei and Jordan, 2016), for the smooth non-convex finite-sum optimization problem. Assuming the smoothness of each component,…
Many machine learning tasks can be formulated as a stochastic compositional optimization (SCO) problem such as reinforcement learning, AUC maximization, and meta-learning, where the objective function involves a nested composition…