Related papers: SRKCD: a stabilized Runge-Kutta method for stochas…
We present a unified theorem for the convergence analysis of stochastic gradient algorithms for minimizing a smooth and convex loss plus a convex regularizer. We do this by extending the unified analysis of Gorbunov, Hanzely \& Richt\'arik…
We present two stochastic descent algorithms that apply to unconstrained optimization and are particularly efficient when the objective function is slow to evaluate and gradients are not easily obtained, as in some PDE-constrained…
A time discretization method is called strongly stable, if the norm of its numerical solution is nonincreasing. It is known that, even for linear semi-negative problems, many explicit Runge--Kutta (RK) methods fail to preserve this…
This paper considers stochastic optimization problems for a large class of objective functions, including convex and continuous submodular. Stochastic proximal gradient methods have been widely used to solve such problems; however, their…
Many machine learning problems can be formulated as minimax problems such as Generative Adversarial Networks (GANs), AUC maximization and robust estimation, to mention but a few. A substantial amount of studies are devoted to studying the…
We analyze convergence rates of stochastic optimization procedures for non-smooth convex optimization problems. By combining randomized smoothing techniques with accelerated gradient methods, we obtain convergence rates of stochastic…
In this paper, we revisit \textsf{ROOT-SGD}, an innovative method for stochastic optimization to bridge the gap between stochastic optimization and statistical efficiency. The proposed method enhances the performance and reliability of…
In this paper, we analyze any-order Runge-Kutta spectral volume schemes (RKSV(s,k)) for solving the one-dimensional scalar hyperbolic equation. The RKSV(s,k) was constructed by using the $s$-th explicit Runge-Kutta method in…
One of main obstacles in verifying the energy dissipation laws of implicit-explicit Runge-Kutta (IERK) methods for phase field equations is to establish the uniform boundedness of stage solutions without the global Lipschitz continuity…
In this paper, we introduce a new stochastic approximation (SA) type algorithm, namely the randomized stochastic gradient (RSG) method, for solving an important class of nonlinear (possibly nonconvex) stochastic programming (SP) problems.…
We present a family of algorithms, called descent algorithms, for optimizing convex and non-convex functions. We also introduce a new first-order algorithm, called rescaled gradient descent (RGD), and show that RGD achieves a faster…
Stochastic optimization methods have been hugely successful in making large-scale optimization problems feasible when computing the full gradient is computationally prohibitive. Using the theory of modified equations for numerical…
The design of numerical integrators for solving stochastic dynamics with high weak order relies on tedious calculations and is subject to a high number of order conditions. The original approaches from the literature consider strong…
Two of the most prominent algorithms for solving unconstrained smooth games are the classical stochastic gradient descent-ascent (SGDA) and the recently introduced stochastic consensus optimization (SCO) [Mescheder et al., 2017]. SGDA is…
Explicit Runge-Kutta methods are classical and widespread techniques in the numerical solution of ordinary differential equations (ODEs). Considering partial differential equations, spatial semidiscretisations can be used to obtain systems…
We consider stochastic convex optimization problems where the objective is an expectation over smooth functions. For this setting we suggest a novel gradient estimate that combines two recent mechanism that are related to notion of…
We study nonconvex finite-sum problems and analyze stochastic variance reduced gradient (SVRG) methods for them. SVRG and related methods have recently surged into prominence for convex optimization given their edge over stochastic gradient…
In this work, we consider constrained stochastic optimization problems under hidden convexity, i.e., those that admit a convex reformulation via non-linear (but invertible) map $c(\cdot)$. A number of non-convex problems ranging from…
In this paper, exponential Runge-Kutta methods of collocation type (ERKC) which were originally proposed in (Appl Numer Math 53:323-339, 2005) are extended to semilinear parabolic problems with time-dependent delay. Two classes of the ERKC…
Stochastic gradient descent with momentum (SGDM) methods have become fundamental optimization tools in machine learning, combining the computational efficiency of stochastic gradients with the acceleration benefits of momentum. Despite…