Related papers: SRKCD: a stabilized Runge-Kutta method for stochas…
We present a novel class of projected gradient (PG) methods for minimizing a smooth but not necessarily convex function over a convex compact set. We first provide a novel analysis of the constant-stepsize PG method, achieving the…
We develop continuous-stage Runge-Kutta-Nystr\"{o}m (csRKN) methods for solving second order ordinary differential equations (ODEs) in this paper. The second order ODEs are commonly encountered in various fields and some of them can be…
A mixed accuracy framework for Runge--Kutta methods presented in [Grant, JSC 2022] has been shown to speed up the computation in diagonally implicit Runge--Kutta (DIRK) methods by using less expensive low accuracy approaches for the…
Stochastic gradient descent (SGD) method is popular for solving non-convex optimization problems in machine learning. This work investigates SGD from a viewpoint of graduated optimization, which is a widely applied approach for non-convex…
In this contribution, we present a numerical analysis of the continuous stochastic gradient (CSG) method, including applications from topology optimization and convergence rates. In contrast to standard stochastic gradient optimization…
We introduce a clipping strategy for Stochastic Gradient Descent (SGD) which uses quantiles of the gradient norm as clipping thresholds. We prove that this new strategy provides a robust and efficient optimization algorithm for smooth…
Explicit Runge-Kutta (RK) integration of hyperbolic initial-boundary value problems with time-dependent Dirichlet data often displays order reduction: the observed convergence order falls below the nominal order because the stage structure…
Strong Stability Preserving (SSP) time integration schemes maintain stability of the forward Euler method for any initial value problem. However, only a small subset of Runge-Kutta (RK) methods are SSP, and many efficient high-order time…
Stochastic gradient descent (SGD) holds as a classical method to build large scale machine learning models over big data. A stochastic gradient is typically calculated from a limited number of samples (known as mini-batch), so it…
Our work focuses on stochastic gradient methods for optimizing a smooth non-convex loss function with a non-smooth non-convex regularizer. Research on this class of problem is quite limited, and until recently no non-asymptotic convergence…
Stochastic gradient methods are dominant in nonconvex optimization especially for deep models but have low asymptotical convergence due to the fixed smoothness. To address this problem, we propose a simple yet effective method for improving…
Segregated Runge-Kutta (SRK) schemes are time integration methods for the incompressible Navier-Stokes equations. In this approach, convection and diffusion can be independently treated either explicitly or implicitly, which in particular…
Structured statistical estimation problems are often solved by Conditional Gradient (CG) type methods to avoid the computationally expensive projection operation. However, the existing CG type methods are not robust to data corruption. To…
Accelerated gradient methods are the cornerstones of large-scale, data-driven optimization problems that arise naturally in machine learning and other fields concerning data analysis. We introduce a gradient-based optimization framework for…
It is widely accepted that the stepsize is of great significance to gradient method. Two efficient gradient methods with approximately optimal stepsizes mainly based on regularization models are proposed for unconstrained optimization. More…
We show that existing Runge-Kutta methods for ordinary differential equations (odes) can be modified to solve stochastic differential equations (sdes) with strong solutions provided that appropriate changes are made to the way stepsizes are…
In this paper, we construct stochastic symplectic Runge--Kutta (SSRK) methods of high strong order for Hamiltonian systems with additive noise. By means of colored rooted tree theory, we combine conditions of mean-square order 1.5 and…
This paper studies a stochastic algorithm for linearly constrained nonconvex optimization, where the objective function is smooth but only unbiased stochastic gradients with bounded variance are available. We propose a momentum-based…
Within the context of hybrid quantum-classical optimization, gradient descent based optimizers typically require the evaluation of expectation values with respect to the outcome of parameterized quantum circuits. In this work, we explore…
The state of the art in solving nonconvex nonsmooth games under uncertainty remains in its infancy. Existing studies primarily rely on stringent growth conditions or local convexity-like properties, making the development of alternative…