Related papers: SRKCD: a stabilized Runge-Kutta method for stochas…
This paper presents a novel stochastic gradient descent algorithm for constrained optimization. The proposed algorithm randomly samples constraints and components of the finite sum objective function and relies on a relaxed logarithmic…
We consider convex-concave saddle-point problems where the objective functions may be split in many components, and extend recent stochastic variance reduction methods (such as SVRG or SAGA) to provide the first large-scale linearly…
This paper discusses stochastic numerical methods of Runge-Kutta type with weak and strong convergences for systems of stochastic differential equations in It\^o form. At the beginning we give a brief overview of the stochastic numerical…
In this paper, we consider a general stochastic optimization problem which is often at the core of supervised learning, such as deep learning and linear classification. We consider a standard stochastic gradient descent (SGD) method with a…
The main theoretical obstacle to establish the original energy dissipation laws of Runge-Kutta methods for phase-field equations is to verify the maximum norm boundedness of the stage solutions without assuming global Lipschitz continuity…
Stochastic optimization lies at the heart of machine learning, and its cornerstone is stochastic gradient descent (SGD), a method introduced over 60 years ago. The last 8 years have seen an exciting new development: variance reduction (VR)…
We examine a multi-stage stochastic optimization problem characterized by stagewise-independent, decision-dependent noises with strict constraints. The problem assumes convexity in that, following a specific relaxation, it transforms into a…
We study the performance of stochastic first-order methods for finding saddle points of convex-concave functions. A notorious challenge faced by such methods is that the gradients can grow arbitrarily large during optimization, which may…
We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient…
Explicit stabilized methods are an efficient alternative to implicit schemes for the time integration of stiff systems of differential equations in large dimension. In this paper, we derive explicit stabilized integrators of orders one and…
This paper focuses on investigating an inexact stochastic model-based optimization algorithm that integrates preconditioning techniques for solving stochastic composite optimization problems. The proposed framework unifies and extends the…
Sampling from a log-concave distribution function is one core problem that has wide applications in Bayesian statistics and machine learning. While most gradient free methods have slow convergence rate, the Langevin Monte Carlo (LMC) that…
A convergence theorem for the continuous weak approximation of the solution of stochastic differential equations by general one step methods is proved, which is an extension of a theorem due to Milstein. As an application, uniform second…
We introduce a novel dynamic learning-rate scheduling scheme grounded in theory with the goal of simplifying the manual and time-consuming tuning of schedules in practice. Our approach is based on estimating the locally-optimal stepsize,…
Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness…
In the paper, we study a class of useful minimax problems on Riemanian manifolds and propose a class of effective Riemanian gradient-based methods to solve these minimax problems. Specifically, we propose an effective Riemannian gradient…
In this paper we consider time-dependent PDEs discretized by a special class of Physics Informed Neural Networks whose design is based on the framework of Runge--Kutta and related time-Galerkin discretizations. The primary motivation for…
The application of Runge-Kutta schemes designed to enjoy a large region of absolute stability can significantly increase the efficiency of numerical methods for PDEs based on a method of lines approach. In this work we investigate the…
We develop error-control based time integration algorithms for compressible fluid dynamics (CFD) applications and show that they are efficient and robust in both the accuracy-limited and stability-limited regime. Focusing on discontinuous…
In this article, a family of two- and three-stage explicit multiquadric (MQ) and inverse multiquadric (IMQ) radial basis functions (RBFs) Runge-Kutta methods are introduced for solving ordinary differential equations. These methods are…