Related papers: An efficient and robust SAV based algorithm for di…
Sharpness-aware Minimization (SAM) has been proposed recently to improve model generalization ability. However, SAM calculates the gradient twice in each optimization step, thereby doubling the computation costs compared to stochastic…
The kinetic Langevin dynamics finds diverse applications in various disciplines such as molecular dynamics and Hamiltonian Monte Carlo sampling. In this paper, a novel splitting scalar auxiliary variable (SSAV) scheme is proposed for the…
We propose an accelerated forward-backward method with fast convergence rate for finding a minimizer of a decomposable nonsmooth convex function over a closed convex set, and name it smoothing accelerated proximal gradient (SAPG) algorithm.…
Reducing the variance of the gradient estimator is known to improve the convergence rate of stochastic gradient-based optimization and sampling algorithms. One way of achieving variance reduction is to design importance sampling strategies.…
In this paper, we propose a novel sufficient decrease technique for variance reduced stochastic gradient descent methods such as SAG, SVRG and SAGA. In order to make sufficient decrease for stochastic optimization, we design a new…
We present a first-order method for solving constrained optimization problems. The method is derived from our previous work, a modified search direction method inspired by singular value decomposition. In this work, we simplify its…
We present two stochastic descent algorithms that apply to unconstrained optimization and are particularly efficient when the objective function is slow to evaluate and gradients are not easily obtained, as in some PDE-constrained…
In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG…
In this paper, we consider a novel auxiliary variable method to obtain energy stable schemes for gradient flows. The auxiliary variable based on energy bounded above does not limited to the hypothetical conditions adopted in previous…
We establish a general framework for developing, efficient energy stable numerical schemes for gradient flows and develop three classes of generalized scalar auxiliary variable approaches (G-SAV). Numerical schemes based on the G-SAV…
This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and…
In this paper, we construct two kinds of exponential SAV approach with relaxation (R-ESAV) for dissipative system. The constructed schemes are linear and unconditionally energy stable. They can guarantee the positive property of SAV without…
Stochastic variance reduced methods have shown strong performance in solving finite-sum problems. However, these methods usually require the users to manually tune the step-size, which is time-consuming or even infeasible for some…
An algorithm is said to be adaptive to a certain parameter (of the problem) if it does not need a priori knowledge of such a parameter but performs competitively to those that know it. This dissertation presents our work on adaptive…
In this paper, two efficient and magnetization norm preserving numerical schemes based on the scalar auxiliary variable (SAV) method are developed for calculating the ground state in micromagnetic structures. The first SAV scheme is based…
Large-scale non-convex sparsity-constrained problems have recently gained extensive attention. Most existing deterministic optimization methods (e.g., GraSP) are not suitable for large-scale and high-dimensional problems, and thus…
In this paper, we propose a novel family of high-order numerical schemes for the gradient flow models based on the scalar auxiliary variable (SAV) approach, which is named the high-order scalar auxiliary variable (HSAV) method. The newly…
We propose two novel conditional gradient-based methods for solving structured stochastic convex optimization problems with a large number of linear constraints. Instances of this template naturally arise from SDP-relaxations of…
In this paper, we propose a regularized auxiliary variable (RAV) approach and construct accurate and robust time-discrete schemes for a large class of gradient flows. By introducing an auxiliary variable $r=0$ and constructing an auxiliary…
In this paper, we propose a novel sufficient decrease technique for stochastic variance reduced gradient descent methods such as SVRG and SAGA. In order to make sufficient decrease for stochastic optimization, we design a new sufficient…