Related papers: A Caputo fractional derivative-based algorithm for…
The certified descent algorithm (CDA) is a gradient-based method for shape optimization which certifies that the direction computed using the shape gradient is a genuine descent direction for the objective functional under analysis. It…
In this paper, a high-order approximation to Caputo-type time-fractional diffusion equations involving an initial-time singularity of the solution is proposed. At first, we employ a numerical algorithm based on the Lagrange polynomial…
This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. The proposed algorithms are theoretically…
Establishing a fast rate of convergence for optimization methods is crucial to their applicability in practice. With the increasing popularity of deep learning over the past decade, stochastic gradient descent and its adaptive variants…
We propose an algorithm for the adaptation of the learning rate for stochastic gradient descent (SGD) that avoids the need for validation set use. The idea for the adaptiveness comes from the technique of extrapolation: to get an estimate…
Stochastic gradient descent (SGD) is a fundamental optimization algorithm widely used in modern machine learning. In this paper, we propose Factor-Augmented SGD (FSGD), a new optimization method that leverages latent factor representations…
Stochastic gradient descent (SGD) gives an optimal convergence rate when minimizing convex stochastic objectives $f(x)$. However, in terms of making the gradients small, the original SGD does not give an optimal rate, even when $f(x)$ is…
Motivated by robust and quantile regression problems, we investigate the stochastic gradient descent (SGD) algorithm for minimizing an objective function $f$ that is locally strongly convex with a sub--quadratic tail. This setting covers…
In this work, we propose an exponentially convergent numerical method for the Caputo fractional propagator $S_\alpha(t)$ and the associated mild solution of the Cauchy problem with time-independent sectorial operator coefficient $A$ and…
A new universal derivative-free optimization method CDOS (Conjugate Direction with Orthogonal Shift) is proposed. The CDOS method was specially developed to solve optimization tasks where the objective function and constraints are black…
We present an algorithm for minimizing a sum of functions that combines the computational efficiency of stochastic gradient descent (SGD) with the second order curvature information leveraged by quasi-Newton methods. We unify these…
We present a method to solve fractional optimal control problems, where the dynamic depends on integer and Caputo fractional derivatives. Our approach consists to approximate the initial fractional order problem with a new one that involves…
The conjugate gradient (CG) method is an efficient iterative method for solving large-scale strongly convex quadratic programming (QP). In this paper we propose some generalized CG (GCG) methods for solving the $\ell_1$-regularized…
We consider the Quantum Natural Gradient Descent (QNGD) scheme which was recently proposed to train variational quantum algorithms. QNGD is Steepest Gradient Descent (SGD) operating on the complex projective space equipped with the…
Due to the high communication cost in distributed and federated learning problems, methods relying on compression of communicated messages are becoming increasingly popular. While in other contexts the best performing gradient-type methods…
We present an efficient algorithm for the evaluation of the Caputo fractional derivative $_0^C\!D_t^\alpha f(t)$ of order $\alpha\in (0,1)$, which can be expressed as a convolution of $f'(t)$ with the kernel $t^{-\alpha}$. The algorithm is…
Coordinate descent (CD) algorithms have become the method of choice for solving a number of optimization problems in machine learning. They are particularly popular for training linear models, including linear support vector machine…
This paper presents a novel algorithm that leverages Stochastic Gradient Descent strategies in conjunction with Random Features to augment the scalability of Conic Particle Gradient Descent (CPGD) specifically tailored for solving sparse…
Based on the continuous time random walk, we derive the Fokker-Planck equations with Caputo-Fabrizio fractional derivative, which can effectively model a variety of physical phenomena, especially, the material heterogeneities and structures…
Line-search methods are commonly used to solve optimization problems. The simplest line search method is steepest descent where one always moves in the direction of the negative gradient. Newton's method on the other hand is a second-order…