Related papers: Variable stepsize SDIMSIMs for ordinary differenti…

This article proposes a new class of general linear method with $p=q$ and $r=s=p+1$. The construction of the present method is carried out using order conditions and error minimization subject to $A$- stability constraints. The proposed…

Time integration methods for solving initial value problems are an important component of many scientific and engineering simulations. Implicit time integrators are desirable for their stability properties, significantly relaxing…

We study Stochastic Gradient Descent with AdaGrad stepsizes: a popular adaptive (self-tuning) method for first-order stochastic optimization. Despite being well studied, existing analyses of this method suffer from various shortcomings:…

In view of a direct and simple improvement of vanilla SGD, this paper presents a fine-tuning of its step-sizes in the mini-batch case. For doing so, one estimates curvature, based on a local quadratic model and using only noisy gradient…

In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an $O(\frac{1}{\sqrt{T}})$ convergence rate for…

This paper presents a class of Two-Step General Linear Methods for the numerical solution of Retarded Functional Differential Equations. Explicit methods up to order five are constructed. To avoid order reduction for mildly stiff problems…

This paper presents a class of Two-Step General Linear Methods for the numerical solution of Retarded Functional Differential Equations. Explicit methods up to order five are constructed. To avoid order reduction for mildly stiff problems…

Stochastic Gradient Descent (SGD) is a popular tool in training large-scale machine learning models. Its performance, however, is highly variable, depending crucially on the choice of the step sizes. Accordingly, a variety of strategies for…

Stochastic gradient-based descent (SGD), have long been central to training large language models (LLMs). However, their effectiveness is increasingly being questioned, particularly in large-scale applications where empirical evidence…

We study a fixed step-size noisy distributed gradient descent algorithm for solving optimization problems in which the objective is a finite sum of smooth but possibly non-convex functions. Random perturbations are introduced to the…

In this work, in order to obtain higher-order schemes for solving forward backward stochastic differential equations, we adopt the high-order multi-step method in [W. Zhao, Y. Fu and T. Zhou, SIAM J. Sci. Comput., 36(4) (2014),…

In this paper, we propose a class of super-schemes for efficiently solving nonlinear unconstrained optimization problems. The proposed approach introduces two novel choices of step-size parameters, leading to efficient descent directions…

We study gradient descent (GD) with a constant stepsize for $\ell_2$-regularized logistic regression with linearly separable data. Classical theory suggests small stepsizes to ensure monotonic reduction of the optimization objective,…

Selecting an effective step-size is a fundamental challenge in first-order optimization, especially for problems with non-Euclidean geometries. This paper presents a novel adaptive step-size strategy for optimization algorithms that rely on…

In this paper we construct a third order method for solving additively split autonomous stiff systems of ordinary differential equations. The constructed additive method is L-stable with respect to the implicit part and allows to use an…

The aim of this work is to apply a semi-implicit (SI) strategy within a Rosenbrock-type and IMEX linear multistep (LM) framework to a sequence of 1D time-dependent partial differential equations (PDEs) with high order spatial derivatives.…

One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms,…

Many popular learning-rate schedules for deep neural networks combine a decaying trend with local perturbations that attempt to escape saddle points and bad local minima. We derive convergence guarantees for bandwidth-based step-sizes, a…

We propose a new per-layer adaptive step-size procedure for stochastic first-order optimization methods for minimizing empirical loss functions in deep learning, eliminating the need for the user to tune the learning rate (LR). The proposed…

The recent literature on first order methods for smooth optimization shows that significant improvements on the practical convergence behaviour can be achieved with variable stepsize and scaling for the gradient, making this class of…