Related papers: Direct Runge-Kutta Discretization Achieves Acceler…

The Obreshkov method is a single-step multi-derivative method used in the numerical solution of differential equations and has been used in recent years in efficient circuit simulation. It has been shown that it can be made of arbitrary…

We study gradient descent (GD) with a constant stepsize for $\ell_2$-regularized logistic regression with linearly separable data. Classical theory suggests small stepsizes to ensure monotonic reduction of the optimization objective,…

This paper concerns the numerical procedure for solving hybrid optimal control problems with sliding modes. The proposed procedure has several features which distinguishes it from the other procedures for the problem. First of all a sliding…

We present unconditionally energy stable Runge-Kutta (RK) discontinuous Galerkin (DG) schemes for solving a class of fourth order gradient flows. Our algorithm is geared toward arbitrarily high order approximations in both space and time,…

Nesterov's accelerated gradient descent method (AGD) is a seminal deterministic first-order method known to achieve the optimal order of iteration complexity for solving convex smooth optimization problems. Two distinct sequences of…

Explicit stabilized methods are an efficient alternative to implicit schemes for the time integration of stiff systems of differential equations in large dimension. In this paper, we derive explicit stabilized integrators of orders one and…

A new class of third order Runge-Kutta methods for stochastic differential equations with additive noise is introduced. In contrast to Platen's method, which to the knowledge of the author has been up to now the only known third order…

This paper shows that the implicit bias of gradient descent on linearly separable data is exactly characterized by the optimal solution of a dual optimization problem given by a smoothed margin, even for general losses. This is in contrast…

In this paper we investigate the generalization error of gradient descent (GD) applied to an $\ell_2$-regularized OLS objective function in the linear model. Based on our analysis we develop new methodology for computationally tractable and…

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by…

We revisit the general framework introduced by Fazylab et al. (SIAM J. Optim. 28, 2018) to construct Lyapunov functions for optimization algorithms in discrete and continuous time. For smooth, strongly convex objective functions, we relax…

Gradient descent and its variants are widely used in machine learning. However, oracle access of gradient may not be available in many applications, limiting the direct use of gradient descent. This paper proposes a method of estimating…

Discrete gradient methods are a class of numerical integrators producing solutions with exact preservation of first integrals of ordinary differential equations. In this paper, we apply order theory combined with the symmetrized Itoh--Abe…

Nesterov's accelerated gradient method for minimizing a smooth strongly convex function $f$ is known to reduce $f(\x_k)-f(\x^*)$ by a factor of $\eps\in(0,1)$ after $k\ge O(\sqrt{L/\ell}\log(1/\eps))$ iterations, where $\ell,L$ are the two…

This study computes the gradient of a function of numerical solutions of ordinary differential equations (ODEs) with respect to the initial condition. The adjoint method computes the gradient approximately by solving the corresponding…

Recent work on high-resolution ordinary differential equations (HR-ODEs) captures fine nuances among different momentum-based optimization methods, leading to accurate theoretical insights. However, these HR-ODEs often appear disconnected,…

Zeroth-order methods are extensively used in machine learning applications where gradients are infeasible or expensive to compute, such as black-box attacks, reinforcement learning, and language model fine-tuning. Existing optimization…

This paper considers the numerical integration of semilinear evolution PDEs using the high order linearly implicit methods developped in a previous paper in the ODE setting. These methods use a collocation Runge--Kutta method as a basis,…

This paper studies the Nesterov-Spokoiny Acceleration (NSA), a variant of the accelerated gradient method by Nesterov and Spokoiny. For smooth convex optimization, NSA achieves a strict $o(1/k^2)$ convergence rate in function value and an…

In order to solve continuous-time optimal control problems, direct methods transcribe the infinite-dimensional problem to a nonlinear program (NLP) using numerical integration methods. In cases where the integration error can be manipulated…