Related papers: 2nd-order Updates with 1st-order Complexity
It is well known that both gradient descent and stochastic coordinate descent achieve a global convergence rate of $O(1/k)$ in the objective value, when applied to a scheme for minimizing a Lipschitz-continuously differentiable,…
We prove lower bounds for higher-order methods in smooth non-convex finite-sum optimization. Our contribution is threefold: We first show that a deterministic algorithm cannot profit from the finite-sum structure of the objective, and that…
These notes focus on the minimization of convex functionals using first-order optimization methods, which are fundamental in many areas of applied mathematics and engineering. The primary goal of this document is to introduce and analyze…
Coordinate descent methods have considerable impact in global optimization because global (or, at least, almost global) minimization is affordable for low-dimensional problems. Coordinate descent methods with high-order regularized models…
In this work, new finite difference schemes are presented for dealing with the upper-convected time derivative in the context of the generalized Lie derivative. The upper-convected time derivative, which is usually encountered in the…
In this paper, we consider both first- and second-order techniques to address continuous optimization problems arising in machine learning. In the first-order case, we propose a framework of transition from deterministic or…
The power flow equations are fundamental to power system planning, analysis, and control. However, the inherent non-linearity and non-convexity of these equations present formidable obstacles in problem-solving processes. To mitigate these…
The smoothed particle hydrodynamics (SPH) method has been widely used to simulate incompressible and slightly compressible fluid flows. Adaptive refinement strategies to dynamically increase the resolution of the particles to capture sharp…
In this paper, we present and analyse a class of "filtered" numerical schemes for second order Hamilton-Jacobi-Bellman equations. Our approach follows the ideas introduced in B.D. Froese and A.M. Oberman, Convergent filtered schemes for the…
Obtaining system parameters and reconstructing the full flow state from limited velocity observations using conventional fluid dynamics solvers can be prohibitively expensive. Here we employ machine learning algorithms to overcome the…
In this paper, we study and analyze zeroth-order stochastic approximation algorithms for solving bilvel problems, when neither the upper/lower objective values, nor their unbiased gradient estimates are available. In particular, exploiting…
Stochastic optimization methods have actively been playing a critical role in modern machine learning algorithms to deliver decent performance. While numerous works have proposed and developed diverse approaches, first-order and…
Training in supervised deep learning is computationally demanding, and the convergence behavior is usually not fully understood. We introduce and study a second-order stochastic quasi-Gauss-Newton (SQGN) optimization method that combines…
We develop and analyze several different second-order algorithms for computing a near-optimal solution path of a convex parametric optimization problem with smooth Hessian. Our algorithms are inspired by a differential equation perspective…
Arising in semi-parametric statistics, control applications, and as sub-problems in global optimization methods, certain optimization problems can have objective functions requiring numerical integration to evaluate, yet gradient function…
The autoencoder model typically uses an encoder to map data to a lower dimensional latent space and a decoder to reconstruct it. However, relying on an encoder for inversion can lead to suboptimal representations, particularly limiting in…
We obtain a new lower bound on the information-based complexity of first-order minimization of smooth and convex functions. We show that the bound matches the worst-case performance of the recently introduced Optimized Gradient Method,…
The problem of recovering partial derivatives of high orders of bivariate functions with finite smoothness is studied. Based on the truncation method, a numerical differentiation algorithm was constructed, which is optimal by the order,…
We focus here on a class of fourth-order parabolic equations that can be written as a system of second-order equations by introducing an auxiliary variable. We design a novel second-order fully discrete mixed finite element method to…
Quasi-Newton methods refer to a class of algorithms at the interface between first and second order methods. They aim to progress as substantially as second order methods per iteration, while maintaining the computational complexity of…