Related papers: On the diameter of subgradient sequences in o-mini…

We present a convergence rate analysis for biased stochastic gradient descent (SGD), where individual gradient updates are corrupted by computation errors. We develop stochastic quadratic constraints to formulate a small linear matrix…

We study the oracle complexity of producing $(\delta,\epsilon)$-stationary points of Lipschitz functions, in the sense proposed by Zhang et al. [2020]. While there exist dimension-free randomized algorithms for producing such points within…

We propose an adaptive zeroth-order method for minimizing differentiable functions with $L$-Lipschitz continuous gradients. The method is designed to take advantage of the eventual compressibility of the gradient of the objective function,…

In this paper, we extend a recently established subgradient method for the computation of Riemannian metrics that optimizes certain singular value functions associated with dynamical systems. This extension is threefold. First, we introduce…

The analysis of gradient descent-type methods typically relies on the Lipschitz continuity of the objective gradient. This generally requires an expensive hyperparameter tuning process to appropriately calibrate a stepsize for a given…

In these notes, we present a general result concerning the Lipschitz regularity of a certain type of set-valued maps often found in constrained optimization and control problems. The class of multifunctions examined in this paper is…

We investigate the properties of minimizers of one-dimensional variational problems when the Lagrangian has no higher smoothness than continuity. An elementary approximation result is proved, but it is shown that this cannot be in general…

We study subspaces of Orlicz spaces $L_M$ spanned by independent copies $f_k$, $k=1,2,\dots$, of a function $f\in L_M$, $\int_0^1 f(t)\,dt=0$. Any such a subspace $H$ is isomorphic to some Orlicz sequence space $\ell_\psi$. In terms of…

We prove lower bounds for the approximation error of the variation-diminishing Schoenberg operator on the interval $[0,1]$ in terms of classical moduli of smoothness depending on the degree of the spline basis using a functional analysis…

We examine gradient descent in matrix factorization and show that under large step sizes the parameter space develops a fractal structure. We derive the exact critical step size for convergence in scalar-vector factorization and show that…

We establish convergence theorems for Riemannian stochastic gradient descents in which the underlying probability spaces vary from iteration to iteration. As applications, we deduce convergence results for Riemannian stochastic gradient…

A hypodifferential is a compact family of affine mappings that defines a local max-type approximation of a nonsmooth convex function. We present a general theory of hypodifferentials of nonsmooth convex functions defined on a Banach space.…

Dual first-order methods are powerful techniques for large-scale convex optimization. Although an extensive research effort has been devoted to studying their convergence properties, explicit convergence rates for the primal iterates have…

Gradient descent (GD) is a collection of continuous optimization methods that have achieved immeasurable success in practice. Owing to data science applications, GD with diminishing step sizes has become a prominent variant. While this…

Fractional derivatives are a well-studied generalization of integer order derivatives. Naturally, for optimization, it is of interest to understand the convergence properties of gradient descent using fractional derivatives. Convergence…

Level proximal subdifferential was introduced by Rockafellar recently for studying proximal mappings of possibly nonconvex functions. In this paper a systematic study of level proximal subdifferential is given. We characterize variational…

This work considers the question: what convergence guarantees does the stochastic subgradient method have in the absence of smoothness and convexity? We prove that the stochastic subgradient method, on any semialgebraic locally Lipschitz…

In this study we consider domains that are composed of an infinite sequence of self-similar rings and corresponding finite element spaces over those domains. The rings are parameterized using piecewise polynomial or tensor-product B-spline…

This is a handbook of simple proofs of the convergence of gradient and stochastic gradient descent type methods. We consider functions that are Lipschitz, smooth, convex, strongly convex, and/or Polyak-{\L}ojasiewicz functions. Our focus is…

In this paper we consider finite sum composite convex optimization problems with many functional constraints. The objective function is expressed as a finite sum of two terms, one of which admits easy computation of (sub)gradients while the…