Related papers: Functional Central Limit Theorem for Stochastic Gr…
Interpreting gradient methods as fixed-point iterations, we provide a detailed analysis of those methods for minimizing convex objective functions. Due to their conceptual and algorithmic simplicity, gradient methods are widely used in…
The paper presents a generalization of the local limit theorem on the convergence of inhomogeneous Markov chains to the diffusion limit for the case where the corresponding process coefficients satisfy weak regularity conditions and…
In this paper, we present a stochastic gradient algorithm for minimizing a smooth objective function that is an expectation over noisy cost samples, and only the latter are observed for any given parameter. Our algorithm employs a gradient…
We propose a new gradient descent algorithm with added stochastic terms for finding the global optimizers of nonconvex optimization problems. A key component in the algorithm is the adaptive tuning of the randomness based on the value of…
We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets. We show the predictor converges to the direction of the max-margin (hard margin SVM) solution. The…
Optimization problems with continuous data appear in, e.g., robust machine learning, functional data analysis, and variational inference. Here, the target function is given as an integral over a family of (continuously) indexed target…
We study Stochastic Gradient Descent (SGD) with diminishing step sizes for convex objective functions. We introduce a definitional framework and theory that defines and characterizes a core property, called curvature, of convex objective…
We study the block-coordinate forward-backward algorithm in which the blocks are updated in a random and possibly parallel manner, according to arbitrary probabilities. The algorithm allows different stepsizes along the block-coordinates to…
Recent studies show that transformer-based architectures emulate gradient descent during a forward pass, contributing to in-context learning capabilities - an ability where the model adapts to new tasks based on a sequence of prompt…
Online averaged stochastic gradient algorithms are more and more studied since (i) they can deal quickly with large sample taking values in high dimensional spaces, (ii) they enable to treat data sequentially, (iii) they are known to be…
In [20], the authors addressed the question of the averaging of a slow-fast Piecewise Deterministic Markov Process (PDMP) in infinite dimension. In the present paper, we carry on and complete this work by the mathematical analysis of the…
In this paper, we first study nonsmooth steepest descent method for nonsmooth functions defined on Hilbert space and establish the corresponding algorithm by proximal subgradients. Then, we use this algorithm to find stationary points for…
In this paper, we propose a new framework to study distributed optimization problems with stochastic gradients by employing a multi-agent system with continuous-time dynamics. Here the goal of the agents is to cooperatively minimize the sum…
We prove a sequence of limiting results about weakly dependent stationary and regularly varying stochastic processes in discrete time. After deducing the limiting distribution for individual clusters of extremes, we present a new type of…
In this work, a generalised version of the central limit theorem is proposed for nonlinear functionals of the empirical measure of i.i.d. random variables, provided that the functional satisfies some regularity assumptions for the…
The problem of minimizing convex functionals of probability distributions is solved under the assumption that the density of every distribution is bounded from above and below. A system of sufficient and necessary first-order optimality…
We consider the hard-edge scaling of the Mittag-Leffler ensemble confined to a fixed disk inside the droplet. Our primary emphasis is on fluctuations of rotationally-invariant additive statistics that depend on the radius and thus give rise…
The main result of this paper is a functional limit theorem for the sine-process. In particular, we study the limit distribution, in the space of trajectories, for the number of particles in a growing interval. The sine-process has the…
Stochastic Gradient Descent (SGD) with Polyak's stepsize has recently gained renewed attention in stochastic optimization. Recently, Orvieto, Lacoste-Julien, and Loizou introduced a decreasing variant of Polyak's stepsize, where convergence…
Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical…