Related papers: The Niceness of Unique Sink Orientations

Randomly initialized first-order optimization algorithms are the method of choice for solving many high-dimensional nonconvex problems in machine learning, yet general theoretical guarantees cannot rule out convergence to critical points of…

We study the complexity of optimizing highly smooth convex functions. For a positive integer $p$, we want to find an $\epsilon$-approximate minimum of a convex function $f$, given oracle access to the function and its first $p$ derivatives,…

The object of study is a soft random geometric graph with vertices given by a Poisson point process on a line and edges between vertices present with probability that has a polynomial decay in the distance between them. Various aspects of…

Large learning rates, when applied to gradient descent for nonconvex optimization, yield various implicit biases including the edge of stability (Cohen et al., 2021), balancing (Wang et al., 2022), and catapult (Lewkowycz et al., 2020).…

We consider a modified random walk which uses unvisited edges whenever possible, and makes a simple random walk otherwise. We call such a walk an edge-process. We assume there is a rule A, which tells the walk which unvisited edge to use…

Random Reshuffling (RR) is an algorithm for minimizing finite-sum functions that utilizes iterative gradient descent steps in conjunction with data reshuffling. Often contrasted with its sibling Stochastic Gradient Descent (SGD), RR is…

For a one-dimensional simple symmetric random walk $(S_n)$, an edge $x$ (between points $x-1$ and $x$) is called a favorite edge at time $n$ if its local time at $n$ achieves the maximum among all edges. In this paper, we show that with…

We analyze the behavior of randomized coordinate gradient descent for nonconvex optimization, proving that under standard assumptions, the iterates almost surely escape strict saddle points. By formulating the method as a nonlinear random…

Consider a random set of points on the unit sphere in $\mathbb{R}^d$, which can be either uniformly sampled or a Poisson point process. Its convex hull is a random inscribed polytope, whose boundary approximates the sphere. We focus on the…

We propose a new framework for analyzing zeroth-order optimization (ZOO) from the perspective of \emph{oblivious randomized sketching}.In this framework, commonly used gradient estimators in ZOO-such as finite difference (FD) and random…

Rank-based zeroth-order (ZO) optimization -- which relies only on the ordering of function evaluations -- offers strong robustness to noise and monotone transformations, and underlies many successful algorithms such as CMA-ES, natural…

We consider symmetric and Hermitian random matrices whose entries are independent and symmetric random variables with an arbitrary variance pattern. Under a novel Short-to-Long Mixing condition, which is sharp in the sense that it precludes…

We show that the pivoting process associated with one line and $n$ points in $r$-dimensional space may need $\Omega(\log^r n)$ steps in expectation as $n \to \infty$. The only cases for which the bound was known previously were for $r \le…

The worst-case expected length f(n) of the path taken by the simplex algorithm with the Random Edge pivot rule on a 3-dimensional linear program with n constraints is shown to be bounded by 1.3445 n <= f(n) <= 1.4943 n for large enough n.

We propose a simple, geometrically-motivated construction of smooth random paths in the plane. The construction is such that, with probability one, the paths have finite curvature everywhere (and the realizations are visually pleasing when…

We consider minimization of a smooth nonconvex function with inexact oracle access to gradient and Hessian (without assuming access to the function value) to achieve approximate second-order optimality. A novel feature of our method is that…

We investigate the iterative behaviour of continuous order preserving subhomogeneous maps that map a polyhedral cone into itself. For these maps we show that every bounded orbit converges to a periodic orbit and, moreover, that there exists…

A cyclic random walk is a random walk whose transition probabilities/rates can be written as a superposition of the empirical measures of a family of finite cycles. This identifies a convex set of models. We discuss the problem of…

The existence of a pivot rule for the simplex method that guarantees a strongly polynomial run-time is a longstanding, fundamental open problem in the theory of linear programming. The leading pivot rule in theory is the shadow pivot rule,…

Estimating the number of vertices of a two dimensional projection, called a shadow, of a polytope is a fundamental tool for understanding the performance of the shadow simplex method for linear programming among other applications. We prove…