Related papers: Baby-Step Giant-Step Algorithms for the Symmetric …
Many popular learning-rate schedules for deep neural networks combine a decaying trend with local perturbations that attempt to escape saddle points and bad local minima. We derive convergence guarantees for bandwidth-based step-sizes, a…
Stochastic Gradient Descent (SGD) and its variants are almost universally used to train neural networks and to fit a variety of other parametric models. An important hyperparameter in this context is the batch size, which determines how…
This paper provides a non-asymptotic analysis of linear stochastic approximation (LSA) algorithms with fixed stepsize. This family of methods arises in many machine learning tasks and is used to obtain approximate solutions of a linear…
We consider speeding up stochastic gradient descent (SGD) by parallelizing it across multiple workers. We assume the same data set is shared among $N$ workers, who can take SGD steps and coordinate with a central server. While it is…
Given a locally finite graph $\Gamma$, an amenable subgroup $G$ of graph automorphisms acting freely and almost transitively on its vertices, and a $G$-invariant activity function $\lambda$, consider the free energy $f_G(\Gamma,\lambda)$ of…
Fix a word $w$ in a free group $F$ on $r$ generators. A $w$-random permutation in the symmetric group $S_N$ is obtained by sampling $r$ independent uniformly random permutations $\sigma_{1},\ldots,\sigma_{r}\in S_{N}$ and evaluating…
Mathematical reasoning has been challenging for large language models (LLMs), and the introduction of step-by-step Chain-of-Thought (CoT) inference has significantly advanced the mathematical capabilities of LLMs. However, current…
We study the small deviation probabilities of a family of very smooth self-similar Gaussian processes. The canonical process from the family has the same scaling property as standard Brownian motion and plays an important role in the study…
This paper deals with the problem of simulating dense dispersed systems composed by large numbers of particles undergoing ballistic aggregation. The most classical approaches for dealing with such problems are represented by the so-called…
A new strategy is presented for systematically treating super-leading logarithmic contributions including higher-order Glauber exchanges for non-global LHC observables in renormalization-group (RG) improved perturbation theory. This…
We propose a new stochastic L-BFGS algorithm and prove a linear convergence rate for strongly convex and smooth functions. Our algorithm draws heavily from a recent stochastic variant of L-BFGS proposed in Byrd et al. (2014) as well as a…
Convergence and generalization are two crucial aspects of performance in neural networks. When analyzed separately, these properties may lead to contradictory results. Optimizing a convergence rate yields fast training, but does not…
We get asymptotics for the volume of large balls in an arbitrary locally compact group G with polynomial growth. This is done via a study of the geometry of G and a generalization of P. Pansu's thesis. In particular, we show that any such G…
We consider the problem of sequentially maximizing an unknown function $f$ over a set of actions of the form $(s,\mathbf{x})$, where the selected actions must satisfy a safety constraint with respect to an unknown safety function $g$. We…
We study the statistical properties of the dynamic trajectory of stochastic gradient descent (SGD). We approximate the mini-batch SGD and the momentum SGD as stochastic differential equations (SDEs). We exploit the continuous formulation of…
A great variety of fundamental optimization and counting problems arising in computer science, mathematics and physics can be reduced to one of the following computational tasks involving polynomials and set systems: given an $m$-variate…
The Barzilai and Borwein (BB) gradient method is one of the most widely-used line-search gradient methods. It computes the step-size for the current iterate by using the information carried in the previous iteration. Recently, William Kahan…
The superiority of stochastic symplectic methods over non-symplectic counterparts has been verified by plenty of numerical experiments, especially in capturing the asymptotic behaviour of the underlying solution process. How can one…
Krylov subspace methods are extensively used in scientific computing to solve large-scale linear systems. However, the performance of these iterative Krylov solvers on modern supercomputers is limited by expensive communication costs. The…
Gaussian processes (GPs) are important models in supervised machine learning. Training in Gaussian processes refers to selecting the covariance functions and the associated parameters in order to improve the outcome of predictions, the core…