Related papers: Baby-Step Giant-Step Algorithms for the Symmetric …

This work considers stepsize schedules for gradient descent on smooth convex objectives. We extend the existing literature and propose a unified technique for constructing stepsizes with analytic bounds for an arbitrary number of…

Recent advances in the theoretical understanding of SGD led to a formula for the optimal batch size minimizing the number of effective data passes, i.e., the number of iterations times the batch size. However, this formula is of no…

The field of discrete event simulation and optimization techniques motivates researchers to adjust classic ranking and selection (R&S) procedures to the settings where the number of populations is large. We use insights from extreme value…

Stochastic Gradient Descent (SGD) is a popular tool in training large-scale machine learning models. Its performance, however, is highly variable, depending crucially on the choice of the step sizes. Accordingly, a variety of strategies for…

The tuning of stochastic gradient algorithms (SGAs) for optimization and sampling is often based on heuristics and trial-and-error rather than generalizable theory. We address this theory--practice gap by characterizing the large-sample…

Stochastic optimization lies at the core of most statistical learning models. The recent great development of stochastic algorithmic tools focused significantly onto proximal gradient iterations, in order to find an efficient approach for…

Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several…

We revisit the selection problem, namely that of computing the $i$th order statistic of $n$ given elements, in particular the classic deterministic algorithm by grouping and partition due to Blum, Floyd, Pratt, Rivest, and Tarjan (1973).…

Linear multistep methods (LMMs) applied to approximate the solution of initial value problems---typically arising from method-of-lines semidiscretizations of partial differential equations---are often required to have certain monotonicity…

In this paper we introduce a novel framework for making exact nonparametric Bayesian inference on latent functions, that is particularly suitable for Big Data tasks. Firstly, we introduce a class of stochastic processes we refer to as…

The spread of a group $G$, written $s(G)$, is the largest $k$ such that for any nontrivial elements $x_1, \dots, x_k \in G$ there exists $y \in G$ such that $G = \langle x_i, y \rangle$ for all $i$. Burness, Guralnick and Harper recently…

We study a distributed learning process observed in human groups and other social animals. This learning process appears in settings in which each individual in a group is trying to decide over time, in a distributed manner, which option to…

Learning representation from relative similarity comparisons, often called ordinal embedding, gains rising attention in recent years. Most of the existing methods are based on semi-definite programming (\textit{SDP}), which is generally…

This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local…

We develop practical techniques to compute with arithmetic groups $H\leq \mathrm{SL}(n,\mathbb{Q})$ for $n>2$. Our approach relies on constructing a principal congruence subgroup in $H$. Problems solved include testing membership in $H$,…

Given a set of n points in the plane, each point having a positive weight, and an integer k>0, we present an optimal O(n \log n)-time deterministic algorithm to compute a step function with k steps that minimizes the maximum weighted…

Given a countably infinite group $G$ acting on some space $X$, an increasing family of finite subsets $G_n$ and $x\in X$, a natural question to ask is what asymptotical distribution the sets $G_nx$ form. More formally, we define for a…

Discrete random probability measures are a key ingredient of Bayesian nonparametric inferential procedures. A sample generates ties with positive probability and a fundamental object of both theoretical and applied interest is the…

The Number Field Sieve (NFS) algorithm is the best known method to compute discrete logarithms (DL) in finite fields $\mathbb{F}\_{p^n}$, with $p$ medium to large and $n \geq 1$ small. This algorithm comprises four steps: polynomial…

The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature. In this paper, we focus instead on batch methods that use a sizeable fraction of the training set at each…