Related papers: Baby-Step Giant-Step Algorithms for the Symmetric …
This paper presents fault-tolerant asynchronous Stochastic Gradient Descent (SGD) algorithms. SGD is widely used for approximating the minimum of a cost function $Q$, as a core part of optimization and learning algorithms. Our algorithms…
Iteration complexities for optimizing smooth functions with first-order algorithms are typically stated in terms of a global Lipschitz constant of the gradient, and near-optimal results are then achieved using fixed step sizes. But many…
Under mild assumptions stochastic gradient methods asymptotically achieve an optimal rate of convergence if the arithmetic mean of all iterates is returned as an approximate optimal solution. However, in the absence of stochastic noise, the…
In this paper, we establish new convergence results for the quantized distributed gradient descent and suggest a novel strategy of choosing the stepsizes for the high-performance of the algorithm. Under the strongly convexity assumption on…
Recent research has observed that in machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) [Cohen, et al., 2021], where the stepsizes are set to be large, resulting in non-monotonic losses…
Gradient descent (GD) on logistic regression has many fascinating properties. When the dataset is linearly separable, it is known that the iterates converge in direction to the maximum-margin separator regardless of how large the step size…
Let $G$ be a finite group generated by $k$ elements. The well-known product replacement algorithm provides an effective method for sampling generating sets of $G$. We study a refinement of this algorithm that is designed to output…
Let $\mathcal{D}$ be a block design admitting a locally transitive automorphism group $G$. We say that $\mathcal{D}$ is $G$-point-locally dihedral if the induced local action $G_x^{\mathcal{D}}$ is dihedral for each point $x$, and that…
The robotic locomotion community is interested in optimal gaits for control. Based on the optimization criterion, however, there could be a number of possible optimal gaits. For example, the optimal gait for maximizing displacement with…
Large language models (LLMs) can memorize and reproduce training sequences verbatim -- a tendency that undermines both generalization and privacy. Existing mitigation methods apply interventions uniformly, degrading performance on the…
Gradient Symbolic Computation is proposed as a means of solving discrete global optimization problems using a neurally plausible continuous stochastic dynamical system. Gradient symbolic dynamics involves two free parameters that must be…
We prove a Law of Iterated Logarithm for random walks on a family of diagonal products constructed by Brieussel and Zheng (2021). This provides a wide variety of new examples of Law of Iterated Logarithm behaviours for random walks on…
Symplectic N-body integrators are widely used to study problems in celestial mechanics. The most popular algorithms are of 2nd and 4th order, requiring 2 and 6 substeps per timestep, respectively. The number of substeps increases rapidly…
Self-correction is a novel method that can stimulate the potential reasoning abilities of large language models (LLMs). It involves detecting and correcting errors during the inference process when LLMs solve reasoning problems. However,…
In this paper we investigate the asymptotic distribution of likelihood ratio tests in models with several groups, when the number of groups converges with the dimension and sample size to infinity. We derive central limit theorems for the…
Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often…
In this paper, we consider a general stochastic optimization problem which is often at the core of supervised learning, such as deep learning and linear classification. We consider a standard stochastic gradient descent (SGD) method with a…
Let G be a simple non-compact linear connected Lie group and H be a closed non-compact semisimple subgroup. We are interested in finding classes of homogeneous spaces G/H admitting proper actions of discrete non virtually abelian subgroups…
For $g\in \mathbb{N}$, let $G=\Sp(2g,\mathbb{Z})$ be the integral symplectic group and $S(g)$ be the set of all positive integers which can occur as the order of an element in $G$. In this paper, we show that $S(g)$ is a bounded subset of…
Sequential hypothesis tests are widely adopted as a principled way to perform multiple tests on data that arrives over time. In particular, researchers frequently utilize group sequential hypothesis tests (GST) to test the same hypotheses…