Related papers: Lower bounds on transformers with infinite precisi…
In a complete metric space that is equipped with a doubling measure and supports a Poincar\'e inequality, we study strict subsets, i.e. sets whose variational capacity with respect to a larger reference set is finite, in the case $p=1$.…
The softmax function is widely used in artificial neural networks for the multiclass classification problems, where the softmax transformation enforces the output to be positive and sum to one, and the corresponding loss function allows to…
Given a prediction task, understanding when one can and cannot design a consistent convex surrogate loss, particularly a low-dimensional one, is an important and active area of machine learning research. The prediction task may be given as…
We propose using mechanistic interpretability -- techniques for reverse engineering model weights into human-interpretable algorithms -- to derive and compactly prove formal guarantees on model performance. We prototype this approach by…
We construct a family of functions suitable for establishing lower bounds on the oracle complexity of first-order minimization of smooth strongly-convex functions. Based on this construction, we derive new lower bounds on the complexity of…
We give a complete solution to an open problem of Thomas Cover in 1987 about the capacity of a relay channel in the general discrete memoryless setting without any additional assumptions. The key step in our approach is to lower bound a…
We bound the number of nearly orthogonal vectors with fixed VC-dimension over $\setpm^n$. Our bounds are of interest in machine learning and empirical process theory and improve previous bounds by Haussler. The bounds are based on a simple…
We provide lower error bounds for randomized algorithms that approximate integrals of functions depending on an unrestricted or even infinite number of variables. More precisely, we consider the infinite-dimensional integration problem on…
Smooth finite-sum optimization has been widely studied in both convex and nonconvex settings. However, existing lower bounds for finite-sum optimization are mostly limited to the setting where each component function is (strongly) convex,…
We prove super-polynomial lower bounds for low-depth arithmetic circuits using the shifted partials measure [Gupta-Kamath-Kayal-Saptharishi, CCC 2013], [Kayal, ECCC 2012] and the affine projections of partials measure [Garg-Kayal-Saha, FOCS…
We study the conditions under which one is able to efficiently apply variance-reduction and acceleration schemes on finite sum optimization problems. First, we show that, perhaps surprisingly, the finite sum structure by itself, is not…
We present a new proof (based on spectral decomposition) of a bound originally proved by Sidelnikov~\, for the frame potentials $\sum_{ij} \left( {\bf P}_i \cdot {\bf P}_j \right)^\ell $ on a unit--sphere in $d$ dimensions. Sidelnikov's…
Vertex direction algorithms have been around for a few decades in the experimental design and mixture models literature. We briefly review this type of algorithm and describe a new member of the family: the support reduction algorithm. The…
Following the ideas from a paper by the same author, we prove abstract maximal restriction results for the Fourier transform. Our results deal mainly with maximal operators of convolution-type and $r-$average maximal functions. As a…
In this paper, we first introduce a lower bound technique for the state complexity of transformations of automata. Namely we suggest first considering the class of full automata in lower bound analysis, and later reducing the size of the…
This paper explores the well known approximation approach to decide weak bisimilarity of Basic Parallel Processes. We look into how different refinement functions can be used to prove weak bisimilarity decidable for certain subclasses. We…
We revisit the isoperimetric inequalities for finitely generated groups introduced and studied by N. Varopoulos, T. Coulhon and L. Saloff-Coste. Namely we show that a lower bound on the isoperimetric quotient of finite subsets in a finitely…
In this paper we present two frameworks in which global maximization of a bounded hessian function over a strongly convex set can be reduced to convex optimization. The first presented framework is a continuation of one of our previous…
Submodular maximization under various constraints is a fundamental problem studied continuously, in both computer science and operations research, since the late $1970$'s. A central technique in this field is to approximately optimize the…
The Transformer model is widely used in various application areas of machine learning, such as natural language processing. This paper investigates the approximation of the H\"older continuous function class…