Related papers: Lower bounds on transformers with infinite precisi…
The softmax function is a ubiquitous component at the output of neural networks and increasingly in intermediate layers as well. This paper provides convex lower bounds and concave upper bounds on the softmax function, which are compatible…
Many canonical machine learning problems boil down to a convex optimization problem with a finite sum structure. However, whereas much progress has been made in developing faster algorithms for this setting, the inherent limitations of…
We deal with the problem of estimating the volume of inclusions using a finite number of boundary measurements in electrical impedance tomography. We derive upper and lower bounds on the volume fractions of inclusions, or more generally two…
This paper presents a lower bound for optimizing a finite sum of $n$ functions, where each function is $L$-smooth and the sum is $\mu$-strongly convex. We show that no algorithm can reach an error $\epsilon$ in minimizing all functions from…
Finite-sum optimization plays an important role in the area of machine learning, and hence has triggered a surge of interest in recent years. To address this optimization problem, various randomized incremental gradient methods have been…
Deriving formal bounds on the expressivity of transformers, as well as studying transformers that are constructed to implement known algorithms, are both effective methods for better understanding the computational power of transformers.…
We study the power sum problem max_{v=1,...,m} | sum_{k=1}^n z_k^v | and by using features of Fejer kernels we give new lower bounds in the case of unimodular complex numbers z_k and m cn^2 for constants c>1.
Certified verification of transformer attention requires bounding the softmax function over interval constraints on the pre-softmax scores. Existing verifiers relax softmax ndependently of the downstream objective, leaving avoidable slack.…
We prove a lower bound on the dimension of the set of maximal border subrank tensors. This is the first such bound of its type.
While transformers have proven enormously successful in a range of tasks, their fundamental properties as models of computation are not well understood. This paper contributes to the study of the expressive capacity of transformers,…
In this paper, we address two main topics. First, we study the problem of minimizing the sum of a smooth function and the composition of a weakly convex function with a linear operator on a closed vector subspace. For this problem, we…
We use the partition functions on S^1 x S^n of various conformal field theories in four and six dimensions in the limit of vanishing coupling to study the high temperature thermodynamics. Certain modular properties exhibited by the…
We prove a sharp upper bound on the Hausdorff dimension of weighted singular vectors in $\mathbb{R}^m$ using dynamics on homogeneous spaces, specifically the method of integral inequalities. Together with the lower bound proved recently by…
In this paper, we aim to build the global convergence theory of encoder-only shallow Transformers under a realistic setting from the perspective of architectures, initialization, and scaling under a finite width regime. The difficulty lies…
The VC-dimension of a set system is a way to capture its complexity and has been a key parameter studied extensively in machine learning and geometry communities. In this paper, we resolve two longstanding open problems on bounding the…
Transformers, especially the decoder-only variants, are the backbone of most modern large language models; yet we do not have much understanding of their expressive power except for the simple $1$-layer case. Due to the difficulty of…
We prove essentially sharp bounds for the $L^p$ restriction of weighted Gauss sums to monomial curves. Getting the $L^2$ upper bound combines the $TT^*$ method for matrices with the first and second derivative test for exponential sums. The…
We study minimization of a structured objective function, being the sum of a smooth function and a composition of a weakly convex function with a linear operator. Applications include image reconstruction problems with regularizers that…
Tensor network methods have been a key ingredient of advances in condensed matter physics and have recently sparked interest in the machine learning community for their ability to compactly represent very high-dimensional objects. Tensor…
We give a simpler proof, via query elimination, of a result due to O'Donnell, Saks, Schramm and Servedio, which shows a lower bound on the zero-error randomized query complexity of a function f in terms of the maximum influence of any…