Related papers: On the method of typical bounded differences
Self-attention and masked self-attention are at the heart of Transformers' outstanding success. Still, our mathematical understanding of attention, in particular of its Lipschitz properties - which are key when it comes to analyzing…
We derive two upper bounds for the probability of deviation of a vector-valued Lipschitz function of a collection of random variables from its expected value. The resulting upper bounds can be tighter than bounds obtained by a direct…
In this work we derive multi-level concentration inequalities for polynomial functions in independent random variables with a $\alpha$-sub-exponential tail decay. A particularly interesting case is given by quadratic forms $f(X_1, \ldots,…
The method to derive uniform bounds with Gaussian and Rademacher complexities is extended to the case where the sample average is replaced by a nonlinear statistic. Tight bounds are obtained for U-statistics, smoothened L-statistics and…
Upper bounds for the probabilities $\mathbb{P}(F\geq \mathbb{E} F + r)$ and $\mathbb{P}(F\leq \mathbb{E} F - r)$ are proved, where $F$ is a certain component count associated with a random geometric graph built over a Poisson point process…
Almost 10 years ago, Impagliazzo and Kabanets (2010) gave a new combinatorial proof of Chernoff's bound for sums of bounded independent random variables. Unlike previous methods, their proof is constructive. This means that it provides an…
It is well known that Hoeffding's inequality has a lot of applications in the signal and information processing fields. How to improve Hoeffding's inequality and find the refinements of its applications have always attracted much…
In this note, we improve some concentration inequalities for martingales with bounded increments. These results recover the missing factor in Freedman-style inequalities and are near optimal. We also provide minor refinements of…
The Lipschitz bound, a technique from robust statistics, can limit the maximum changes in the output concerning the input, taking into account associated irrelevant biased factors. It is an efficient and provable method for examining the…
A concentration result for quadratic form of independent subgaussian random variables is derived. If the moments of the random variables satisfy a "Bernstein condition", then the variance term of the Hanson-Wright inequality can be…
One of the main open problems in the theory of multi-category margin classification is the form of the optimal dependency of a guaranteed risk on the number C of categories, the sample size m and the margin parameter gamma. From a practical…
During recent years the interest of optimization and machine learning communities in high-probability convergence of stochastic optimization methods has been growing. One of the main reasons for this is that high-probability complexity…
Bounds of the accuracy of the normal approximation to the distribution of a sum of independent random variables are improved under relaxed moment conditions, in particular, under the absence of moments of orders higher than the second.…
In many statistical linear inverse problems, one needs to recover classes of similar curves from their noisy images under an operator that does not have a bounded inverse. Problems of this kind appear in many areas of application.…
We prove anti-concentration bounds for the inner product of two independent random vectors. For example, we show that if $A,B$ are subsets of the cube $\{\pm 1\}^n$ with $|A| \cdot |B| \geq 2^{1.01 n}$, and $X \in A$ and $Y \in B$ are…
The concentration of measure phenomenon in Gauss' space states that every $L$-Lipschitz map $f$ on $\mathbb R^n$ satisfies \[ \gamma_{n} \left(\{ x : | f(x) - M_{f} | \geqslant t \} \right) \leqslant 2 e^{ - \frac{t^2}{ 2L^2} }, \quad t>0,…
The focal-loss has become a widely used alternative to cross-entropy in class-imbalanced classification problems, particularly in computer vision. Despite its empirical success, a systematic information-theoretic study of the focal-loss…
Let $X,X_1,\ldots,X_n$ be independent identically distributed random variables. In this paper we study the behavior of concentration functions of weighted sums $\sum_{k=1}^{n} X_k a_k$ with respect to the arithmetic structure of…
In this report, we aim to exemplify concentration inequalities and provide easy to understand proofs for it. Our focus is on the inequalities which are helpful in the design and analysis of machine learning algorithms.
This paper discusses a general and useful stability principle which, roughly speaking, says that given a uniformly continuous function defined on an arbitrary metric space, if the function is bounded on the constraint set and we slightly…