Related papers: Monotonically Decreasing Sequence of Divergences
Twisted symmetries, widely studied in the last decade, proved to be as effective as standard ones in the analysis and reduction of nonlinear equations. We explain this effectiveness in terms of a Lie-Frobenius reduction; this requires to…
Information divergence that measures the difference between two nonnegative matrices or tensors has found its use in a variety of machine learning problems. Examples are Nonnegative Matrix/Tensor Factorization, Stochastic Neighbor…
The forward Kullback-Leibler (KL) divergence is a ubiquitous objective for fitting a parameterized distribution to samples due to its tractability and equivalence to maximum likelihood estimation (MLE). Its inherent asymmetry, however, may…
This work presents an upper-bound to value that the Kullback-Leibler (KL) divergence can reach for a class of probability distributions called quantum distributions (QD). The aim is to find a distribution $U$ which maximizes the KL…
Discretization of continuous-time diffusion processes is a widely recognized method for sampling. However, it seems to be a considerable restriction when the potentials are often required to be smooth (gradient Lipschitz). This paper…
This paper is focused on $f$-divergences, consisting of three main contributions. The first one introduces integral representations of a general $f$-divergence by means of the relative information spectrum. The second part provides a new…
This paper is concerned with the convergence of power sequences and stability of Hilbert space operators, where "convergence" and "stability" refer to weak, strong and norm topologies. It is proved that an operator has a convergent power…
This paper proposes two linear projection methods for supervised dimension reduction using only the first and second-order statistics. The methods, each catering to a different parameter regime, are derived under the general Gaussian model…
Kernel embeddings of distributions and the Maximum Mean Discrepancy (MMD), the resulting distance between distributions, are useful tools for fully nonparametric two-sample testing and learning on distributions. However, it is rarely that…
Properties of compositions and convex combinations of averaged nonexpansive operators are investigated and applied to the design of new fixed point algorithms in Hilbert spaces. An extended version of the forward-backward splitting…
We develop and analyze $M$-estimation methods for divergence functionals and the likelihood ratios of two probability distributions. Our method is based on a non-asymptotic variational characterization of $f$-divergences, which allows the…
The discrepancy function measures the deviation of the empirical distribution of a point set in $[0,1]^d$ from the uniform distribution. In this paper, we study the classical discrepancy function with respect to the BMO and exponential…
We show how a rescaling of fractional operators with bounded kernels may help circumvent their documented deficiencies, for example, the inconsistency at zero or the lack of inverse integral operator. On the other hand, we build a novel…
``Orderly divergence'' deals with limit theorems for weighted stochastic Gamma integrals of otherwise nonintegrable functions. Although for monotonic functions this category usually coincides with the classical notion of weighted limit…
The data processing inequality is central to information theory and motivates the study of monotonic divergences. However, it is not clear operationally we need to consider all such divergences. We establish a simple method for Pinsker…
The logistic loss (a.k.a. cross-entropy loss) is one of the most popular loss functions used for multiclass classification. It is also the loss function of choice for next-token prediction in language modeling. It is associated with the…
Recognizing subtle historical patterns is central to modeling and forecasting problems in time series analysis. Here we introduce and develop a new approach to quantify deviations in the underlying hidden generators of observed data…
We establish quantitative comparisons between classical distances for probability distributions belonging to the class of convex probability measures. Distances include total variation distance, Wasserstein distance, Kullback-Leibler…
Inversion of operators is a fundamental concept in data processing. Inversion of linear operators is well studied, supported by established theory. When an inverse either does not exist or is not unique, generalized inverses are used. Most…
This paper introduces a variational approximation framework using direct optimization of what is known as the {\it scale invariant Alpha-Beta divergence} (sAB divergence). This new objective encompasses most variational objectives that use…