Related papers: Generalized Information Inequalities via Submodula…
Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these…
We introduce a novel generalization of entropy and conditional entropy from which most definitions from the literature can be derived as particular cases. Within this general framework, we investigate the problem of designing…
In the paper, the authors review origins, motivations, and generalizations of a series of inequalities involving several exponential functions and sums, establish three new inequalities involving finite exponential functions and sums by…
Information inequalities govern the ultimate limitations in information theory and as such play an pivotal role in characterizing what values the entropy of multipartite states can take. Proving an information inequality, however, quickly…
We present a new family of information-theoretic generalization bounds, in which the training loss and the population loss are compared through a jointly convex function. This function is upper-bounded in terms of the disintegrated,…
The aim of this article is to establish new two-functions minimax inequalities extending classical results such as Simons' minimax theorem. Our results will be proved in a non-compact setting. We also prove, under general conditions, that…
"Bounds on information combining" are entropic inequalities that determine how the information (entropy) of a set of random variables can change when these are combined in certain prescribed ways. Such bounds play an important role in…
Inspired by the recent work by R.Pal et al., we give further refined inequalities for a convex Riemann integrable function, applying the standard Hermite-Hadamard inequality. Our approach is different from their one in \cite{PSMA2016}. As…
Some new inequalities of Karamata type are established with a convex function in this paper. The methods of our proof allow us to obtain an extended version of the reverse of Jensen inequality given by Pe{\v} cari\'c and Mi\'ci\'c. Applying…
A new trend in deep learning, represented by Mutual Information Neural Estimation (MINE) and Information Noise Contrast Estimation (InfoNCE), is emerging. In this trend, similarity functions and Estimated Mutual Information (EMI) are used…
In this paper we propose a general algorithmic framework for first-order methods in optimization in a broad sense, including minimization problems, saddle-point problems and variational inequalities. This framework allows to obtain many…
Optimizing non-convex functions is a fundamental challenge across machine learning and combinatorial optimization. We introduce and study $\gamma$-weakly $\theta$-up-concavity, a novel first-order condition that characterizes a broad class…
We introduce a weak version of the classical length function, termed the weak length function, defined on subsets of $R$-modules over a unital ring $R$, and further consider the concept of mean weak length for $R\Gamma$-modules associated…
We propose a new interpretation of measures of information and disorder by connecting these concepts to group theory in a new way. Entropy and group theory are connected here by their common relation to sets of permutations. A combinatorial…
In this work, we present a variety of novel information-theoretic generalization bounds for learning algorithms, from the supersample setting of Steinke & Zakynthinou (2020)-the setting of the "conditional mutual information" framework. Our…
Submodularity is a fundamental phenomenon in combinatorial optimization. Submodular functions occur in a variety of combinatorial settings such as coverage problems, cut problems, welfare maximization, and many more. Therefore, a lot of…
The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. In this paper, we establish a number of simple inequalities for the weighted entropies…
In a nutshell, submodular functions encode an intuitive notion of diminishing returns. As a result, submodularity appears in many important machine learning tasks such as feature selection and data summarization. Although there has been a…
In this paper we introduce a biparametric family of transformations which can be seen as an extension of the so-called up and down transformations. This new class of transformations allows to us to introduce new informational functionals,…
The purpose of this paper is to make a comprehensive connection between the basic results and properties derived from the two kinds of topologies (namely the $(\epsilon,\lambda)-$topology introduced by the author and the stronger locally…