Related papers: Directed Information on Abstract Spaces: Propertie…
We introduce \textbf{Directed Information $\gamma$-covering}, a simple but general framework for redundancy-aware context engineering. Directed information (DI), a causal analogue of mutual information, measures asymmetric predictiveness…
We study the decentralized consensus and stochastic optimization problems with compressed communications over static directed graphs. We propose an iterative gradient-based algorithm that compresses messages according to a desired…
Traditional channel capacity based on the discrete spatial dimensions mismatches the continuous electromagnetic fields. For the wireless communication system in a limited region, the spatial discretization may results in information loss…
Mutual information is widely used, in a descriptive way, to measure the stochastic dependence of categorical random variables. In order to address questions such as the reliability of the descriptive value, one must consider…
This paper is devoted to the mathematical study of some divergences based on the mutual information well-suited to categorical random vectors. These divergences are generalizations of the "entropy distance" and "information distance". Their…
In many areas of applied mathematics, engineering, and social and natural sciences, decentralization of information is a key aspect determining how to approach a problem. In this review article, we study information structures in a…
We study data processing inequalities that are derived from a certain class of generalized information measures, where a series of convex functions and multiplicative likelihood ratios are nested alternately. While these information…
Capacity analysis for channels with side information at the receiver has been an active area of interest. This problem is well investigated for the case of finite alphabet channels. However, the results are not easily generalizable to the…
We study the asymptotic properties of distributed consensus algorithms over switching directed random networks. More specifically, we focus on consensus algorithms over independent and identically distributed, directed random graphs, where…
Probabilistic representation spaces convey information about a dataset and are shaped by factors such as the training data, network architecture, and loss function. Comparing the information content of such spaces is crucial for…
This article reports an algorithm for multi-agent distributed optimization problems with a common decision variable, local linear equality and inequality constraints and set constraints with convergence rate guarantees.…
In information theory, the link between continuous information and discrete information is established through well-known sampling theorems. Sampling theory explains, for example, how frequency-filtered music signals are reconstructible…
We examine a class of deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical…
The mutual information is bounded from above by a decreasing affine function of the square of the distance between the input distribution and the set of all capacity-achieving input distributions $\Pi_{\mathcal{A}}$, on small enough…
Researchers from different areas have independently defined extensions of the usual weak convergence of laws of stochastic processes with the goal of adequately accounting for the flow of information. Natural approaches are convergence of…
What does the informational complexity of dynamical networked systems tell us about intrinsic mechanisms and functions of these complex systems? Recent complexity measures such as integrated information have sought to operationalize this…
We investigate notions of ambiguity and partial information in categorical distributional models of natural language. Probabilistic ambiguity has previously been studied using Selinger's CPM construction. This construction works well for…
In this work we focus our attention on distributed optimization problems in the context where the communication time between the server and the workers is non-negligible. We obtain novel methods supporting bidirectional compression (both…
Considering the constrained stochastic optimization problem over a time-varying random network, where the agents are to collectively minimize a sum of objective functions subject to a common constraint set, we investigate asymptotic…
Directed information (DI) is an information measure that attempts to capture directionality in the flow of information from one random process to another. It is closely related to other causal influence measures, such as transfer entropy,…