Related papers: Non-parametric Message Important Measure: Storage …
Neural networks (NNs) can achieved high performance in various fields such as computer vision, and natural language processing. However, deploying NNs in resource-constrained safety-critical systems has challenges due to uncertainty in the…
We define Persistent Mutual Information (PMI) as the Mutual (Shannon) Information between the past history of a system and its evolution significantly later in the future. This quantifies how much past observations enable long term…
It is not obvious how to extend Shannon's original information entropy to higher dimensions, and many different approaches have been tried. We replace the English text symbol sequence originally used to illustrate the theory by a discrete,…
The concept of attention, numerical weights that emphasize the importance of particular data, has proven to be very relevant in artificial intelligence. Relative entropy (RE, aka Kullback-Leibler divergence) plays a central role in…
Low-dimensional embedding, manifold learning, clustering, classification, and anomaly detection are among the most important problems in machine learning. The existing methods usually consider the case when each instance has a fixed,…
Task-oriented communication aims to extract and transmit task-relevant information to significantly reduce the communication overhead and transmission latency. However, the unpredictable distribution shifts between training and test data,…
Transfer entropy is used to establish a measure of causal relationships between two variables. Symbolic transfer entropy, as an estimation method for transfer entropy, is widely applied due to its robustness against non-stationarity. This…
Shannon entropy is the most crucial foundation of Information Theory, which has been proven to be effective in many fields such as communications. Renyi entropy and Chernoff information are other two popular measures of information with…
In communications, unknown variables are usually modelled as random variables, and concepts such as independence, entropy and information are defined in terms of the underlying probability distributions. In contrast, control theory often…
A new approach to data compression is developed and applied to multimedia content. This method separates messages into components suitable for both lossless coding and 'lossy' or statistical coding techniques, compressing complex objects by…
In the era of big data, it is necessary to split extremely large data sets across multiple computing nodes and construct estimators using the distributed data. When designing distributed estimators, it is desirable to minimize the amount of…
Semantic communication, regarded as the breakthrough beyond the Shannon paradigm, aims at the successful transmission of semantic information conveyed by the source rather than the accurate reception of each single symbol or bit regardless…
Communication in poor network environment is always a difficult problem, since troubles such as bit errors and packet loss may often occur. It is generally believed that it is impossible to transmit data both accurately and efficiently in…
In this Thesis, several results in quantum information theory are collected, most of which use entropy as the main mathematical tool. *While a direct generalization of the Shannon entropy to density matrices, the von Neumann entropy behaves…
Transformers, known for their attention mechanisms, have proven highly effective in focusing on critical elements within complex data. This feature can effectively be used to address the time-varying channels in wireless communication…
Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax…
The principles are elaborated which underlie the applications of general nonclassical states to communication and measurement systems. Relevant classical communication concepts are reviewed. Communication and measurement processes are…
It is known that given the real sum of two independent uniformly distributed lattice points from the same nested lattice codebook, the eavesdropper can obtain at most 1 bit of information per channel regarding the value of one of the…
Semantic communication is deemed as a revolution of Shannon's paradigm in the six-generation (6G) wireless networks. It aims at transmitting the extracted information rather than the original data, which receivers will try to recover.…
For a closed-loop control system with a digital channel between the sensor and the controller, the notion of invariance entropy quantifies the smallest average rate of information transmission above which a given compact subset of the state…