Related papers: Guesswork, large deviations and Shannon entropy
From the output produced by a memoryless deletion channel from a uniformly random input of known length $n$, one obtains a posterior distribution on the channel input. The difference between the Shannon entropy of this distribution and that…
We prove the full equivalence between Assembly Theory (AT) and Shannon Entropy via a method based upon the principles of statistical compression renamed `assembly index' that belongs to the LZ family of popular compression algorithms (ZIP,…
It is shown that the standard expression for the information entropy, originally due to Shannon, is only valid for a particular set of states. For the general case of statistical mechanics, one needs to include an additional term in the…
The paper describes an approach to measuring convergence of an algorithm to its result in terms of an entropy-like function of partitions of its inputs of a given length. The goal is to look at the algorithmic data processing from the…
Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to…
Recently, a new type of set, named as random permutation set (RPS), is proposed by considering all the permutations of elements in a certain set. For measuring the uncertainty of RPS, the entropy of RPS is presented. However, the maximum…
Zipf's law, and power laws in general, have attracted and continue to attract considerable attention in a wide variety of disciplines - from astronomy to demographics to software structure to economics to linguistics to zoology, and even…
An algorithm is developed to gradually relax the Differential Privacy (DP) guarantee of a randomized response. The output from each relaxation maintains the same probability distribution as a standard randomized response with the equivalent…
Statistical properties of coupled dynamic-stochastic systems are studied within a combination of the maximum information principle and the superstatistical approach. The conditions at which the Shannon entropy functional leads to a…
We consider a simplified version of a solvable model by Mandal and Jarzynski, which constructively demonstrates the interplay between work extraction and the increase of the Shannon entropy of an information reservoir which is in contact…
We present quantitative relations between work and information that are valid both for finite sized and internally correlated systems as well in the thermodynamical limit. We suggest work extraction should be viewed as a game where the…
Feature selection, in the context of machine learning, is the process of separating the highly predictive feature from those that might be irrelevant or redundant. Information theory has been recognized as a useful concept for this task, as…
Measuring the complexity of tree structures can be beneficial in areas that use tree data structures for storage, communication, and processing purposes. This complexity can then be used to compress tree data structures to their…
Differential Privacy (DP) considers a scenario in which an adversary has almost complete information about the entries of a database. This worst-case assumption is likely to overestimate the privacy threat faced by an individual in…
A stochastic action principle for stochastic dynamics is revisited. We present first numerical diffusion experiments showing that the diffusion path probability depend exponentially on average Lagrangian action. This result is then used to…
Differential privacy is a notion of privacy that has become very popular in the database community. Roughly, the idea is that a randomized query mechanism provides sufficient privacy protection if the ratio between the probabilities of two…
We calculate Shannon information entropy of trapped interacting bosons in both the position and momentum spaces, $S_r$ and $S_k$ respectively. The total entropy maintains the fuctional form $S=a + b \ln N$ for repulsive bosons. At the…
We define an entropy based on a chosen governing probability distribution. If a certain kind of measurements follow such a distribution it also gives us a suitable scale to study it with. This scale will appear as a link function that is…
Quantifying the similarity between symbolic sequences is a traditional problem in Information Theory which requires comparing the frequencies of symbols in different sequences. In numerous modern applications, ranging from DNA over music to…
This work examines a novel question: how much randomness is needed to achieve local differential privacy (LDP)? A motivating scenario is providing {\em multiple levels of privacy} to multiple analysts, either for distribution or for…