Related papers: Guesswork, large deviations and Shannon entropy
The estimation of entropy rates for stationary discrete-valued stochastic processes is a well studied problem in information theory. However, estimating the entropy rate for stationary continuous-valued stochastic processes has not received…
It is known that given the real sum of two independent uniformly distributed lattice points from the same nested lattice codebook, the eavesdropper can obtain at most 1 bit of information per channel regarding the value of one of the…
Given a collection of strings, each with an associated probability of occurrence, the guesswork of each of them is their position in a list ordered from most likely to least likely, breaking ties arbitrarily. Guesswork is central to several…
We prove a Bernstein-type bound for the difference between the average of negative log-likelihoods of independent discrete random variables and the Shannon entropy, both defined on a countably infinite alphabet. The result holds for the…
The paper examines relationships between the conditional Shannon entropy and the expectation of $\ell_{\alpha}$-norm for joint probability distributions. More precisely, we investigate the tight bounds of the expectation of…
Security questions are one of the mechanisms used to recover passwords. Strong answers to security questions (i.e. high entropy) are hard for attackers to guess or obtain using social engineering techniques (e.g. monitoring of social…
Password guessers are instrumental for assessing the strength of passwords. Despite their diversity and abundance, little is known about how different guessers compare to each other. We perform in-depth analyses and comparisons of the…
The Random Permutation Set (RPS) is a new type of set proposed recently, which can be regarded as the generalization of evidence theory. To measure the uncertainty of RPS, the entropy of RPS and its corresponding maximum entropy have been…
We use large language models (LLMs) to uncover long-ranged structure in English texts from a variety of sources. The conditional entropy or code length in many cases continues to decrease with context length at least to $N\sim 10^4$…
We start with reviewing the origin of the idea that entropy and the Second Law are associated with the Arrow of Time. We then introduced a new definition of entropy based on Shannons Measure of Information, SMI. The SMI may be defined on…
Even if a probability distribution is properly normalizable, its associated Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze conditions under which this phenomenon can occur. Roughly speaking, this happens when…
In this paper, some general properties of Shannon information measures are investigated over sets of probability distributions with restricted marginals. Certain optimization problems associated with these functionals are shown to be…
Shannon entropy is widely used to quantify the uncertainty of discrete random variables. But when normalized to the unit interval, as is often done in practice, it no longer conveys the alphabet sizes of the random variables being studied.…
Data is the cornerstone of large language models (LLMs), but not all data is useful for model learning. Carefully selected data can better elicit the capabilities of LLMs with much less computational overhead. Most methods concentrate on…
The concept of entropy, firstly introduced in information theory, rapidly became popular in many applied sciences via Shannon's formula to measure the degree of heterogeneity among observations. A rather recent research field aims at…
In a previous paper, we proposed a probability interpretation for higher genus amplitudes of BMN (Berenstein-Maldacena-Nastase) strings in a pp-wave background with infinite negative curvature. This provides a natural definition of the…
We prove the large deviation principle for several entropy and cross entropy estimators based on return times and waiting times on shift spaces over finite alphabets. We consider shift-invariant probability measures satisfying some…
A common countermeasure against side-channel attacks on secret key cryptographic implementations is $d$th-order masking, which splits each sensitive variable into $d+1$ random shares. In this paper, maximal leakage bounds on the probability…
We investigate how to measure and define the entropy of a simple chaotic system, three hard spheres on a ring. A novel approach is presented, which does not assume the ergodic hypothesis. It consists of transforming the particles collision…
In this paper we show that the expected generalisation performance of a learning machine is determined by the distribution of risks or equivalently its logarithm -- a quantity we term the risk entropy -- and the fluctuations in a quantity…