Related papers: Entropy of a Zipfian Distributed Lexicon
The entropy of a hierarchical network topology in an ensemble of sparse random networks with "hidden variables" associated to its nodes, is the log-likelihood that a given network topology is present in the chosen ensemble.We obtain a…
Informational entropy is often identified as physical entropy. This is surprising because the two quantities are differently defined and furthermore the former is a subjective quantity while the latter is an objective one. We describe the…
This work explores entropy analysis as a tool for probing information distribution within Transformer-based architectures. By quantifying token-level uncertainty and examining entropy patterns across different stages of processing, we aim…
Physics concepts have often been borrowed and independently developed by other fields of science. In this perspective a significant example is that of entropy in Information Theory. The aim of this paper is to provide a short and…
The Renyi distribution ensuring the maximum of a Renyi entropy is investigated for a particular case of a power--law Hamiltonian. Both Lagrange parameters, $\alpha$ and $\beta$ can be excluded. It is found that $\beta$ does not depend on a…
We introduce the concept of inflation word entropy for random substitutions with a constant and primitive substitution matrix. Previous calculations of the topological entropy of such systems implicitly used this concept and established…
Zipf's law in its basic incarnation is an empirical probability distribution governing the frequency of usage of words in a language. As Terence Tao recently remarked, it still lacks a convincing and satisfactory mathematical explanation.…
We propose an entropy function for simplicial complices. Its value gives the expected cost of the optimal encoding of sequences of vertices of the complex, when any two vertices belonging to the same simplex are indistinguishable. We show…
Sentence is a basic linguistic unit, however, little is known about how information content is distributed across different positions of a sentence. Based on authentic language data of English, the present study calculated the entropy and…
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible constrained to match empirical data, for instance, feature expectations. We seek to generalize…
We derive a tight lower bound on equivocation (conditional entropy), or equivalently a tight upper bound on mutual information between a signal variable and channel outputs. The bound is in terms of the joint distribution of the signals and…
Aging is a universal consequence of life, yet researchers have identified no universal theme. This manuscript considers aging from the perspective of entropy, wherein things fall apart. We first examine biological information change as a…
It is often stated that if one is presented with a snapshot of the positions of the molecules of a glass and one of a liquid, one is unable to tell the difference. Here we argue instead that given several such snapshots taken over a…
Entropy functionals are computed for non-stationary distributions of particles of Lorentz gas and hard disks. The distributions consisting of beams of particles are found to have the largest amount of entropy and entropy increase. The…
In natural language processing, the entropy of a language is a measure of its unpredictability and complexity. The first study on this subject was conducted by Claude Shannon in 1951. By having participants predict the next character in a…
Zipf's law is a hallmark of several complex systems with a modular structure, such as books composed by words or genomes composed by genes. In these component systems, Zipf's law describes the empirical power law distribution of component…
In the first chapter of Shannon's "A Mathematical Theory of Communication," it is shown that the maximum entropy rate of an input process of a constrained system is limited by the combinatorial capacity of the system. Shannon considers…
This paper focuses on the problem of finding a distribution for an associated entropic vector in the entropy space nearest to a given, possibly non-entropic, target vector for random variables with a constraint on alphabet size. We show the…
Entropy and information provide natural measures of correlation among elements in a network. We construct here the information theoretic analog of connected correlation functions: irreducible $N$--point correlation is measured by a decrease…
Some general considerations on the notion of entropy in physics are presented. An attempt is made to clarify the question of the differentiation between physical entropy (the Clausius-Boltzmann one) and quantities called entropies…