Related papers: Entropy of a Zipfian Distributed Lexicon

Although an intimate relation between entropy and diffusion has been advocated for many years and even seems to have been verified in theory and experiments, a quantitatively reliable study, and any derivation of an algebraic relation…

This paper introduces an objective metric for evaluating a parsing scheme. It is based on Shannon's original work with letter sequences, which can be extended to part-of-speech tag sequences. It is shown that this regular language is an…

We evaluated the impact of changing the observation scale over the entropy measures for text descriptions. MIDI coded Music, computer code and two human natural languages were studied at the scale of characters, words, and at the…

Entropy rate of sequential data-streams naturally quantifies the complexity of the generative process. Thus entropy rate fluctuations could be used as a tool to recognize dynamical perturbations in signal sources, and could potentially be…

In this paper we derive the maximum entropy characteristics of a particular rank order distribution, namely the discrete generalized beta distribution, which has recently been observed to be extremely useful in modelling many several…

Here we deconstruct, and then in a reasoned way reconstruct, the concept of "entropy of a system," paying particular attention to where the randomness may be coming from. We start with the core concept of entropy as a COUNT associated with…

The channel output entropy of a transmitted word is the entropy of the possible channel outputs and similarly, the input entropy of a received word is the entropy of all possible transmitted words. The goal of this work is to study these…

We consider shift spaces in which elements of the alphabet may overlap nontransitively. We define a notion of entropy for such spaces, give several techniques for computing lower bounds for it, and show that it is equal to a limit of…

Mixture distributions are a workhorse model for multimodal data in information theory, signal processing, and machine learning. Yet even when each component density is simple, the differential entropy of the mixture is notoriously hard to…

Entropy is a very useful concept from physics that tries to explain how a system behaves from a point of view of the thermodynamics. However, there are two ways to explain entropy, and it depends on if we are studying a microsystem or a…

This paper addresses the problem of determining the distance between two regular languages. It will show how to expand Jaccard distance, which works on finite sets, to potentially-infinite regular languages. The entropy of a regular…

We present an impossibility result, called a theorem about facts and words, which pertains to a general communication system. The theorem states that the number of distinct words used in a finite text is roughly greater than the number of…

Dissipative quantum systems are frequently described within the framework of the so-called "system-plus-reservoir" approach. In this work we assign their description to the Maximum Entropy Formalism and compare the resulting thermodynamic…

There are three ways to conceptualize entropy: entropy as an extensive thermodynamic quantity of physical systems (Clausius, Boltzmann, Gibbs), entropy as a measure for information production of ergodic sources (Shannon), and entropy as a…

We formulate a stochastic process, FiLex, as a mathematical model of lexicon entropy in deep learning-based emergent language systems. Defining a model mathematically allows it to generate clear predictions which can be directly and…

An efficient approach to the calculation of the $\epsilon$-entropy is proposed. The method is based on the idea of looking at the information content of a string of data, by analyzing the signal only at the instants when the fluctuations…

Entropy rate is a real valued functional on the space of discrete random sources which lacks a closed formula even for subclasses of sources which have intuitive parameterizations. A good way to overcome this problem is to examine its…

The weak law of large numbers implies that, under mild assumptions on the source, the Renyi entropy per produced symbol converges (in probability) towards the Shannon entropy rate. This paper quantifies the speed of this convergence for…

Entropy serves as a central observable in equilibrium thermodynamics. However, many biological and ecological systems operate far from thermal equilibrium. Here we show that entropy production can characterize the behavior of such…

The entropy of network ensembles characterizes the amount of information encoded in the network structure, and can be used to quantify network complexity, and the relevance of given structural properties observed in real network datasets…