Related papers: Entropy in Science of Science
Shannon information entropy is a natural measure of probability (de)localization and thus (un)predictability in various procedures of data analysis for model systems. We pay particular attention to links between the Shannon entropy and the…
We study how the Shannon entropy of sequences produced by an information source converges to the source's entropy rate. We synthesize several phenomenological approaches to applying information theoretic measures of randomness and memory to…
Shannon and Renyi entropies are quantitative measures of uncertainty in a data set. They are developed by Renyi in the context of entropy theory. These measures have been studied in the case of the multivariate t-distributions. We extend…
We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the…
A new method is proposed for analyzing complexity and studying the information in random geometric networks using Tsallis entropy tool. Tsallis entropy of the ensemble of random geometric networks is calculated based on the components of…
People employ their knowledge to recognize things. This paper is concerned with how to measure people's knowledge for recognition and how it changes. The discussion is based on three assumptions. Firstly, we construct two evolution process…
Information entropy has been proved to be an effective tool to quantify the structural importance of complex networks. In the previous work (Xu et al, 2016 \cite{xu2016}), we measure the contribution of a path in link prediction with…
In the analysis of any type of system, granting maximum information extraction from its data is non-trivial. Confidence in successful information extraction typically builds on prior knowledge of the studied system or on the user's…
We develop a general formalism for representing and understanding structure in complex systems. In our view, structure is the totality of relationships among a system's components, and these relationships can be quantified using information…
Entanglement measures based on a logarithmic functional form naturally emerge in any attempt to quantify the degree of entanglement in the state of a multipartite quantum system. These measures can be regarded as generalizations of the…
Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order patterns with distinct properties across scales. Investigating how…
The tension between qualitative theorizing and quantitative methods is pervasive in the social sciences, and poses a constant challenge to empirical research. But in science studies as an interdisciplinary specialty, there are additional…
Systematic generalization remains challenging for current language models, which are known to be both sensitive to semantically similar permutations of the input and to struggle with known concepts presented in novel contexts. Although…
The rapid scaling of artificial intelligence models has revealed a fundamental tension between model capacity (storage) and inference efficiency (computation). While classical information theory focuses on transmission and storage limits,…
Shannon's entropy and other entropy-based concepts are derived from the new, more general concept of relative divergence of one "grading' function on a linearly ordered set from another such function. The definition of relative divergence…
We explore the relation between entanglement entropy of quantum many body systems and the distribution of corresponding, properly selected, observables. Such a relation is necessary to actually measure the entanglement entropy. We show that…
Information theory provides principled ways to analyze different inference and learning problems such as hypothesis testing, clustering, dimensionality reduction, classification, among others. However, the use of information theoretic…
The formalism of statistical mechanics can be generalized by starting from more general measures of information than the Shannon entropy and maximizing those subject to suitable constraints. We discuss some of the most important examples of…
Data stream mining problem has caused widely concerns in the area of machine learning and data mining. In some recent studies, ensemble classification has been widely used in concept drift detection, however, most of them regard…
The Shannon entropy, and related quantities such as mutual information, can be used to quantify uncertainty and relevance. However, in practice, it can be difficult to compute these quantities for arbitrary probability distributions,…