Related papers: Modelling informational entropy
The paper presents a lightweight discussion of relations between energy, entropy, information, and intelligence, based on an analysis of the energy needed for computation.
Knowledge and information are becoming the primary resources of the emerging information society. To exploit the potential of available expert knowledge, comprehension and application skills (i.e. expert competences) are necessary. The…
As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other…
In this study, the output of large language models (LLM) is considered an information source generating an unlimited sequence of symbols drawn from a finite alphabet. Given the probabilistic nature of modern LLMs, we assume a probabilistic…
Entropy or information is a fundamental quantity contained in a system in statistical mechanics and information theory. In this paper, a definition of classical information entropy of parton distribution functions is suggested. The…
The importance of explainability in machine learning continues to grow, as both neural-network architectures and the data they model become increasingly complex. Unique challenges arise when a model's input features become high dimensional:…
Current machine learning models are evaluated through behavioral snapshots, with benchmark accuracies, win rates and outcome-based metrics. Model explanations and evaluations, however, are fundamentally intertwined: understanding why a…
This work demonstrates how the concept of the entropic potential of events -- a parameter quantifying the influence of discrete events on the expected future entropy of a system -- can enhance uncertainty quantification, decision-making,…
For a computational system to be intelligent, it should be able to perform, at least, basic deductions. Nonetheless, since deductions are, in some sense, equivalent to tautologies, it seems that they do not provide new information. The…
There are many ways we can not know. Even in systems that we created ourselves, as, for example, systems in mathematical logic, Go\"edel and Tarski's theorems impose limits on what we can know. As we try to speak of the real world, things…
Many branches of theoretical and applied mathematics require a quantifiable notion of complexity. One such circumstance is a topological dynamical system - which involves a continuous self-map on a metric space. There are many notions of…
Society's capacity for algorithmic problem-solving has never been greater. Artificial Intelligence is now applied across more domains than ever, a consequence of powerful abstractions, abundant data, and accessible software. As capabilities…
Concept-based Models aim to improve interpretability by predicting high-level intermediate concepts, representing a promising approach for deployment in high-risk scenarios. However, they are known to suffer from information leakage,…
Entropy and information can be considered dual: entropy is a measure of the subspace defined by the information constraining the given ambient space. Negative entropies, arising in na\"ive extensions of the definition of entropy from…
One of the greatest research challenges of this century is to understand the neural basis for how behavior emerges in brain-body-environment systems. To this end, research has flourished along several directions but have predominantly…
Written language is complex. A written text can be considered an attempt to convey a meaningful message which ends up being constrained by language rules, context dependence and highly redundant in its use of resources. Despite all these…
People employ their knowledge to recognize things. This paper is concerned with how to measure people's knowledge for recognition and how it changes. The discussion is based on three assumptions. Firstly, we construct two evolution process…
Entropy is a central concept in physics, but can be challenging to calculate even for systems that are easily simulated. This is exacerbated out of equilibrium, where generally little is known about the distribution characterizing simulated…
Autoregressive language models achieve remarkable performance, yet a unified theory explaining their internal mechanisms, how training shapes representations, and why these representations support complex behavior remains incomplete. We…
In this work, we argue that ignorance can be inherently understood as a hyperintensional notion. When faced with two logically or necessarily equivalent propositions, an agent may be ignorant of one while not of the other. To capture…