Related papers: Relating description complexity to entropy
We introduce a cumulant-expansion framework for quantifying how large language models (LLMs) internalize higher-order statistical structure during next-token prediction. By treating the softmax entropy of each layer's logit distribution as…
Castelnuovo-Mumford regularity and any extended degree function can be thought of as complexity measures for the structure of finitely generated graded modules. A recent result of Doering, Gunston, Vasconcelos shows that both can be…
The main result is that: function descriptions are not made equal, and they can be categorised in at least two categories using various computational methods for function evaluation. The result affects Kolmogorov complexity and Random…
A theorem of A.A. Brudno says that the Kolmogorov-Sinai entropy of a subshift X over $\mathbb{N}$ with respect to an ergodic measure $\mu$ equals the asymptotic Kolmogorov complexity of almost every word $\omega$ in X. The purpose of this…
Ontology, and more broadly, Knowledge Graph Matching is a challenging task in which expressiveness has not been fully addressed. Despite the increasing use of embeddings and language models for this task, approaches for generating…
We provide an ultimately fine-grained analysis of the data complexity and rewritability of ontology-mediated queries (OMQs) based on an EL ontology and a conjunctive query (CQ). Our main results are that every such OMQ is in AC0,…
Entropy principles based on thermodynamic consistency requirements are widely used for constitutive modeling in continuum mechanics, providing physical constraints on a priori unknown constitutive functions. The well-known M\"uller-Liu…
In a recent paper Andrei N. Soklakov explained the foundations of the Lagrangian formulation of classical particle mechanics by means of Kolmogorov complexity. In the present paper we use some of Soklakov ideas in order to derive the second…
We survey diverse approaches to the notion of information: from Shannon entropy to Kolmogorov complexity. Two of the main applications of Kolmogorov complexity are presented: randomness and classification. The survey is divided in two parts…
Descriptive complexity theory aims at inferring a problem's computational complexity from the syntactic complexity of its description. A cornerstone of this theory is Fagin's Theorem, by which a graph property is expressible in existential…
Evaluating large language models across many benchmarks is expensive, yet many benchmarks are highly correlated. We formalize the selection of a small, informative subset as submodular maximization under a multivariate Gaussian model.…
The complexity of a system description is a function of the entropy of its symbolic description. Prior to computing the entropy of the system description, an observation scale has to be assumed. In natural language texts, typical scales are…
A practical measure for the complexity of sequences of symbols (``strings'') is introduced that is rooted in automata theory but avoids the problems of Kolmogorov-Chaitin complexity. This physical complexity can be estimated for ensembles…
We analyze the problem of quantifying entanglement in pure and mixed states of fermionic systems with fixed number parity yet not necessarily fixed particle number. The "mode entanglement" between one single-particle level and its…
Increased availability of open-source software repositories and recent advances in code analysis using large language models (LLMs) has triggered a wave of new work to automate software engineering tasks that were previously very difficult…
The ability of Large Language Models (LLMs) to perform reasoning tasks such as deduction has been widely investigated in recent years. Yet, their capacity to generate proofs-faithful, human-readable explanations of why conclusions…
We propose modal Markov logic as an extension of propositional Markov logic to reason under the principle of maximum entropy for modal logics K45, KD45, and S5. Analogous to propositional Markov logic, the knowledge base consists of…
Ontologies often require knowledge representation on multiple levels of abstraction, but description logics (DLs) are not well-equipped for supporting this. We propose an extension of DLs in which abstraction levels are first-class citizens…
It is well known that normality can be described as incompressibility via finite automata. Still the statement and the proof of this result as given by Becher and Heiber (2013) in terms of "lossless finite-state compressors" do not follow…
Human cognition exhibits systematic compositionality, the algebraic ability to generate infinite novel combinations from finite learned components, which is the key to understanding and reasoning about complex logic. In this work, we…