Related papers: Reinterpreting Maximum Entropy in Ecology: a null …
Food web topology and energy flow rates across food web linkages can influence ecosystem properties such as stability. Stability predictions from current models of energy flow are often sensitive to details in their formulation, and their…
An amended MaxEnt formulation for systems displaced from the conventional MaxEnt equilibrium is proposed. This formulation involves the minimization of the Kullback-Leibler divergence to a reference $Q$ (or maximization of Shannon…
Many biological systems perform close to their physical limits, but promoting this optimality to a general principle seems to require implausibly fine tuning of parameters. Using examples from a wide range of systems, we show that this…
The thermodynamic definition of entropy can be extended to nonequilibrium systems based on its relation to information. To apply this definition in practice requires access to the physical system's microstates, which may be prohibitively…
The ubiquity of machine learning (ML) and the demand for ever-larger models bring an increase in energy consumption and environmental impact. However, little is known about the energy scaling laws in ML, and existing research focuses on…
Using the concept of discrete noiseless channels, it was shown by Shannon in A Mathematical Theory of Communication that the ultimate performance of an encoder for a constrained system is limited by the combinatorial capacity of the system…
The metriplectic formalism is useful for describing complete dynamical systems which conserve energy and produce entropy. This creates challenges for model reduction, as the elimination of high-frequency information will generally not…
Here we deconstruct, and then in a reasoned way reconstruct, the concept of "entropy of a system," paying particular attention to where the randomness may be coming from. We start with the core concept of entropy as a COUNT associated with…
Extremal principles are fundamental in our interpretation of phenomena in nature. One of the best known examples is the second law of thermodynamics, governing most physical and chemical systems and stating the continuous increase of…
Understanding how network function constrains neural connectivity is a central challenge in neuroscience. An influential approach is to train neural networks with gradient descent on cognitive tasks and characterize the resulting…
We exhibit compelling evidence regarding how well does the MaxEnt principle describe the rank-distribution of city-populations via an exhaustive study of the 50 Spanish provinces (more than 8000 cities) in a time-window of 15 years…
The classical Maximum-Entropy Principle (MEP) based on Shannon entropy is widely used to construct least-biased probability distributions from partial information. However, the Shore-Johnson axioms that single out the Shannon functional…
This paper develops generalizations of empowerment to continuous states. Empowerment is a recently introduced information-theoretic quantity motivated by hypotheses about the efficiency of the sensorimotor loop in biological organisms, but…
The pervasive presence spatial and size structure in biological populations challenges fundamental assumptions at the heart of continuum models of population dynamics based on mean densities (local or global) only. Individual-based models…
Entropy is a central concept in physics, but can be challenging to calculate even for systems that are easily simulated. This is exacerbated out of equilibrium, where generally little is known about the distribution characterizing simulated…
Despite their claimed biological plausibility, most self organizing networks have strict topological constraints and consequently they cannot take into account a wide range of external stimuli. Furthermore their evolution is conditioned by…
This is a preliminary article stating and proving a new maximum entropy theorem. The entropies that we consider can be used as measures of biodiversity. In that context, the question is: for a given collection of species, which frequency…
Microbial ecosystems exhibit a surprising amount of functionally relevant diversity at all levels of taxonomic resolution, presenting a significant challenge for most modeling frameworks. A long-standing hope of theoretical ecology is that…
We study language models as evolving model organisms and ask when autoregressive next-token learning selects for world-tracking representations. For any encoding of latent world states, the Bayes-optimal next-token cross-entropy decomposes…
Understanding the assembly of ecosystems to estimate the number of species at different spatial scales is a challenging problem. Until now, maximum entropy approaches have lacked the important feature of considering space in an explicit…