Related papers: Understanding and Designing Complex Systems: Respo…
We show how rate-distortion theory provides a mechanism for automated theory building by naturally distinguishing between regularity and randomness. We start from the simple principle that model variables should, as much as possible, render…
Computational mechanics, an approach to structural complexity, defines a process's causal states and gives a procedure for finding them. We show that the causal-state representation--an $\epsilon$-machine--is the minimal one consistent with…
We comment on some conceptual and and technical problems related to computational mechanics, point out some errors in several papers, and straighten out some wrong priority claims. We present explicitly the correct algorithm for…
Computational mechanics is a method for discovering, describing and quantifying patterns, using tools from statistical physics. It constructs optimal, minimal models of stochastic processes and their underlying causal structures. These…
We study the compression of data in the case where the useful information is contained in a set rather than a vector, i.e., the ordering of the data points is irrelevant and the number of data points is unknown. Our analysis is based on…
Rate distortion theory was developed for optimizing lossy compression of data, but it also has a lot of applications in statistics. In this paper we will see how rate distortion theory can be used to analyze a complicated data set involving…
Organisms have to keep track of the information in the environment that is relevant for adaptive behaviour. Transmitting information in an economical and efficient way becomes crucial for limited-resourced agents living in high-dimensional…
The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an…
When constructing models of the world, we aim for optimal compressions: models that include as few details as possible while remaining as accurate as possible. But which details -- or features measured in data -- should we choose to include…
To analyze high-dimensional systems, many fields in science and engineering rely on high-level descriptions, sometimes called "macrostates," "coarse-grainings," or "effective theories". Examples of such descriptions include the…
Classical rate-distortion theory requires knowledge of an elusive source distribution. Instead, we analyze rate-distortion properties of individual objects using the recently developed algorithmic rate-distortion theory. The latter is based…
Recently, it has been recognized that phase transitions play an important role in the probabilistic analysis of combinatorial optimization problems. However, there are in fact many other relations that lead to close ties between computer…
This survey articles focuses on emerging connections between the fields of machine learning and data compression. While fundamental limits of classical (lossy) data compression are established using rate-distortion theory, the connections…
The exploration of complex physical or technological processes usually requires exploiting available information from different sources: (i) physical laws often represented as a family of parameter dependent partial differential equations…
Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference. This inference can be made more efficient by partitioning the process across multiple devices, which, in turn,…
Computational mechanics quantifies structure in a stochastic process via its causal states, leading to the process's minimal, optimal predictor---the $\epsilon$-machine. We extend computational mechanics to communication channels between…
This essay advocates the view that any problem that has a meaningful empirical content, can be formulated in constructive, more definitely, finite terms. We consider combinatorial models of dynamical systems and approaches to statistical…
When predictive models are used to support complex and important decisions, the ability to explain a model's reasoning can increase trust, expose hidden biases, and reduce vulnerability to adversarial attacks. However, attempts at…
Finite abstractions are discrete approximations of dynamical systems, such that the set of abstraction trajectories contains all system trajectories. There is a consensus that abstractions suffer from the curse of dimensionality: for the…
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One…