Related papers: Understanding and Designing Complex Systems: Respo…
We study the neural network (NN) compression problem, viewing the tension between the compression ratio and NN performance through the lens of rate-distortion theory. We choose a distortion metric that reflects the effect of NN compression…
Complexity science offers a wide range of measures for quantifying unpredictability, structure, and information. Yet, a systematic conceptual organization of these measures is still missing. We present a unified framework that locates…
We define the complexity of a continuous-time linear system to be the minimum number of bits required to describe its forward increments to a desired level of fidelity, and compute this quantity using the rate distortion function of a…
In this introductory article we present the basics of an approach to implementing computational interpreting of natural language aiming to model the meanings of words and phrases. Unlike other approaches, we attempt to define the meanings…
How can we compress language models without sacrificing accuracy? The number of compression algorithms for language models is rapidly growing to benefit from remarkable advances of recent language models without side effects due to the…
Predictive state representations (PSRs) offer an expressive framework for modelling partially observable systems. By compactly representing systems as functions of observable quantities, the PSR learning approach avoids using local-minima…
This paper is concerned with quantum data compression of asymptotically many independent and identically distributed copies of ensembles of mixed quantum states. The encoder has access to a side information system. The figure of merit is…
To cluster, classify and represent are three fundamental objectives of learning from high-dimensional data with intrinsic structure. To this end, this paper introduces three interpretable approaches, i.e., segmentation (clustering) via the…
The past century has seen a steady increase in the need of estimating and predicting complex systems and making (possibly critical) decisions with limited information. Although computers have made possible the numerical evaluation of…
The principle goal of computational mechanics is to define pattern and structure so that the organization of complex systems can be detected and quantified. Computational mechanics developed from efforts in the 1970s and early 1980s to…
Predictive statistical mechanics is a form of inference from available data, without additional assumptions, for predicting reproducible phenomena. By applying it to systems with Hamiltonian dynamics, a problem of predicting the macroscopic…
Rate-distortion theory provides bounds for compressing data produced by an information source to a specified encoding rate that is strictly less than the source's entropy. This necessarily entails some loss, or distortion, between the…
Modern applications require methods that are computationally feasible on large datasets but also preserve statistical efficiency. Frequently, these two concerns are seen as contradictory: approximation methods that enable computation are…
Computational models are quantitative representations of systems. By analyzing and comparing the outputs of such models, it is possible to gain a better understanding of the system itself. Though as the complexity of model outputs…
The purpose of this paper is to point out a new connection between information theory and dynamical systems. In the information theory side, we consider rate distortion theory, which studies lossy data compression of stochastic processes…
In this document, some general results in approximation theory and matrix analysis with applications to sparse identification of time series models and nonlinear discrete-time dynamical systems are presented. The aforementioned theoretical…
The project of physics discovery is often equivalent to finding the most concise description of a physical system. The description with optimum predictive capability for a dataset generated by a physical system is one that minimizes both…
In this chapter, I review the main methods and techniques of complex systems science. As a first step, I distinguish among the broad patterns which recur across complex systems, the topics complex systems science commonly studies, the tools…
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to…
Top-performing machine learning systems, such as deep neural networks, large ensembles and complex probabilistic graphical models, can be expensive to store, slow to evaluate and hard to integrate into larger systems. Ideally, we would like…