Related papers: Analyzing Training Using Phase Transitions in Entr…
Persistent entropy (PE) is an information-theoretic summary statistic of persistence barcodes that has been widely used to detect regime changes in complex systems. Despite its empirical success, a general theoretical understanding of when…
In this paper, we utilize information theory to study the fundamental performance limitations of generic feedback systems, where both the controller and the plant may be any causal functions/mappings while the disturbance can be with any…
Given two distinct subsets $A,B$ in the state space of some dynamical system, Transition Path Theory (TPT) was successfully used to describe the statistical behavior of transitions from $A$ to $B$ in the ergodic limit of the stationary…
This tutorial investigates the convergence of statistical mechanics and learning theory, elucidating the potential enhancements in machine learning methodologies through the integration of foundational principles from physics. The tutorial…
We define an entropy for a quantum field theory by combining quantum fluctuations, scaling and the maximum entropy concept. This entropy has different behavior in asymptotically free and non--asymptotically free theories. We find that the…
Non-stationary sequences arise naturally in control, forecasting, and decision-making. The data-generating process shifts at unknown times, and models must detect the change, discard or downweight obsolete evidence, and adapt to new…
Maximum entropy models are considered by many to be one of the most promising avenues of language modeling research. Unfortunately, long training times make maximum entropy research difficult. We present a novel speedup technique: we change…
Within the task of collaborative filtering two challenges for computing conditional probabilities exist. First, the amount of training data available is typically sparse with respect to the size of the domain. Thus, support for higher-order…
We demonstrate that absorbing phase transitions in one dimension may be induced by the dynamics of a single site. As an example we consider a one-dimensional model of diffusing particles, where a single site at the boundary evolves…
In order to gain a deeper understanding of complex systems and infer key information using minimal data, I classify all configurations based on classical probability, starting from the dimensions of energy and different categories of…
Conditional differential entropy provides an intuitive measure for relatively ranking time-series complexity by quantifying uncertainty in future observations given past context. However, its direct computation for high-dimensional…
The reliable prediction of the temporal behavior of complex systems is key in numerous scientific fields. This strong interest is however hindered by modeling issues: often, the governing equations describing the physics of the system under…
We study the temporal rate of variations of the von Neumann entropy in an open quantum system which interacts with a bath. We show that for almost all initial states of the bath and the system, the time-average of the rate of entropy change…
Maximum-entropy ensembles are key primitives in statistical mechanics from which thermodynamic properties can be derived. Over the decades, several approaches have been put forward in order to justify from minimal assumptions the use of…
Temporal point processes are the dominant paradigm for modeling sequences of events happening at irregular intervals. The standard way of learning in such models is by estimating the conditional intensity function. However, parameterizing…
Tuning parameters are parameters involved in an estimating procedure for the purpose of reducing the risk of some other estimator. Examples include the degree of penalization in penalized regression and likelihood problems, as well as the…
Transfer entropy is used to establish a measure of causal relationships between two variables. Symbolic transfer entropy, as an estimation method for transfer entropy, is widely applied due to its robustness against non-stationarity. This…
Making decisions freely presupposes that there is some indeterminacy in the environment and in the decision making engine. The former is reflected on the behavioral changes due to communicating: few changes indicate rigid environments;…
In this paper we introduce the concept of random time changes in dynamical systems. The subordination principle may be applied to study the long time behavior of the random time systems. We show, under certain assumptions on the class of…
In this paper, we obtain fundamental $\mathcal{L}_{p}$ bounds in sequential prediction and recursive algorithms via an entropic analysis. Both classes of problems are examined by investigating the underlying entropic relationships of the…