Related papers: Information theory and learning: a physical approa…
We critically analyse the point of view for which laws of nature are just a mean to compress data. Discussing some basic notions of dynamical systems and information theory, we show that the idea that the analysis of large amount of data by…
Previous theoretical results pertaining to meta-learning on sequences build on contrived assumptions and are somewhat convoluted. We introduce new information-theoretic tools that lead to an elegant and very general decomposition of error…
Recent years have seen significant activity on the problem of using data for the purpose of learning properties of quantum systems or of processing classical or quantum data via quantum computing. As in classical learning, quantum learning…
Information theory is gaining popularity as a tool to characterize performance of biological systems. However, information is commonly quantified without reference to whether or how a system could extract and use it; as a result,…
Open world Machine Learning (OWML) aims to develop intelligent systems capable of recognizing known categories, rejecting unknown samples, and continually learning from novel information. Despite significant progress in open set…
Automated decision making systems are increasingly being used in real-world applications. In these systems for the most part, the decision rules are derived by minimizing the training error on the available historical data. Therefore, if…
Synthetic data becomes crucial for large language model training, but its effectiveness is highly inconsistent. We provide an information-theoretic account of this inconsistency: synthetic data improves a model only when the…
We analyze the notion that physical theories are quantitative and testable by observations in experiments. This leads us to propose a new, Bayesian, interpretation of probabilities in physics that unifies their current use in classical…
Probability theory is fundamental for modeling uncertainty, with traditional probabilities being real and non-negative. Complex probability extends this concept by allowing complex-valued probabilities, opening new avenues for analysis in…
This Thesis explores how tools from Statistical Physics and Information Theory can help us describe and understand complex systems. In the first part, we study the interplay between internal interactions, environmental changes, and…
Eliciting information to reduce uncertainty about a latent entity is a critical task in many application domains, e.g., assessing individual student learning outcomes, diagnosing underlying diseases, or learning user preferences. Though…
Consider a predictor, a learner, whose input is a stream of discrete items. The predictor's task, at every time point, is probabilistic multiclass prediction, i.e. to predict which item may occur next by outputting zero or more candidate…
General regression and classification models are constructed as linear combinations of simple rules derived from the data. Each rule consists of a conjunction of a small number of simple statements concerning the values of individual input…
Information is a key concept in evolutionary biology. Information is stored in biological organism's genomes, and used to generate the organism as well as to maintain and control it. Information is also "that which evolves". When a…
While information theory has been introduced to characterize the fundamental limitations of control and filtering for a few decades, the existing information-theoretic methods are indirect and cumbersome for analyzing the limitations of…
Comparisons are made for the amount of agreement of the composite likelihood information criteria and their full likelihood counterparts when making decisions among the fits of different models, and some properties of penalty term for…
We review briefly the concepts underlying complex systems and probability distributions. The later are often taken as the first quantitative characteristics of complex systems, allowing one to detect the possible occurrence of regularities…
In this position paper, I first describe a new perspective on machine learning (ML) by four basic problems (or levels), namely, "What to learn?", "How to learn?", "What to evaluate?", and "What to adjust?". The paper stresses more on the…
Recent work analyzing in-context learning (ICL) has identified a broad set of strategies that describe model behavior in different experimental conditions. We aim to unify these findings by asking why a model learns these disparate…
Reinforcement learning systems are often concerned with balancing exploration of untested actions against exploitation of actions that are known to be good. The benefit of exploration can be estimated using the classical notion of Value of…