Related papers: Learning in Neural Networks Based on a Generalized…
Many decision-making algorithms draw inspiration from the inner workings of individual biological systems. However, it remains unclear whether collective behavior among biological species can also lead to solutions for computational tasks.…
Understanding how network function constrains neural connectivity is a central challenge in neuroscience. An influential approach is to train neural networks with gradient descent on cognitive tasks and characterize the resulting…
From the sampling of data to the initialisation of parameters, randomness is ubiquitous in modern Machine Learning practice. Understanding the statistical fluctuations engendered by the different sources of randomness in prediction is…
Sagawa and Ueda established a fluctuation theorem of information exchange by revealing the role of correlations in stochastic thermodynamics and unified the non-equilibrium thermodynamics of measurement and feedback control [T. Sagawa and…
Fluctuation theorems are a class of equalities that express universal properties of the probability distribution of a fluctuating path functional such as heat, work or entropy production over an ensemble of trajectories during a…
Adaptive networks are well-suited to perform decentralized information processing and optimization tasks and to model various types of self-organized and complex behavior encountered in nature. Adaptive networks consist of a collection of…
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output…
On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is…
Competitive interactions represent one of the driving forces behind evolution and natural selection in biological and sociological systems. For example, animals in an ecosystem may vie for food or mates; in a market economy, firms may…
Humans can often predict physical outcomes after only a few observations, a capability known as physical intuition. The mechanisms underlying this efficient learning remain elusive. Here, we introduce a variational learning framework in…
Federated learning is an emerging technique used to prevent the leakage of private information. Unlike centralized learning that needs to collect data from users and store them collectively on a cloud server, federated learning makes it…
To improve how neural networks function it is crucial to understand their learning process. The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to…
Humans communicate, receive, and store information using sequences of items -- from words in a sentence or notes in music to abstract concepts in lectures and books. The networks formed by these items (nodes) and the sequential transitions…
Networked dynamical systems, i.e., systems of dynamical units coupled via nontrivial interaction topologies, constitute models of broad classes of complex systems, ranging from gene regulatory and metabolic circuits in our cells to…
Modular neural networks outperform nonmodular neural networks on tasks ranging from visual question answering to robotics. These performance improvements are thought to be due to modular networks' superior ability to model the compositional…
Information-theoretic bounded rationality describes utility-optimizing decision-makers whose limited information-processing capabilities are formalized by information constraints. One of the consequences of bounded rationality is that…
We develop original models to study interacting agents in financial markets and in social networks. Within these models randomness is vital as a form of shock or news that decays with time. Agents learn from their observations and learning…
Information propagation characterizes how input correlations evolve across layers in deep neural networks. This framework has been well studied using mean-field theory, which assumes infinitely wide networks. However, these assumptions…
Learning is a complex dynamical process shaped by a range of interconnected decisions. Careful design of hyperparameter schedules for artificial neural networks or efficient allocation of cognitive resources by biological learners can…
We apply the theory of learning to physically renormalizable systems in an attempt to develop a theory of biological evolution, including the origin of life, as multilevel learning. We formulate seven fundamental principles of evolution…