Related papers: Dataset-learning duality and emergent criticality

In machine learning, there is a fundamental trade-off between ease of optimization and expressive power. Neural Networks, in particular, have enormous expressive power and yet are notoriously challenging to train. The nature of that…

The presence of both critical behavior and oscillating patterns in brain dynamics is a very interesting issue. In this paper, we consider a model for a neuron population, where each neuron is modeled by an over-damped rotator. We find that…

Recent studies have shown that high disparities in effective learning rates (ELRs) across layers in deep neural networks can negatively affect trainability. We formalize how these disparities evolve over time by modeling weight dynamics…

Gradient-based meta-learning algorithms have gained popularity for their ability to train models on new tasks using limited data. Empirical observations indicate that such algorithms are able to learn a shared representation across tasks,…

Trainable activation functions, whose parameters are optimized alongside network weights, offer increased expressivity compared to fixed activation functions. Specifically, trainable activation functions defined as ratios of polynomials…

Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient…

In recent years self organised critical neuronal models have provided insights regarding the origin of the experimentally observed avalanching behaviour of neuronal systems. It has been shown that dynamical synapses, as a form of short-term…

At the point of a second order phase transition also termed as a critical point, systems display long range order and their macroscopic behaviors are independent of the microscopic details making up the system. Due to these properties, it…

Lifelong learning capabilities are crucial for artificial autonomous agents operating on real-world data, which is typically non-stationary and temporally correlated. In this work, we demonstrate that dynamically grown networks outperform…

Neural plasticity is an important functionality of human brain, in which number of neurons and synapses can shrink or expand in response to stimuli throughout the span of life. We model this dynamic learning process as an $L_0$-norm…

This work concerns the control of unknown nonlinear systems corrupted by disturbances. For such systems, we propose an anti-disturbance dual control approach with active learning of the disturbances. Our approach holds the dual property of…

A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions between the large number of network parameters. Such non-trivial…

We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich…

Deciphering the underpinnings of the dynamical processes leading to information transmission, processing, and storing in the brain is a crucial challenge in neuroscience. An inspiring but speculative theoretical idea is that such dynamics…

Gradient-based algorithms are a cornerstone of artificial neural network training, yet it remains unclear whether biological neural networks use similar gradient-based strategies during learning. Experiments often discover a diversity of…

As a promising computational paradigm, occurrence of critical states in artificial and biological neural networks has attracted wide-spread attention. An often-made explicit or implicit assumption is that one single critical state is…

The rapid advances in artificial intelligence (AI) have largely been driven by scaling deep neural networks (DNNs) - increasing model size, data, and computational resources. Yet performance is ultimately governed by network dynamics. The…

Understanding how the brain learns to compute functions reliably, efficiently and robustly with noisy spiking activity is a fundamental challenge in neuroscience. Most sensory and motor tasks can be described as dynamical systems and could…

In continual learning problems, it is often necessary to overwrite components of a neural network's learned representation in response to changes in the data stream; however, neural networks often exhibit \primacy bias, whereby early…

The critical brain hypothesis posits that neural circuitry operates near criticality to reap the computational benefits of accessing a wide range of timescales. The theory of critical phenomena generally predicts heavy-tailed (power-law)…