Related papers: Teacher-student learning for a binary perceptron w…
The ability of a brain or a neural network to efficiently learn depends crucially on both the task structure and the learning rule. Previous works have analyzed the dynamical equations describing learning in the relatively simplified…
While classical in many theoretical settings - and in particular in statistical physics-inspired works - the assumption of Gaussian i.i.d. input data is often perceived as a strong limitation in the context of statistics and machine…
This article traces the development of fluctuation theory and its deep connection to irreversibility, from equilibrium to near-equilibrium, and finally to far-from-equilibrium systems. Classical fluctuation theorems, which capture the…
Conventional ensemble learning combines students in the space domain. On the other hand, in this paper we combine students in the time domain and call it time domain ensemble learning. In this paper, we analyze the generalization…
In this work we study generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes. We experimentally demonstrate that as meta-training progresses, the meta-test solutions,…
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data. Here we study the generalisation dynamics of two-layer neural networks in a…
The Hopfield model in a transverse field is investigated in order to clarify how quantum fluctuations affect the macroscopic behavior of neural networks. Using the Trotter decomposition and the replica method, we find that the $\alpha$ (the…
Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model complexity. However, in most applications the compressed…
Weak-to-strong generalization is a phenomenon in post-training whereby a strong student model, when finetuned solely with feedback from a weaker teacher, can not only surpass the teacher, but can improve upon its own capabilities. Recent…
We study the decrease of fluctuations of diagonal matrix elements of observables and of Husimi densities of quantum mechanical wave functions around their mean value upon approaching the semi-classical regime ($\hbar \rightarrow 0$). The…
Restricted Boltzmann Machines (RBMs) are generative models designed to learn from data with a rich underlying structure. In this work, we explore a teacher-student setting where a student RBM learns from examples generated by a teacher RBM,…
Model distillation has been a popular method for producing interpretable machine learning. It uses an interpretable "student" model to mimic the predictions made by the black box "teacher" model. However, when the student model is sensitive…
Imitation learning methods seek to learn from an expert either through behavioral cloning (BC) of the policy or inverse reinforcement learning (IRL) of the reward. Such methods enable agents to learn complex tasks from humans that are…
In ensemble teacher learning, ensemble teachers have only uncertain information about the true teacher, and this information is given by an ensemble consisting of an infinite number of ensemble teachers whose variety is sufficiently rich.…
We establish a way to handle main collective fluctuations in correlated quantum systems based on a Fluctuation Local Field concept. This technique goes beyond standard mean-field approaches, such as Hartree-Fock and dynamical mean-field…
RGB-based imitation learning requires many demonstrations to generalize to unseen objects or scenes, motivating research into intermediate representations to improve generalization for robotic manipulation. Visual foundation models enable…
We derive an extended fluctuation theorem for a geometric pumping in a spin-boson system under a periodic control of environmental temperatures by using a Markovian quantum master equation. We perform the Monte-Carlo simulation and obtain…
The use of advanced quantum neuron models for pattern recognition applications requires fault tolerance. Therefore, it is not yet possible to test such models on a large scale in currently available quantum processors. As an alternative, we…
Decoherence, resulting from unwanted interaction between a qubit and its environment, poses a serious challenge towards the development of quantum technologies. Recently, researchers have started analysing how real-time Hamiltonian learning…
Regularization can mitigate the generalization gap between training and inference by introducing inductive bias. Existing works have already proposed various inductive biases from diverse perspectives. However, none of them explores…