相关论文: Dimensional Criticality at Grokking Across MLPs an…

A principled understanding of generalization in deep learning may require unifying disparate observations under a single conceptual framework. Previous work has studied \emph{grokking}, a training dynamic in which a sustained period of…

Grokking, or delayed generalization, is an intriguing learning phenomenon where test set loss decreases sharply only after a model's training set loss has converged. This challenges conventional understanding of the training dynamics in…

We study grokking, the onset of generalization long after overfitting, in a classical ridge regression setting. We prove end-to-end grokking results for learning over-parameterized linear regression models using gradient descent with weight…

Grokking -- the sudden generalisation that appears long after a model has perfectly memorised its training data -- has been widely observed but lacks a quantitative theory explaining the length of the delay. We show that grokking is a…

Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct…

Distribution shifts on graphs -- the discrepancies in data distribution between training and employing a graph machine learning model -- are ubiquitous and often unavoidable in real-world scenarios. These shifts may severely deteriorate…

Grokking, the unusual phenomenon for algorithmic datasets where generalization happens long after overfitting the training data, has remained elusive. We aim to understand grokking by analyzing the loss landscapes of neural networks,…

Networks are ubiquitous in diverse real-world systems. Many empirical networks grow as the number of nodes increases with time. Percolation transitions in growing random networks can be of infinite order. However, when the growth of large…

With large-scale Monte Carlo simulations, we investigate the nonsteady relaxation at the dynamic depinning transition in the two-dimensional Gaussian random-field Ising model. The dynamic scaling behavior is carefully analyzed, and the…

The transformation of the free-energy landscape from smooth to hierarchical is one of the richest features of mean-field disordered systems. A well-studied example is the de Almeida-Thouless transition for spin glasses in a magnetic field,…

We investigate the phenomenon of grokking -- delayed generalization accompanied by non-monotonic test loss behavior -- in a simple binary logistic classification task, for which "memorizing" and "generalizing" solutions can be strictly…

The dynamic properties of a classical tracer particle in a random, disordered medium are investigated close to the localization transition. For Lorentz models obeying Newtonian and diffusive motion at the microscale, we have performed…

We consider random walks evolving on two models of connected and undirected graphs and study the exact large deviations of a local dynamical observable. We prove, in the thermodynamic limit, that this observable undergoes a first-order…

We present a class of stochastic processes in which the large deviation functions of time-integrated observables exhibit singularities that relate to dynamical phase transitions of trajectories. These illustrative examples include Brownian…

We investigate a random--neighbours version of the two dimensional non-conserving earthquake model of Olami, Feder and Christensen [Phys. Rev. Lett. {\bf 68}, 1244 (1992)]. We show both analytically and numerically that criticality can be…

Mechanistic interpretability typically relies on post-hoc analysis of trained networks. We instead adopt an interventional approach: testing hypotheses a priori by modifying architectural topology to observe training dynamics. We study…

The Anderson transitions in a random magnetic field in three dimensions are investigated numerically. The critical behavior near the transition point is analyzed in detail by means of the transfer matrix method with high accuracy for…

Grokking refers to delayed generalization in which the increase in test accuracy of a neural network occurs appreciably after the improvement in training accuracy This paper introduces several practical metrics including variance under…

We study the percolation of strongly connected clusters (SCCs), in which sites are mutually reachable through directed paths, in systems with randomly oriented bonds by extensive simulations on hypercubic lattices from dimension $d=2$ to…

Grokking, a phenomenon where machine learning models generalize long after overfitting, has been primarily observed and studied in algorithmic tasks. This paper explores grokking in real-world datasets using deep neural networks for…