相关论文: Dimensional Criticality at Grokking Across MLPs an…

Why does a Transformer that has memorized its training set wait thousands of steps before it generalizes? Existing accounts locate this delay in norm minimization, feature emergence, or the late discovery of sparse subnetworks. These…

Increasing the L2 regularization of Deep Neural Networks (DNNs) causes a first-order phase transition into the under-parametrized phase -- the so-called onset-of learning. We explain this transition via the scalar (Ricci) curvature of the…

Grokking, referring to the abrupt improvement in test accuracy after extended overfitting, offers valuable insights into the mechanisms of model generalization. Existing researches based on progress measures imply that grokking relies on…

A fast method is presented for simulating the dielectric-breakdown model using iterated conformal mappings. Numerical results for the dimension and for corrections to scaling are in good agreement with the recent RG prediction of an upper…

Out-Of-Distribution (OOD) generalization has gained increasing attentions for machine learning on graphs, as graph neural networks (GNNs) often exhibit performance degradation under distribution shifts. Existing graph OOD methods tend to…

Grokking is proposed and widely studied as an intricate phenomenon in which generalization is achieved after a long-lasting period of overfitting. In this work, we propose NeuralGrok, a novel gradient-based approach that learns an optimal…

While deep learning demonstrates its strong ability to handle independent and identically distributed (IID) data, it often suffers from out-of-distribution (OoD) generalization, where the test data come from another distribution (w.r.t. the…

The phenomenon of grokking in over-parameterized neural networks has garnered significant interest. It involves the neural network initially memorizing the training set with zero training error and near-random test error. Subsequent…

We present a theoretical explanation of the ``grokking'' phenomenon, where a model generalizes long after overfitting,for the originally-studied problem of modular addition. First, we show that early in gradient descent, when the ``kernel…

Our study connects the physics of disordered integer-dimensional systems and regular self-similar objects by studying spectral properties of fractal agglomerates with tunable dimension. The latter is controlled by parameter $\alpha$ of the…

Deep neural networks (DNNs) exhibit crackling-like avalanches whose origin lacks a mechanistic explanation. Here, I derive a stochastic theory of deep information propagation (DIP) by incorporating Central Limit Theorem (CLT)-level…

Grokking, the sudden generalization that occurs after prolonged overfitting, is a surprising phenomenon challenging our understanding of deep learning. Although significant progress has been made in understanding grokking, the reasons…

Grokking is the intriguing phenomenon where a model learns to generalize long after it has fit the training data. We show both analytically and numerically that grokking can surprisingly occur in linear networks performing linear tasks in a…

The vulcanization transition - the crosslink-density-controlled equilibrium phase transition from the liquid to the amorphous solid state - is explored analytically from a renormalization group perspective. The analysis centers on a minimal…

We study the dynamical large deviations (LD) of a class of one-dimensional kinetically constrained models whose (tilted) generators can be mapped into themselves via duality transformations. We consider four representative models in detail:…

Traditional machine learning methods heavily rely on the independent and identically distribution assumption, which imposes limitations when the test distribution deviates from the training distribution. To address this crucial issue,…

The coil-globule transition of an isolated polymer has been well established to be a second-order phase transition described by a standard tricritical O(0) field theory. We provide compelling evidence from Monte Carlo simulations in four…

Disorder is ubiquitous in solid-state systems, and its crucial influence on transport properties was revealed by the discovery of Anderson localization. Generally speaking, all bulk states will be exponentially localized in the strong…

Transition points mark qualitative changes in the macroscopic properties of large complex systems. Explosive transitions, exhibiting properties of both continuous and discontinuous phase transitions, have recently been uncovered in network…

We perform large-scale numerical simulations to investigate the critical behavior of $k$-core percolation in two dimensions with an extended interaction range $r$. By systematically varying both the core index $k$ and the interaction range…