How to explain grokking
Statistical Mechanics
2025-12-09 v4 Machine Learning
Authors:
S. V. Kozyrev
Abstract
Explanation of grokking (delayed generalization) in learning is given by modeling grokking by the stochastic gradient Langevin dynamics (Brownian motion) and applying the ideas of thermodynamics.
Cite
@article{arxiv.2412.18624,
title = {How to explain grokking},
author = {S. V. Kozyrev},
journal= {arXiv preprint arXiv:2412.18624},
year = {2025}
}
Comments
Discussion slightly improved
Related papers
View all related →
Machine Learning · Computer Science
Grokking in Linear Models for Logistic Regression
Nataraj Das, Atreya Vedantam, Chandrashekar Lakshminarayanan
2026-02-10
Machine Learning · Computer Science
Grokking Explained: A Statistical Phenomenon
Breno W. Carvalho, Artur S. d'Avila Garcez, Luís C. Lamb, Emílio Vital Brazil
2025-02-05
Machine Learning · Statistics
Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding
Noam Levi, Alon Beck, Yohai Bar-Sinai
2024-02-06
Disordered Systems and Neural Networks · Physics
Grokking in the Ising Model
Karolina Hutchison, David Yevick
2026-02-06
Machine Learning · Computer Science
Grokking Beyond the Euclidean Norm of Model Parameters
Pascal Jr Tikeng Notsawo, Guillaume Dumas, Guillaume Rabusseau
2025-07-14
Machine Learning · Computer Science
Grokfast: Accelerated Grokking by Amplifying Slow Gradients
Jaerin Lee, Bong Gyun Kang, Kihoon Kim, Kyoung Mu Lee
2024-06-06
Machine Learning · Statistics
Grokking at the Edge of Linear Separability
Alon Beck, Noam Levi, Yohai Bar-Sinai
2025-07-22
Machine Learning · Computer Science
The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold
Tiberiu Musat
2026-01-12
Machine Learning · Computer Science
GrokAlign: Geometric Characterisation and Acceleration of Grokking
Thomas Walker, Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk
2025-08-01
Artificial Intelligence · Computer Science
The Norm-Separation Delay Law of Grokking: A First-Principles Theory of Delayed Generalization
Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung, Phan Thanh Duc
2026-05-05
Machine Learning · Computer Science
Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Model
Zhiwei Xu, Zhiyu Ni, Yixin Wang, Wei Hu
2025-04-21
Machine Learning · Computer Science
Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition
Kenzo Clauw, Sebastiano Stramaglia, Daniele Marinazzo
2024-08-20
Statistical Mechanics · Physics
Grokking phase transitions in learning local rules with gradient descent
Bojan Žunkovič, Enej Ilievski
2022-10-28
Machine Learning · Computer Science
Mechanistic Insights into Grokking from the Embedding Layer
H. V. AlquBoj, Hilal AlQuabeh, Velibor Bojkovic, Munachiso Nwadike +1
2025-05-22
Machine Learning · Statistics
A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learning
Abdessamed Qchohi, Simone Rossi
2026-04-15
Machine Learning · Computer Science
Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
Ali Saheb Pasand, Elvis Dohmatob
2026-05-19
Machine Learning · Computer Science
Deep Networks Always Grok and Here is Why
Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk
2024-06-10
Machine Learning · Computer Science
Grokking of Diffusion Models: Case Study on Modular Addition
Joon Hyeok Kim, Yong-Hyun Park, Mattis Dalsætra Østby, Jiatao Gu
2026-04-21
Machine Learning · Computer Science
To Grok Grokking: Provable Grokking in Ridge Regression
Mingyue Xu, Gal Vardi, Itay Safran
2026-02-09
Machine Learning · Computer Science
Towards Understanding Grokking: An Effective Theory of Representation Learning
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud +2
2022-10-17
Machine Learning · Computer Science
Unifying Grokking and Double Descent
Xander Davies, Lauro Langosco, David Krueger
2023-03-14
Machine Learning · Computer Science
Omnigrok: Grokking Beyond Algorithmic Data
Ziming Liu, Eric J. Michaud, Max Tegmark
2023-03-24
Machine Learning · Computer Science
A rationale from frequency perspective for grokking in training neural network
Zhangchen Zhou, Yaoyu Zhang, Zhi-Qin John Xu
2024-05-29
Machine Learning · Computer Science
Exploring Grokking: Experimental and Mechanistic Investigations
Hu Qiye, Zhou Hao, Yu RuoXi
2024-12-17
Machine Learning · Computer Science
Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition
Mohamad Amin Mohamadi, Zhiyuan Li, Lei Wu, Danica J. Sutherland
2024-07-18