Related papers: Mpemba Effect in Large-Language Model Training Dyn…
Training language models currently requires pre-determining a fixed compute budget because the typical cosine learning rate schedule depends on the total number of steps. In contrast, the Warmup-Stable-Decay (WSD) schedule uses a constant…
Learning rate warm-up - increasing the learning rate at the beginning of training - has become a ubiquitous heuristic in modern deep learning, yet its theoretical foundations remain poorly understood. In this work, we provide a principled…
The Mpemba effect describes the situation in which a hot system cools faster than an identical copy that is initiated at a colder temperature. In many of the experimental observations of the effect, e.g. in water and clathrate hydrates, it…
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more efficient solution would be to enable the continual pre-training…
The counterintuitive Mpemba effect, wherein a hotter system cools faster, critically lacks a general macroscopic theory. Here, starting from linear irreversible thermodynamics, we formulate a generalized Newton's cooling law,…
The quantum Mpemba effect is a phenomenon characterized by an exponential relaxation from a non-equililbrium state to a steady state. This effect was predicted with an analysis of the Liouvillian superoperator and experimentally…
The Mpemba effect, in which a hotter system cools faster than a colder one, remains one of the most intriguing anomalies in thermodynamics. Here, we investigate its microscopic origin within the framework of quantum resource theories and…
Learning rate scheduling is essential in transformer training, where the final annealing plays a crucial role in getting the best performance. However, the mechanisms behind this cooldown phase, with its characteristic drop in loss, remain…
We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised fine-tuning (SFT). Decay-based learning rate schedulers are…
Deep Learning based Weather Prediction (DLWP) models have been improving rapidly over the last few years, surpassing state of the art numerical weather forecasts by significant margins. While much of the optimization effort is focused on…
Due to the scarcity of high-quality data, large language models (LLMs) are often trained on mixtures of data with varying quality levels, even after sophisticated data curation. A natural approach to better leverage high-quality data is…
The Warmup Stable Decay (WSD) learning rate scheduler has recently become popular, largely due to its good performance and flexibility when training large language models. It remains an open question whether the remarkable performance of…
Vanishing gradients and overfitting are central problems in machine learning, yet are typically analyzed in asymptotic regimes that obscure their dynamical origins. Here we provide a dynamical description of learning in multi-layer…
Most of our intuition about the behavior of physical systems is shaped by observations at or near thermal equilibrium. However, even a thermal quench can lead to states far from thermal equilibrium, where counterintuitive, anomalous effects…
Large-batch training has been essential in leveraging large-scale datasets and models in deep learning. While it is computationally beneficial to use large batch sizes, it often requires a specially designed learning rate (LR) schedule to…
It is common in deep learning to warm up the learning rate $\eta$, often by a linear schedule between $\eta_{\text{init}} = 0$ and a predetermined target $\eta_{\text{trgt}}$. In this paper, we show through systematic experiments using SGD…
We investigate the direct and inverse Mpemba effects within the framework of the time-delayed Newton's law of cooling by introducing and analyzing the Descartes protocol, a three-reservoir thermal scheme in which each sample undergoes a…
We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant schedule with linear cooldown; in particular,…
The classical Mpemba effect is the counterintuitive phenomenon where hotter water freezes faster than colder water due to the breakdown of Newton's law of cooling after a sudden temperature quench. The genuine nonequilibrium post-quench…
Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence. For this reason, the learning rate (LR) is often decreased when the size of a model is increased. Such a…