Related papers: Mpemba Effect in Large-Language Model Training Dyn…
Large language models are powerful but costly. We ask whether meta-learning can make the pretraining of small language models not only better but also more interpretable. We integrate first-order MAML with subset-masked LM pretraining,…
We study optimal learning-rate schedules (LRSs) under the functional scaling law (FSL) framework introduced in Li et al. (2025), which accurately models the loss dynamics of both linear regression and large language model (LLM)…
An ancient and counterintuitive phenomenon know as the Mpemba effect (water can cool faster when initially heated up) showcases the critical role of initial conditions in relaxation processes. How to realize and utilize this effect for…
The Mpemba effect occurs when a system prepared at a hot temperature cools down faster to the bath temperature than an identical system starting at a warm temperature. We derive the condition for the Mpemba effect in the small-diffusion…
Pretraining large language models is a costly process. To make this process more efficient, several methods have been proposed to optimize model architecture/parametrization and hardware use. On the parametrization side, $\mu P$ (Maximal…
We investigate the Mpemba effect in a dilute sheared granular gas with a velocity-dependent restitution coefficient. Using kinetic theory based on Grad's moment method, we analyze the relaxation dynamics following a sudden change in the…
The Mpemba effect -- where hot systems cool faster than colder ones -- has intrigued both classical and quantum thermodynamics. As compared to classical systems, quantum systems add complexity due to quantum correlations. Recent works have…
Despite extensive research, the fundamental physical mechanisms underlying the Mpemba effect, a phenomenon where a substance cools faster after initially being heated, remain elusive. Although historically linked with water, the Mpemba…
Learning rate is one of the most important hyper-parameters that has a significant influence on neural network training. Learning rate schedules are widely used in real practice to adjust the learning rate according to pre-defined schedules…
Regional rainfall-runoff modeling is an old but still mostly out-standing problem in Hydrological Sciences. The problem currently is that traditional hydrological models degrade significantly in performance when calibrated for multiple…
The Mpemba effect is a counterintuitive physical phenomenon where a hot system cools faster than a warm one. In recent years, theoretical analyses of the Mpemba effect have been developed for microscopic systems and experimentally verified.…
The Mpemba effect is a thermodynamic anomaly in which a system farther away in temperature from equilibrium thermalizes before one that is initially closer. The effect has been experimentally observed across a wide range of systems,…
Robust hydrological simulation is key for sustainable development, water management strategies, and climate change adaptation. In recent years, deep learning methods have been demonstrated to outperform mechanistic models at the task of…
The Mpemba effect originally referred to the observation that, under certain thermalizing dynamics, initially hotter samples can cool faster than colder ones. This effect has since been generalized to other anomalous relaxation behaviors…
Under certain conditions, it takes a shorter time to cool a hot system than to cool the same system initiated at a lower temperature. This phenomenon - the "Mpemba Effect" - is well known in water, and has recently been observed in other…
We study a one-dimensional quasiperiodic tight-binding model with simultaneous off-diagonal (hopping) and diagonal (onsite) modulations. Using the inverse participation ratio and the wave-packet centroid, we construct…
We study adaptive learning rate scheduling for norm-constrained optimizers (e.g., Muon and Lion). We introduce a generalized smoothness assumption under which local curvature decreases with the suboptimality gap and empirically verify that…
The Mpemba effect is a fingerprint of the anomalous relaxation phenomenon wherein an initially hotter system equilibrates faster than an initially colder system when both are quenched to the same low temperature. Experiments on a single…
Temperature is a crucial hyperparameter in large language models (LLMs), controlling the trade-off between exploration and exploitation during text generation. High temperatures encourage diverse but noisy outputs, while low temperatures…
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant…