Related papers: Mpemba Effect in Large-Language Model Training Dyn…

Recent work has identified a counterintuitive phenomenon termed "Hyperfitting", where fine-tuning Large Language Models (LLMs) to near-zero training loss on small datasets surprisingly enhances open-ended generation quality and mitigates…

Training large models is both resource-intensive and time-consuming, making it crucial to understand the quantitative relationship between model performance and hyperparameters. In this paper, we present an empirical law that describes how…

Recent analysis on the training dynamics of Transformers has unveiled an interesting characteristic: the training loss plateaus for a significant number of training steps, and then suddenly (and sharply) drops to near--optimal values. To…

Slow relaxation processes spanning widely separated timescales pose fundamental challenges for probing steady-state properties and engineering functional quantum systems, such as quantum heat engines and quantum computing devices. We…

The so-called Mpemba effect, i.e. the observation that the warmer of two otherwise identical systems cools faster when both are refrigerated in the same thermal reservoir, is a hotly debated topic in condensed mater physics and statistical…

The ability of Large Language Models (LLMs) to extract context from natural language problem descriptions naturally raises questions about their suitability in autonomous decision-making settings. This paper studies the behaviour of these…

Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining…

Under certain conditions, two samples of fluid at different initial temperatures present a counterintuitive behavior known as the Mpemba effect: it is the hotter system that cools sooner. Here, we show that the Mpemba effect is present in…

Schedule-Free Learning has shown promise as a practical anytime training method for machine learning, showing success across dozens of standard benchmark problems. However, strong performance for LLM training has only been demonstrated at…

Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fresh insights into LLM training dynamics. On the theoretical…

This study investigates the relationships which deep learning methods can identify between the input and output data. As a case study, rainfall-runoff modeling in a snow-dominated watershed by means of a long- and short-term memory (LSTM)…

The Mpemba effect has initially been noticed in macroscopic systems -- namely that hot water can freeze faster than cold water -- but recently its extension to open quantum systems has attracted significant attention. This phenomenon can be…

Increasing the number of parameters in language models is a common strategy to enhance their performance. However, smaller language models remain valuable due to their lower operational costs. Despite their advantages, smaller models…

Parameter-efficient fine-tuning (PEFT), particularly Low-Rank Adaptation (LoRA), adapts large language models (LLMs) by training only a small fraction of parameters. However, as the rank of the low-rank matrices used for adaptation…

Transferring the optimal learning rate from small to large neural networks can enable efficient training at scales where hyperparameter tuning is otherwise prohibitively expensive. To this end, the Maximal Update Parameterization (muP)…

The Mpemba effect and its inverse can be understood as a result of nonequilibrium thermodynamics. In polymers, changes of state are generally non-equilibrium processes. However, the Mpemba effect has been rarely reported in the…

Setting the learning rate (LR) for a deep learning model is a critical part of successful training. Choosing LRs is often done empirically with trial and error. In this work, we explore a solvable model of optimal LR schedules for a…

When a hot system cools down faster than an equivalent cold one, it exhibits the Mpemba Effect. This counterintuitive phenomenon was observed in several systems including water, magnetic alloys and polymers. In most experiments the system…

Training large language models (LLMs) typically involves pre-training on massive corpora, only to restart the process entirely when new data becomes available. A more efficient and resource-conserving approach would be continual…

Scaling laws have transformed our understanding of large language models by linking upstream metrics like cross-entropy loss to design factors such as model size, training data, and compute. However, these conventional laws fail to capture…