English

IMU-1: Sample-Efficient Pre-training of Small Language Models

Machine Learning 2026-02-04 v1 Artificial Intelligence

Abstract

We present IMU-1, a 430M-parameter language model trained on 72B tokens that approaches the benchmark performance of models trained on 56x more data. We describe a validated training recipe combining recent architectural interventions (QK-norm attention, per-head gating, value residuals, LayerNorm scaling) with optimization advances (NorMuon with cautious weight decay, muP parametrization) and a three-stage training schedule with post-hoc checkpoint EMA. We provide ablations for each component and release code, weights and data to enable reproduction: https://huggingface.co/thepowerfuldeez/imu1_base

Keywords

Cite

@article{arxiv.2602.02522,
  title  = {IMU-1: Sample-Efficient Pre-training of Small Language Models},
  author = {George Grigorev},
  journal= {arXiv preprint arXiv:2602.02522},
  year   = {2026}
}

Comments

16 pages

R2 v1 2026-07-01T09:32:36.599Z