实用标度定律:在数据受限世界中将计算转化为性能
机器学习
2026-05-12 v1
摘要
现代模型训练所依据的标度定律是在单一条件下校准的:数据丰富、单次预训练。该主导性标度定律形式Chinchilla的 在此条件之外存在三项结构性局限:当唯一数据缩小时会发散,而非在无监督基线处饱和;它无法表示容量超过数据时的过拟合现象;以及它混合了总体例数与可用唯一例数。我们提出闭式扩展形式 ,其中 ,将损失分解为欠容量、欠训练和过拟合三个组成部分。该形式在不可归约损失E与由损失类型决定的无监督基线L0之间饱和,并在数据丰富、单次预训练的极限情况下还原为Chinchilla形式。我们在四个多epoch实验中进行验证,涵盖四种架构家族(MLP、ResNet、傅里叶神经算子和变压器),跨视觉、科学ML和语言领域,并对五个已发布的LLM标度定律网格进行重新拟合。 extrapolating to higher compute and larger unique data than seen at fit time, our form achieves state-of-the-art RMSE on every published LLM grid we evaluate and on most cells of our constructed experiments. Once calibrated, the form admits a cost-aware allocation that recovers Chinchilla's optimum when data is free and shifts toward smaller corpora and more epochs as data grows expensive.
引用
@article{arxiv.2605.09189,
title = {Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World},
author = {Christopher M. Bryant and Hao Liu},
journal= {arXiv preprint arXiv:2605.09189},
year = {2026}
}