中文

实用标度定律:在数据受限世界中将计算转化为性能

机器学习 2026-05-12 v1

摘要

现代模型训练所依据的标度定律是在单一条件下校准的:数据丰富、单次预训练。该主导性标度定律形式Chinchilla的 L=E+A/Nα+B/DβL = E + A/N^\alpha + B/D^\beta 在此条件之外存在三项结构性局限:当唯一数据缩小时会发散,而非在无监督基线处饱和;它无法表示容量超过数据时的过拟合现象;以及它混合了总体例数与可用唯一例数。我们提出闭式扩展形式 L(N,D,T)=E+(L0E)h/(1+h)L(N, D, T) = E + (L_0 - E)\,h/(1+h),其中 h=a/Nα+b/Tβ+cNγ/Dδh = a/N^\alpha + b/T^\beta + c\,N^\gamma/D^\delta,将损失分解为欠容量、欠训练和过拟合三个组成部分。该形式在不可归约损失E与由损失类型决定的无监督基线L0之间饱和,并在数据丰富、单次预训练的极限情况下还原为Chinchilla形式。我们在四个多epoch实验中进行验证,涵盖四种架构家族(MLP、ResNet、傅里叶神经算子和变压器),跨视觉、科学ML和语言领域,并对五个已发布的LLM标度定律网格进行重新拟合。 extrapolating to higher compute and larger unique data than seen at fit time, our form achieves state-of-the-art RMSE on every published LLM grid we evaluate and on most cells of our constructed experiments. Once calibrated, the form admits a cost-aware allocation that recovers Chinchilla's optimum when data is free and shifts toward smaller corpora and more epochs as data grows expensive.

关键词

引用

@article{arxiv.2605.09189,
  title  = {Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World},
  author = {Christopher M. Bryant and Hao Liu},
  journal= {arXiv preprint arXiv:2605.09189},
  year   = {2026}
}