Apertus：面向全球语言环境的开放且合规大语言模型
计算与语言
2025-12-03 v2 人工智能
机器学习
摘要
我们提出 Apertus，这是一个完全开放的大型语言模型（LLM）套件，旨在解决今天开放模型生态中两个系统性短板：数据合规性和多语言表征。与许多先前模型在没有可重复数据管道或不顾内容所有权权利的情况下发布模型权重不同，Apertus 模型仅在开放可用的数据上进行预训练，后向遵守 `robots.txt` 排除，并过滤非授权、有毒和可识别个人信息的内容。为缓解记忆风险，我们采用 Goldfish 目标进行预训练，显著抑制数据逐字回顾，同时保留下游任务性能。Apertus 模型还扩展了多语言覆盖范围，在超过 1800 种语言上训练 15T token，其中约 40% 的预训练数据用于非英语内容。在 8B 和 70B 规模下，Apertus 在多语言基准测试中接近甚至超越了完全开放模型的最佳结果，与开源权重的对手相当。除了模型权重，我们随模型发布了包括数据准备脚本、检查点、评估套件和训练代码在内的所有科学制品，采用宽松许可证，使其能够透明地审计和扩展。
关键词
引用
@article{arxiv.2509.14233,
title = {Apertus: Democratizing Open and Compliant LLMs for Global Language Environments},
author = {Project Apertus and Alejandro Hernández-Cano and Alexander Hägele and Allen Hao Huang and Angelika Romanou and Antoni-Joan Solergibert and Barna Pasztor and Bettina Messmer and Dhia Garbaya and Eduard Frank Ďurech and Ido Hakimi and Juan García Giraldo and Mete Ismayilzada and Negar Foroutan and Skander Moalla and Tiancheng Chen and Vinko Sabolčec and Yixuan Xu and Michael Aerni and Badr AlKhamissi and Inés Altemir Mariñas and Mohammad Hossein Amani and Matin Ansaripour and Ilia Badanin and Harold Benoit and Emanuela Boros and Nicholas Browning and Fabian Bösch and Maximilian Böther and Niklas Canova and Camille Challier and Clement Charmillot and Jonathan Coles and Jan Deriu and Arnout Devos and Lukas Drescher and Daniil Dzenhaliou and Maud Ehrmann and Dongyang Fan and Simin Fan and Silin Gao and Miguel Gila and María Grandury and Diba Hashemi and Alexander Hoyle and Jiaming Jiang and Mark Klein and Andrei Kucharavy and Anastasiia Kucherenko and Frederike Lübeck and Roman Machacek and Theofilos Manitaras and Andreas Marfurt and Kyle Matoba and Simon Matrenok and Henrique Mendonça and Fawzi Roberto Mohamed and Syrielle Montariol and Luca Mouchel and Sven Najem-Meyer and Jingwei Ni and Gennaro Oliva and Matteo Pagliardini and Elia Palme and Andrei Panferov and Léo Paoletti and Marco Passerini and Ivan Pavlov and Auguste Poiroux and Kaustubh Ponkshe and Nathan Ranchin and Javi Rando and Mathieu Sauser and Jakhongir Saydaliev and Muhammad Ali Sayfiddinov and Marian Schneider and Stefano Schuppli and Marco Scialanga and Andrei Semenov and Kumar Shridhar and Raghav Singhal and Anna Sotnikova and Alexander Sternfeld and Ayush Kumar Tarun and Paul Teiletche and Jannis Vamvas and Xiaozhe Yao and Hao Zhao and Alexander Ilic and Ana Klimovic and Andreas Krause and Caglar Gulcehre and David Rosenthal and Elliott Ash and Florian Tramèr and Joost VandeVondele and Livio Veraldi and Martin Rajman and Thomas Schulthess and Torsten Hoefler and Antoine Bosselut and Martin Jaggi and Imanol Schlag},
journal= {arXiv preprint arXiv:2509.14233},
year = {2025}
}