holistic capability preservation: 面向紧凑且全面推理模型的探索
摘要
本技术报告介绍了 Ring-Lite-Distill，这是一个源自我们开源混合专家 (Mixture-of-Experts, MoE) 大型语言模型 (LLM) Ling-Lite 的轻量级推理模型。该研究表明，通过严谨的高质量数据精选和巧妙的训练范式，紧凑型 MoE 模型 Ling-Lite 能够进一步训练以实现卓越的推理能力，同时保持其参数高效的架构，仅激活 275 亿参数，构建出一种高效的轻量级推理架构。特别地，在构建该模型时，我们不仅聚焦于提升高级推理能力（例如高难度数学问题求解），更致力于开发具备更全面Competency覆盖范围的推理模型。我们的做法确保了在不同难度水平的推理任务中都能获得涵盖，同时保持通用能力，如指令遵循、工具使用和知识保留。我们展示了，Ring-Lite-Distill 的推理能力达到与 DeepSeek-R1-Distill-Qwen-7B 相当，其通用能力显著超越 DeepSeek-R1-Distill-Qwen-7B。该模型可在 https://huggingface.co/inclusionAI 访问获取。
引用
@article{arxiv.2504.07158,
title = {Holistic Capability Preservation: Towards Compact Yet Comprehensive Reasoning Models},
author = {Ling Team and Caizhi Tang and Chilin Fu and Chunwei Wu and Jia Guo and Jianwen Wang and Jingyu Hu and Liang Jiang and Meng Li and Peng Jiao and Pingping Liu and Shaomian Zheng and Shiwei Liang and Shuaicheng Li and Yalin Zhang and Yingting Wu and Yongkang Liu and Zhenyu Huang},
journal= {arXiv preprint arXiv:2504.07158},
year = {2025}
}
备注
Based on the further discussion of the working group, the current version is deemed unsuitable for release. We are currently undertaking further work that is expected to involve significant revisions, but this process will require some additional time. We plan to proceed with the release once these updates have been fully implemented