虚拟宽度网络
机器学习
2025-11-18 v2 人工智能
摘要
我们引入虚拟宽度网络 (VWN)，一种在不增加隐层大小计算成本的前提下，提供更宽表示能力的框架。VWN 将表示宽度与主干宽度解耦，在保持主干计算几乎恒定的同时扩展嵌入空间。在大规模实验中，8倍的扩展可加速下一标记和第三标记预测优化超过2倍。优势随训练过程的推进而放大，由于损失间隙增大且收敛速度比例增加，表明VWN不仅令人印象深刻，还随规模增长而变得更加有效。此外，我们识别出虚拟宽度与损失减少之间约为对数线性的比例关系，这为探索虚拟宽度扩展作为大规模模型效率的新维度提供了初步经验依据和动机。
引用
@article{arxiv.2511.11238,
title = {Virtual Width Networks},
author = {Seed and Baisheng Li and Banggu Wu and Bole Ma and Bowen Xiao and Chaoyi Zhang and Cheng Li and Chengyi Wang and Chengyin Xu and Chi Zhang and Chong Hu and Daoguang Zan and Defa Zhu and Dongyu Xu and Du Li and Faming Wu and Fan Xia and Ge Zhang and Guang Shi and Haobin Chen and Hongyu Zhu and Hongzhi Huang and Huan Zhou and Huanzhang Dou and Jianhui Duan and Jianqiao Lu and Jianyu Jiang and Jiayi Xu and Jiecao Chen and Jin Chen and Jin Ma and Jing Su and Jingji Chen and Jun Wang and Jun Yuan and Juncai Liu and Jundong Zhou and Kai Hua and Kai Shen and Kai Xiang and Kaiyuan Chen and Kang Liu and Ke Shen and Liang Xiang and Lin Yan and Lishu Luo and Mengyao Zhang and Ming Ding and Mofan Zhang and Nianning Liang and Peng Li and Penghao Huang and Pengpeng Mu and Qi Huang and Qianli Ma and Qiyang Min and Qiying Yu and Renming Pang and Ru Zhang and Shen Yan and Shen Yan and Shixiong Zhao and Shuaishuai Cao and Shuang Wu and Siyan Chen and Siyu Li and Siyuan Qiao and Tao Sun and Tian Xin and Tiantian Fan and Ting Huang and Ting-Han Fan and Wei Jia and Wenqiang Zhang and Wenxuan Liu and Xiangzhong Wu and Xiaochen Zuo and Xiaoying Jia and Ximing Yang and Xin Liu and Xin Yu and Xingyan Bin and Xintong Hao and Xiongcai Luo and Xujing Li and Xun Zhou and Yanghua Peng and Yangrui Chen and Yi Lin and Yichong Leng and Yinghao Li and Yingshuan Song and Yiyuan Ma and Yong Shan and Yongan Xiang and Yonghui Wu and Yongtao Zhang and Yongzhen Yao and Yu Bao and Yuehang Yang and Yufeng Yuan and Yunshui Li and Yuqiao Xian and Yutao Zeng and Yuxuan Wang and Zehua Hong and Zehua Wang and Zengzhi Wang and Zeyu Yang and Zhengqiang Yin and Zhenyi Lu and Zhexi Zhang and Zhi Chen and Zhi Zhang and Zhiqi Lin and Zihao Huang and Zilin Xu and Ziyun Wei and Zuo Wang},
journal= {arXiv preprint arXiv:2511.11238},
year = {2025}
}