Phi-3 技术报告：高度强大的语言模型在您的手机上本地部署
计算与语言
2024-09-04 v4 人工智能
摘要
我们介绍了 phi-3-mini，这是一个由 33 亿个参数构成的语言模型，训练使用了 33000 亿个 token。无论是通过学术基准测试还是内部测试，phi-3-mini 的整体性能都与 Mixtral 8x7B 和 GPT-3.5 相当（例如，phi-3-mini 在 MMLU 上取得 69%，在 MT-bench 上取得 8.38），尽管其体积小到可以部署在手机上。我们的训练数据集是 phi-2 所用数据集的放大版本，由大量过滤后的公开网页数据和合成数据构成。该模型还进一步对鲁棒性、安全性和聊天格式进行了对齐。我们还提供了参数规模化结果，其中包括 7B、14B 模型训练 48000 亿 token，称为 phi-3-small、phi-3-medium，两者都显著优于 phi-3-mini（例如，分别在 MMLU 上取得 75%、78%，在 MT-bench 上取得 8.7、8.9）。为了提升多语言、多模态和长上下文能力，我们引入了 phi-3.5 系列中的三个模型：phi-3.5-mini、phi-3.5-MoE 和 phi-3.5-Vision。phi-3.5-MoE 是一个 16 x 3.8B 的 MoE 模型，拥有 66 亿个活跃参数，在语言推理、数学和代码任务方面的性能优于其他规模相似的开源模型，如 Llama 3.1 和 Mixtral 系列，同时与 Gemini-1.5-Flash 和 GPT-4o-mini 持平。与此同时，phi-3.5-Vision 是一个由 42 亿个参数构成的模型，源自 phi-3.5-mini，擅长推理任务，能够处理单图像和文本提示，以及多图像和文本提示。
引用
@article{arxiv.2404.14219,
title = {Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone},
author = {Marah Abdin and Jyoti Aneja and Hany Awadalla and Ahmed Awadallah and Ammar Ahmad Awan and Nguyen Bach and Amit Bahree and Arash Bakhtiari and Jianmin Bao and Harkirat Behl and Alon Benhaim and Misha Bilenko and Johan Bjorck and Sébastien Bubeck and Martin Cai and Qin Cai and Vishrav Chaudhary and Dong Chen and Dongdong Chen and Weizhu Chen and Yen-Chun Chen and Yi-Ling Chen and Hao Cheng and Parul Chopra and Xiyang Dai and Matthew Dixon and Ronen Eldan and Victor Fragoso and Jianfeng Gao and Mei Gao and Min Gao and Amit Garg and Allie Del Giorno and Abhishek Goswami and Suriya Gunasekar and Emman Haider and Junheng Hao and Russell J. Hewett and Wenxiang Hu and Jamie Huynh and Dan Iter and Sam Ade Jacobs and Mojan Javaheripi and Xin Jin and Nikos Karampatziakis and Piero Kauffmann and Mahoud Khademi and Dongwoo Kim and Young Jin Kim and Lev Kurilenko and James R. Lee and Yin Tat Lee and Yuanzhi Li and Yunsheng Li and Chen Liang and Lars Liden and Xihui Lin and Zeqi Lin and Ce Liu and Liyuan Liu and Mengchen Liu and Weishung Liu and Xiaodong Liu and Chong Luo and Piyush Madan and Ali Mahmoudzadeh and David Majercak and Matt Mazzola and Caio César Teodoro Mendes and Arindam Mitra and Hardik Modi and Anh Nguyen and Brandon Norick and Barun Patra and Daniel Perez-Becker and Thomas Portet and Reid Pryzant and Heyang Qin and Marko Radmilac and Liliang Ren and Gustavo de Rosa and Corby Rosset and Sambudha Roy and Olatunji Ruwase and Olli Saarikivi and Amin Saied and Adil Salim and Michael Santacroce and Shital Shah and Ning Shang and Hiteshi Sharma and Yelong Shen and Swadheen Shukla and Xia Song and Masahiro Tanaka and Andrea Tupini and Praneetha Vaddamanu and Chunyu Wang and Guanhua Wang and Lijuan Wang and Shuohang Wang and Xin Wang and Yu Wang and Rachel Ward and Wen Wen and Philipp Witte and Haiping Wu and Xiaoxia Wu and Michael Wyatt and Bin Xiao and Can Xu and Jiahang Xu and Weijian Xu and Jilong Xue and Sonali Yadav and Fan Yang and Jianwei Yang and Yifan Yang and Ziyi Yang and Donghan Yu and Lu Yuan and Chenruidong Zhang and Cyril Zhang and Jianwen Zhang and Li Lyna Zhang and Yi Zhang and Yue Zhang and Yunan Zhang and Xiren Zhou},
journal= {arXiv preprint arXiv:2404.14219},
year = {2024}
}
备注
24 pages