JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency
Abstract
We introduce JoyAI-LLM Flash, an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency in the sub-50B parameter regime. JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline, including supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and large-scale reinforcement learning (RL) across diverse environments. To improve token efficiency, JoyAI-LLM Flash strategically balances \emph{thinking} and \emph{non-thinking} cognitive modes and introduces FiberPO, a novel RL algorithm inspired by fibration theory that decomposes trust-region maintenance into global and local components, providing unified multi-scale stability control for LLM policy optimization. To enhance architectural sparsity, the model comprises 48B total parameters while activating only 2.7B parameters per forward pass, achieving a substantially higher sparsity ratio than contemporary industry leading models of comparable scale. To further improve inference throughput, we adopt a joint training-inference co-design that incorporates dense Multi-Token Prediction (MTP) and Quantization-Aware Training (QAT). We release the checkpoints for both JoyAI-LLM-48B-A3B Base and its post-trained variants on Hugging Face to support the open-source community.
Cite
@article{arxiv.2604.03044,
title = {JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency},
author = {Aichen Cai and Anmeng Zhang and Anyu Li and Bo Zhang and Bohua Cai and Chang Li and Changjian Jiang and Changkai Lu and Chao Xue and Chaocai Liang and Cheng Zhang and Dongkai Liu and Fei Wang and Guoqiang Huang and Haijian Ke and Han Lin and Hao Wang and Ji Miao and Jiacheng Zhang and Jialong Shi and Jifeng Zhu and Jingjing Qian and Junhui Luo and Junwu Xiong and Lam So and Liang Huang and Ming Ke and Mingyang Li and Panfeng Shi and Peng Hao and Qi Wang and Qian Lai and Qiaoqiao Yuan and Qingyu Yin and Qiong Cao and Qixiang Wang and Rongcheng Bian and Rongduo Han and Shaoqiang Zheng and Shi Hu and Shi Suo and Shijie Ren and Shijin Zhang and Shiying Fan and Shuai Xie and Tianyi Zhang and Wei Liu and Wentao Tan and Xianghan Meng and Xiaodong He and Xing Pan and Xiran Wang and Xuyang Peng and Ya Zhang and Yang Liu and Yangyang Duan and Yanxu Chen and Yicheng Gong and Yidan Huang and Yifei Liu and Yinhao Bai and Yongqiang Liu and Yuesong Zhang and Yuqi Zhang and Zerui Xie and Zhenfang Wang and Zhennan Shen and Zheyuan Liu and Zhuwei Zeng},
journal= {arXiv preprint arXiv:2604.03044},
year = {2026}
}
Comments
Xiaodong He is the corresponding author