LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation
Abstract
Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn. To address this bottleneck, we propose LoopFM (Learning frOm HistOrical ReP*resentations of FM), a framework that opens a high-bandwidth transfer channel by structuring FM intermediate embeddings as input features (e.g., user history sequence) for downstream VMs, without requiring real-time FM inference at serving and architectural coupling between FM and VM. We provide a theoretical framework for LoopFM with a gain decomposition and transfer-ratio analysis. On three public benchmarks, LoopFM demonstrates strong AUC improvements (e.g., 6\%+ on TaobaoAd) and complementary knowledge transfer capability with KD. On industrial-scale systems (billions of examples, trillion-parameter FMs), LoopFM approximately doubles the knowledge transfer ratio on top of KD, delivering a +0.5\% conversion improvement in Y1H1, and a +1.03\% and +1.22\% conversion improvement from two individual launches respectively in Y1H2.
Cite
@article{arxiv.2605.29280,
title = {LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation},
author = {Shali Jiang and Hua Zheng and Boyang Liu and Laming Chen and Kenny Lov and Chuanqi Xu and Lisang Ding and Qinghai Zhou and Can Cui and Xiaolong Liu and Xiaoyi Liu and Yasmine Badr and Xin Xu and Jiyan Yang and Ellie Dingqiao Wen and Gerard Jonathan Mugisha Akkerhuis and Chenxiao Guan and Rong Jin and Ruichao Qiu and Xian Chen and Shifu Xu and Zhehui Zhou and Ping Chen and Rui Yang and Haicheng Chen and Xiangge Meng and Song Zhou and Dharak Kharod and Shuyu Xu and Qiang Jin and Qiao Yang and Wankun Zhu and Qin Huang and Yuzhen Huang and Darren Liu and Parish Aggarwal and Hui Zhou and Erzhuo Wang and Shuo Chang and Xiaorui Gan and Wenlin Chen and Santanu Kolay and Huayu Li},
journal= {arXiv preprint arXiv:2605.29280},
year = {2026}
}
Comments
Shali Jiang, Hua Zheng, Boyang Liu contributed equally to this work