English

360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training

Computation and Language 2025-10-09 v2 Machine Learning
Authors: Haosheng Zou , Xiaowei Lv , Shousheng Jia , Lin Li , Xiaochun Gong , Xiangzheng Zhang
View on arXiv ↗ PDF ↗

Abstract

Adding sequence parallelism into LLaMA-Factory, we open-sourced 360-LLaMA-Factory at https://github.com/Qihoo360/360-LLaMA-Factory. 360-LLaMA-Factory has received wide recognition and used in models such as Light-R1 arXiv:2503.10460, TinyR1 arXiv:2503.04872, Kaggle AIMO math models and also in large companies' training frameworks. This technical report delves deeper into the different sequence parallel modes behind 360-LLaMA-Factory and discusses our implementation insights.

Cite

@article{arxiv.2505.22296,
  title  = {360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training},
  author = {Haosheng Zou and Xiaowei Lv and Shousheng Jia and Lin Li and Xiaochun Gong and Xiangzheng Zhang},
  journal= {arXiv preprint arXiv:2505.22296},
  year   = {2025}
}

Comments

v2: sp for vlms; code at https://github.com/Qihoo360/360-LLaMA-Factory

Related papers

View all related →
Machine Learning · Computer Science
RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use
Jiajun Chai, Guojun Yin, Zekun Xu, Chuhuai Yue +8
2025-09-10
Distributed, Parallel, and Cluster Computing · Computer Science
LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
Diandian Gu, Peng Sun, Qinghao Hu, Ting Huang +10
2024-06-27
Distributed, Parallel, and Cluster Computing · Computer Science
Context Parallelism for Scalable Million-Token Inference
Amy Yang, Jingyi Yang, Aya Ibrahim, Xinfeng Xie +5
2025-04-22
Machine Learning · Computer Science
Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
Istabrak Abbes, Gopeshh Subbaraj, Matthew Riemer, Nizar Islah +5
2025-08-05
Distributed, Parallel, and Cluster Computing · Computer Science
Amoeba: Runtime Tensor Parallel Transformation for LLM Inference Services
Haoyu Chen, Xue Li, Kun Qian, Yu Guan +2
2026-04-23
Computation and Language · Computer Science
360Zhinao Technical Report
360Zhinao Team
2024-05-24
Computation and Language · Computer Science
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye +3
2024-07-01
Computation and Language · Computer Science
Learning to Verify Summary Facts with Fine-Grained LLM Feedback
Jihwan Oh, Jeonghwan Choi, Nicole Hee-Yeon Kim, Taewon Yun +1
2024-12-17
Computation and Language · Computer Science
Extending Llama-3's Context Ten-Fold Overnight
Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao +3
2024-05-01
Machine Learning · Computer Science
Sequence Parallelism: Long Sequence Training from System Perspective
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li +1
2022-05-24
Machine Learning · Computer Science
DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
Tianhao Hu, Xiangcheng Liu, Youshao Xiao, Yang Zheng +14
2026-04-30
Computation and Language · Computer Science
LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models
Jianxin Yang
2023-11-10
Distributed, Parallel, and Cluster Computing · Computer Science
FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
Yujie Wang, Shiju Wang, Shenhan Zhu, Fangcheng Fu +6
2025-02-12
Computation and Language · Computer Science
LongAlign: A Recipe for Long Context Alignment of Large Language Models
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He +5
2024-02-01
Computer Vision and Pattern Recognition · Computer Science
Scaling RL to Long Videos
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu +10
2025-10-01
Distributed, Parallel, and Cluster Computing · Computer Science
Efficient Parallel Reinforcement Learning Framework using the Reactor Model
Jacky Kwok, Marten Lohstroh, Edward A. Lee
2024-02-06
Computation and Language · Computer Science
ICX360: In-Context eXplainability 360 Toolkit
Dennis Wei, Ronny Luss, Xiaomeng Hu, Lucas Monteiro Paes +5
2025-11-17
Computation and Language · Computer Science
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu +3
2025-02-18
Computer Vision and Pattern Recognition · Computer Science
MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia +4
2024-04-25
Distributed, Parallel, and Cluster Computing · Computer Science
Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model-Based Recommendation Systems
Haowei Yang, Yu Tian, Zhongheng Yang, Zhao Wang +2
2025-06-25
Machine Learning · Computer Science
Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models
Bowen Ping, Chengyou Jia, Minnan Luo, Hangwei Qian +1
2026-03-17
Machine Learning · Computer Science
mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs
Zhengmao Ye, Dengchun Li, Zetao Hu, Tingfeng Lan +7
2024-09-19
Machine Learning · Computer Science
TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree
Yu-Yang Qian, Yuan-Ze Xu, Zhen-Yu Zhang, Peng Zhao +1
2025-06-13
Computation and Language · Computer Science
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo +2
2023-05-30
Computation and Language · Computer Science
LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction
Ziwei Wang, Jie Zhou, Qin Chen, Min Zhang +4
2025-02-06
R2 v1 2026-07-01T02:46:16.151Z