Training large language models typically demands extensive GPU memory and substantial financial investment, which poses a barrier for many small- to medium-sized teams. In this paper, we propose a full-parameter pre-training and fine-tuning framework based on block coordinate descent (BCD), enhanced with engineering optimizations, to enable efficient training of large-scale models on cost-effective RTX 4090, A100 and A800 GPU clusters. Under identical hardware configurations, we reduce the training cost of a 7B model to 33% on A100/A800 and only 2.6% on RTX 4090, compared to standard full-parameter training. It also enables large models previously restricted to A100 clusters to be trained on RTX 4090 without degrading performance. BCD achieves comparable or better accuracy than full-parameter and fine-tuning methods at most cases, with lower GPU consumption and improved hardware utilization.
@article{arxiv.2506.12037,
title = {Exploiting Block Coordinate Descent for Cost-Effective LLM Model Training},
author = {Zeyu Liu and Yan Li and Yunquan Zhang and Boyang Zhang and Guoyong Jiang and Xin Zhang and Limin Xiao and Weifeng Zhang and Daning Cheng},
journal= {arXiv preprint arXiv:2506.12037},
year = {2025}
}
Comments
We have revised certain details of the manuscript and incorporated new experimental