Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augments transformers with boosting principles through subgrid token selection and importance-weighted sampling. Our method incorporates a least square boosting objective directly into the transformer pipeline, enabling more efficient training and improved performance. Across multiple fine-grained text classification benchmarks, BoostTransformer demonstrates both faster convergence and higher accuracy, surpassing standard transformers while minimizing architectural search overhead.
@article{arxiv.2508.02924,
title = {Tricks and Plug-ins for Gradient Boosting with Transformers},
author = {Biyi Fang and Truong Vo and Jean Utke and Diego Klabjan},
journal= {arXiv preprint arXiv:2508.02924},
year = {2025}
}
Comments
Update the title of the pdf file only. The old version has a different title to the arxiv abstract. arXiv admin note: substantial text overlap with arXiv:2203.00761