Industry practitioners always face the problem of choosing the appropriate model for deployment under different considerations, such as to maximize a metric that is crucial for production, or to reduce the total cost given financial concerns. In this work, we focus on the text classification task and present a quantitative analysis for this challenge. Using classification accuracy as the main metric, we evaluate the classifiers' performances for a variety of models, including large language models, along with their associated costs, including the annotation cost, training (fine-tuning) cost, and inference cost. We then discuss the model choices for situations like having a large number of samples needed for inference. We hope our work will help people better understand the cost/quality trade-offs for the text classification task.
@article{arxiv.2301.07006,
title = {Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks},
author = {Shi Zong and Josh Seltzer and Jiahua and Pan and Kathy Cheng and Jimmy Lin},
journal= {arXiv preprint arXiv:2301.07006},
year = {2023}
}