Recent studies have demonstrated that few-shot learning allows LLMs to generate training data for supervised models at a low cost. However, the quality of LLM-generated data may not entirely match that of human-labeled data. This raises a crucial question: how should one balance the trade-off between the higher quality but more expensive human data and the lower quality yet substantially cheaper LLM-generated data? In this paper, we synthesized training data for conversational semantic frame analysis using GPT-4 and examined how to allocate budgets optimally to achieve the best performance. Our experiments, conducted across various budget levels, reveal that optimal cost-efficiency is achieved by combining both human and LLM-generated data across a wide range of budget levels. Notably, as the budget decreases, a higher proportion of LLM-generated data becomes more preferable.
@article{arxiv.2410.06550,
title = {Investigating Cost-Efficiency of LLM-Generated Training Data for Conversational Semantic Frame Analysis},
author = {Shiho Matta and Yin Jou Huang and Fei Cheng and Hirokazu Kiyomaru and Yugo Murawaki},
journal= {arXiv preprint arXiv:2410.06550},
year = {2024}
}
Comments
12 pages including 4 pages of references and appendix. 7 figures