Knowledge distillation offers a transformative pathway to developing powerful, yet efficient, small language models (SLMs) suitable for resource-constrained environments. In this paper, we benchmark the performance and computational cost of distilled models against their vanilla and proprietary counterparts, providing a quantitative analysis of their efficiency. Our results demonstrate that distillation creates a superior performance-tocompute curve. We find that creating a distilled 8B model is over 2,000 times more compute-efficient than training its vanilla counterpart, while achieving reasoning capabilities on par with, or even exceeding, standard models ten times its size. These findings validate distillation not just as a compression technique, but as a primary strategy for building state-of-the-art, accessible AI
@article{arxiv.2602.20164,
title = {Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings},
author = {Sachin Gopal Wani and Eric Page and Ajay Dholakia and David Ellison},
journal= {arXiv preprint arXiv:2602.20164},
year = {2026}
}
Comments
16 pages, 5 figures, accepted at the the 2025 TPCTC Conference