Large Language Models (LLMs) ) have demonstrated promise in boosting productivity across AI-powered tools, yet existing benchmarks like Massive Multitask Language Understanding (MMLU) inadequately assess enterprise-specific task complexities. We propose a 14-task framework grounded in Bloom's Taxonomy to holistically evaluate LLM capabilities in enterprise contexts. To address challenges of noisy data and costly annotation, we develop a scalable pipeline combining LLM-as-a-Labeler, LLM-as-a-Judge, and corrective retrieval-augmented generation (CRAG), curating a robust 9,700-sample benchmark. Evaluation of six leading models shows open-source contenders like DeepSeek R1 rival proprietary models in reasoning tasks but lag in judgment-based scenarios, likely due to overthinking. Our benchmark reveals critical enterprise performance gaps and offers actionable insights for model optimization. This work provides enterprises a blueprint for tailored evaluations and advances practical LLM deployment.
@article{arxiv.2506.20274,
title = {Enterprise Large Language Model Evaluation Benchmark},
author = {Liya Wang and David Yi and Damien Jose and John Passarelli and James Gao and Jordan Leventis and Kang Li},
journal= {arXiv preprint arXiv:2506.20274},
year = {2025}
}
Comments
Submitted to MLNLP 2025 at https://csity2025.org/mlnlp/index