While Large Language Models (LLMs) achieve near-human performance on standard benchmarks, their capabilities often fail to generalize to complex, real-world problems. To bridge this gap, we introduce DeepQuestion, a scalable, automated framework that systematically elevates the cognitive complexity of existing datasets. Grounded in Bloom's taxonomy, DeepQuestion generates (1) scenario-based problems to test the application of knowledge in noisy, realistic contexts, and (2) instruction-based prompts that require models to create new questions from a given solution path, assessing synthesis and evaluation skills. Our extensive evaluation across ten leading open-source and proprietary models reveals a stark performance decline with accuracy dropping by up to 70% as tasks ascend the cognitive hierarchy. These findings underscore that current benchmarks overestimate true reasoning abilities and highlight the critical need for cognitively diverse evaluations to guide future LLM development.
@article{arxiv.2505.24532,
title = {DeepQuestion: Systematic Generation of Real-World Challenges for Evaluating LLMs Performance},
author = {Ali Khoramfar and Ali Ramezani and Mohammad Mahdi Mohajeri and Mohammad Javad Dousti and Majid Nili Ahmadabadi and Heshaam Faili},
journal= {arXiv preprint arXiv:2505.24532},
year = {2026}
}