KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes
Abstract
Discovering insights from a real-world data lake potentially containing unclean, semi-structured, and unstructured data requires a variety of data processing tasks, ranging from extraction and cleaning to integration, analysis, and modeling. This process often also demands domain knowledge and project-specific insight. While AI models have shown remarkable results in reasoning and code generation, their abilities to design and execute complex pipelines that solve these data-lake-to-insight challenges remain unclear. We introduce KramaBench which consists of 104 manually curated and solved challenges spanning 1700 files, 24 data sources, and 6 domains. KramaBench focuses on testing the end-to-end capabilities of AI systems to solve challenges which require automated orchestration of different data tasks. KramaBench also features a comprehensive evaluation framework assessing the pipeline design and individual data task implementation abilities of AI systems. We evaluate 8 LLMs using our single-agent reference framework DS-Guru, alongside both open- and closed-source single- and multi-agent systems, and find that while current agentic systems may handle isolated data-science tasks and generate plausible draft pipelines, they struggle with producing working end-to-end pipelines. On KramaBench, the best system reaches only 55% end-to-end accuracy in the full data-lake setting. Even with perfect retrieval, the accuracy tops out at 62%. Leading LLMs can identify up to 42% of important data tasks but can only fully implement 20% of individual data tasks. Our code, reference framework, and data are available at https://github.com/mitdbg/KramaBench.
Cite
@article{arxiv.2506.06541,
title = {KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes},
author = {Eugenie Lai and Gerardo Vitagliano and Ziyu Zhang and Om Chabra and Sivaprasad Sudhir and Anna Zeng and Anton A. Zabreyko and Chenning Li and Ferdi Kossmann and Jialin Ding and Jun Chen and Markos Markakis and Matthew Russo and Weiyang Wang and Ziniu Wu and Michael J. Cafarella and Lei Cao and Samuel Madden and Tim Kraska},
journal= {arXiv preprint arXiv:2506.06541},
year = {2026}
}