We present ACE-Bench (Azure SDK Coding Evaluation Benchmark), an execution-free benchmark that provides fast, reproducible pass or fail signals for whether large language model (LLM)-based coding agents use Azure SDKs correctly-without provisioning cloud resources or maintaining fragile end-to-end test environments. ACE-Bench turns official Azure SDK documentation examples into self-contained coding tasks and validates solutions with task-specific atomic criteria: deterministic regex checks that enforce required API usage patterns and reference-based LLM-judge checks that capture semantic workflow constraints. This design makes SDK-centric evaluation practical in day-to-day development and CI: it reduces evaluation cost, improves repeatability, and scales to new SDKs and languages as documentation evolves. Using a lightweight coding agent, we benchmark multiple state-of-the-art LLMs and quantify the benefit of retrieval in an MCP-enabled augmented setting, showing consistent gains from documentation access while highlighting substantial cross-model differences.
@article{arxiv.2604.09564,
title = {ACE-Bench: A Lightweight Benchmark for Evaluating Azure SDK Usage Correctness},
author = {Wenxing Zhu and Simeng Qi and Junkui Chen and Yan Xie and Min Huang and Jingkan He and Xiao Wang and Cheng Chen and Sijing Meng and Tianqi Zhang},
journal= {arXiv preprint arXiv:2604.09564},
year = {2026}
}