How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons. Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements.
@article{arxiv.2506.09050,
title = {ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering},
author = {Yuki Imajuku and Kohki Horie and Yoichi Iwata and Kensho Aoki and Naohiro Takahashi and Takuya Akiba},
journal= {arXiv preprint arXiv:2506.09050},
year = {2025}
}
Comments
Accepted at NeurIPS 2025 Datasets & Benchmarks Track