Evaluative claims about LLM infrastructure -- ``workload X is fastest on hardware Y with software Z'' -- depend on a complex configuration space spanning hardware accelerators, interconnect bandwidth, software frameworks, parallelism plans, and communication libraries. Current infrastructure evaluation benchmarks publish a small set of end-to-end numbers that do not explain why one configuration outperforms another. We present CCL-Bench, a trace-based benchmark that addresses the limitations of existing benchmarks by recording reusable evidence for every ML workload. Each contributed data point in CCL-Bench packages an execution trace, a YAML workload card, and the launch scripts. We have developed a community-extensible toolkit to compute fine-grained compute, memory, and communication efficiency metrics from this evidence. Using CCL-Bench, we surface three claims that summary-statistic benchmarks cannot support: (i) higher compute-communication overlap can coincide with longer training step time and reveal inefficient parallelization choices, (ii) doubling TPU interconnect bandwidth yields a much higher end-to-end improvement in step time than doubling GPU interconnect bandwidth on small and medium workloads, and (iii) the best-tuned configuration on one training framework can run up to 3× slower than the best-tuned configuration on a peer framework on identical hardware.
@article{arxiv.2605.06544,
title = {CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure},
author = {Eric Ding and Byungsoo Oh and Bhaskar Kataria and Kaiwen Guo and Jelena Gvero and Abhishek Vijaya Kumar and Arjun Devraj and Lindsey Bowen and Atharv Sonwane and Emaad Manzoor and Rachee Singh},
journal= {arXiv preprint arXiv:2605.06544},
year = {2026}
}