English

FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale

Software Engineering 2025-11-19 v1 Artificial Intelligence Machine Learning Programming Languages

Abstract

Flaky tests that non-deterministically pass or fail waste developer time and slow release cycles. While large language models (LLMs) show promise for automatically repairing flaky tests, existing approaches like FlakyDoctor fail in industrial settings due to the context problem: providing either too little context (missing critical production code) or too much context (overwhelming the LLM with irrelevant information). We present FlakyGuard, which addresses this problem by treating code as a graph structure and using selective graph exploration to find only the most relevant context. Evaluation on real-world flaky tests from industrial repositories shows that FlakyGuard repairs 47.6 % of reproducible flaky tests with 51.8 % of the fixes accepted by developers. Besides it outperforms state-of-the-art approaches by at least 22 % in repair success rate. Developer surveys confirm that 100 % find FlakyGuard's root cause explanations useful.

Keywords

Cite

@article{arxiv.2511.14002,
  title  = {FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale},
  author = {Chengpeng Li and Farnaz Behrang and August Shi and Peng Liu},
  journal= {arXiv preprint arXiv:2511.14002},
  year   = {2025}
}

Comments

To appear in ASE 2025

R2 v1 2026-07-01T07:42:25.094Z