Large Language Models (LLMs) are increasingly used to refactor unit tests, improving readability and structure while preserving behavior. Evaluating such refactorings, however, remains difficult: metrics like CodeBLEU penalize beneficial renamings and edits, while semantic similarities overlook readability and modularity. We propose CTSES, a first step toward human-aligned evaluation of refactored tests. CTSES combines CodeBLEU, METEOR, and ROUGE-L into a composite score that balances semantics, lexical clarity, and structural alignment. Evaluated on 5,000+ refactorings from Defects4J and SF110 (GPT-4o and Mistral-Large), CTSES reduces false negatives and provides more interpretable signals than individual metrics. Our emerging results illustrate that CTSES offers a proof-of-concept for composite approaches, showing their promise in bridging automated metrics and developer judgments.
@article{arxiv.2506.06767,
title = {Beyond Surface Similarity: Evaluating LLM-Based Test Refactorings with Structural and Semantic Awareness},
author = {Wendkûuni C. Ouédraogo and Yinghua Li and Xueqi Dang and Xin Zhou and Anil Koyuncu and Jacques Klein and David Lo and Tegawendé F. Bissyandé},
journal= {arXiv preprint arXiv:2506.06767},
year = {2025}
}