Multi-agent LLM-driven systems for software development are rapidly gaining traction, offering new opportunities to enhance productivity. At Atlassian, we deployed Human-in-the-Loop Software Development Agents to resolve Jira work items and evaluated the generated code quality using functional correctness testing and GPT-based similarity scoring. This paper highlights two major challenges: the high computational costs of unit testing and the variability in LLM-based evaluations. We also propose future research directions to improve evaluation frameworks for Human-In-The-Loop software development tools.
@article{arxiv.2506.11009,
title = {Human-In-The-Loop Software Development Agents: Challenges and Future Directions},
author = {Jirat Pasuksmit and Wannita Takerngsaksiri and Patanamon Thongtanunam and Chakkrit Tantithamthavorn and Ruixiong Zhang and Shiyan Wang and Fan Jiang and Jing Li and Evan Cook and Kun Chen and Ming Wu},
journal= {arXiv preprint arXiv:2506.11009},
year = {2025}
}
Comments
The International Conference on Mining Software Repositories (MSR) 2025, Industry track