English
Related papers

Related papers: Do LLMs generate test oracles that capture the act…

200 papers

Large Language Models (LLMs) are nowadays extensively used for various types of software engineering tasks, primarily code generation. Previous research has shown how suitable prompt engineering could help developers in improving their code…

Recently, the large language models (LLMs) have shown extraordinary ability in understanding natural language and generating programming code. It has been a common practice of software engineers to consult LLMs when encountering coding…

Computation and Language · Computer Science 2024-01-30 Li Zhong , Zilong Wang

Large language models (LLMs) have behaved well in generating unit tests for Java projects. However, the performance for covering the complex focal methods within the projects is poor. Complex methods comprise many conditions and loops,…

Software Engineering · Computer Science 2024-08-22 Zejun Wang , Kaibo Liu , Ge Li , Zhi Jin

Automated test generation is essential for software quality assurance, with coverage rate serving as a key metric to ensure thorough testing. Recent advancements in Large Language Models (LLMs) have shown promise in improving test…

Software Engineering · Computer Science 2026-02-26 WeiZhe Xu , Mengyu Liu , Fanxin Kong

Automated test generation is crucial for ensuring the reliability and robustness of software applications while at the same time reducing the effort needed. While significant progress has been made in test generation research, generating…

Software Engineering · Computer Science 2025-01-30 Shaker Mahmud Khandaker , Fitsum Kifetew , Davide Prandi , Angelo Susi

The rapid integration of Large Language Models (LLMs) into software engineering practice is reshaping how software testing activities are performed. LLMs are increasingly used to support software testing. Consequently, software testing…

Software Engineering · Computer Science 2026-03-30 Peng Yang , Yunfeng Zhu , Chao Chang , Shengcheng Yu , Zhenyu Chen , Yong Tang

Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods.…

Software Engineering · Computer Science 2025-02-10 Niels Mündler , Mark Niklas Müller , Jingxuan He , Martin Vechev

Large language models (LLMs) have shown astonishing capability of generating software code, leading to its use to support developers in programming. Proposed tools have relied either on assistants for improved auto-complete or multi-agents,…

Software Engineering · Computer Science 2026-04-10 Jorge Melegati

Large Language Models (LLMs) have demonstrated their remarkable capabilities in numerous fields. This survey focuses on how LLMs empower users, regardless of their technical background, to use human languages to automatically generate…

Software Engineering · Computer Science 2025-04-03 Nam Huynh , Beiyu Lin

Automating hardware design could obviate a significant amount of human error from the engineering process and lead to fewer errors. Verilog is a popular hardware description language to model and design digital systems, thus generating…

Programming Languages · Computer Science 2022-12-22 Shailja Thakur , Baleegh Ahmad , Zhenxing Fan , Hammond Pearce , Benjamin Tan , Ramesh Karri , Brendan Dolan-Gavitt , Siddharth Garg

Background: The rise of Large Language Models (LLMs) in software development has opened new possibilities for code generation. Despite the widespread use of this technology, it remains unclear how well LLMs generate code solutions in terms…

Software Engineering · Computer Science 2025-08-04 Alfred Santa Molison , Marcia Moraes , Glaucia Melo , Fabio Santos , Wesley K. G. Assuncao

Large language models (LLMs) have become essential tools in software development, widely used for requirements engineering, code generation and review tasks. Software engineers often rely on LLMs to assess whether system code implementation…

Software Engineering · Computer Science 2025-08-19 Haolin Jin , Huaming Chen

Research on using Large Language Models (LLMs) in system development is expanding, especially in automated code and test generation. While E2E testing is vital for ensuring application quality, most test generation research has focused on…

Software Engineering · Computer Science 2025-03-25 Takuro Morimoto , Harumi Haraguchi

Unit testing is an essential but resource-intensive step in software development, ensuring individual code units function correctly. This paper introduces AgoneTest, an automated evaluation framework for Large Language Model-generated (LLM)…

Software Engineering · Computer Science 2025-11-27 Andrea Lops , Fedelucio Narducci , Azzurra Ragone , Michelantonio Trizio , Claudio Bartolini

Recent advancements in large language models (LLMs) have demonstrated exceptional success in a wide range of general domain tasks, such as question answering and following instructions. Moreover, LLMs have shown potential in various…

Software Engineering · Computer Science 2023-07-04 Yutian Tang , Zhijie Liu , Zhichao Zhou , Xiapu Luo

Unit testing is a fundamental practice in modern software engineering, with the aim of ensuring the correctness, maintainability, and reliability of individual software components. Very recently, with the advances in Large Language Models…

Software Engineering · Computer Science 2025-06-19 Quanjun Zhang , Chunrong Fang , Siqi Gu , Ye Shang , Zhenyu Chen , Liang Xiao

Large-Language Models (LLMs) are changing the way learners acquire knowledge outside the classroom setting. Previous studies have shown that LLMs seem effective in generating to short and simple questions in introductory CS courses using…

Programming Languages · Computer Science 2026-03-09 Yihan Zhang , Brigitte Pientka , Xujie Si

Generating code from a natural language programming task is one of the most successful applications of Large Language Models (LLMs). Yet, the generated program may be buggy. Without an oracle, such as an existing, correct implementation or…

Programming Languages · Computer Science 2025-12-16 Thomas Valentin , Ardi Madadi , Gaetano Sapia , Marcel Böhme

Open Source Software (OSS) has become a very important and crucial infrastructure worldwide because of the value it provides. OSS typically depends on contributions from developers across diverse backgrounds and levels of experience. Making…

Software Engineering · Computer Science 2025-10-08 Elijah Kayode Adejumo , Brittany Johnson

Large Language Models (LLMs) are widely used in Software Engineering (SE) for various tasks, including generating code, designing and documenting software, adding code comments, reviewing code, and writing test scripts. However, creating…

Software Engineering · Computer Science 2024-06-12 Abdul Malik Sami , Zeeshan Rasheed , Muhammad Waseem , Zheying Zhang , Herda Tomas , Pekka Abrahamsson