Related papers: Investigating Test Overfitting on SWE-bench
Excessive reuse of test data has become commonplace in today's machine learning workflows. Popular benchmarks, competitions, industrial scale tuning, among other applications, all involve test data reuse beyond guidance by statistical…
Choice overload - in which larger choice sets are detrimental to a chooser's well-being - is potentially of great importance in the design of economic policy. Yet the current evidence on its prevalence is inconclusive. We argue that…
Overfitting is a common problem in machine learning, which means the model too closely fits the training data while performing poorly in the test data. Among various methods of coping with overfitting, dropout is one of the representative…
Code Agent development is an extremely active research area, where a reliable performance metric is critical for tracking progress and guiding new developments. This demand is underscored by the meteoric rise in popularity of SWE-Bench.…
A primary concern of excessive reuse of test datasets in machine learning is that it can lead to overfitting. Multiclass classification was recently shown to be more resistant to overfitting than binary classification. In an open problem of…
In the context of test case based automated program repair (APR), the research community call the patches that pass all the test cases but fail to actually fix the bug test case overfitted patches. Currently, overfitted patches has to be…
A view on software testing, taken in a broad sense and considered a important activity is presented. We discuss the methods and techniques for applying tests and the reasons we recognize make it difficult for industry to adopt the advances…
Context. Refactoring has been widely investigated in the past in relation to production code quality, yet still little is known on how developers apply refactoring on test code. Specifically, there is still a lack of investigation into how…
Adversarial training has been empirically shown to be more prone to overfitting than standard training. The exact underlying reasons still need to be fully understood. In this paper, we identify one cause of overfitting related to current…
In software testing, the large size of the input domain makes exhaustively testing the inputs a daunting and often impossible task. Pair-wise testing is a popular approach to combinatorial testing problems. This paper reviews Pair-wise…
Objective: This paper proposes a framework to support the scientific research of standards so that they can be better measured, evaluated, and designed. Methods: Beginning with the notion of common models, the framework describes the…
Context: Research software is essential for developing advanced tools and models to solve complex research problems and drive innovation across domains. Therefore, it is essential to ensure its correctness. Software testing plays a vital…
Understanding how software defects manifest and evolve in production environments is critical for improving reliability. While previous research has largely focused on pre-release defects, the nature of residual faults, i.e., those escaping…
Quantitative properties of stochastic systems are usually specified in logics that allow one to compare the measure of executions satisfying certain temporal properties with thresholds. The model checking problem for stochastic systems with…
Issue resolution, a complex Software Engineering (SWE) task integral to real-world development, has emerged as a compelling challenge for artificial intelligence. The establishment of benchmarks like SWE-bench revealed this task as…
Over the past few years, deep learning methods have been applied for a wide range of Software Engineering (SE) tasks, including in particular for the important task of automatically predicting and localizing faults in software. With the…
Machine learning components are now central to AI-infused software systems, from recommendations and code assistants to clinical decision support. As regulations and governance frameworks increasingly require deleting sensitive data from…
Automatic program repair usually relies heavily on test cases for both bug identification and fix validation. The issue is that writing test cases is tedious, running them takes much time, and validating a fix through tests does not…
The main problems of Software Engineering appear as a result of incompatibilities. For example, the quality of organization of the production process depends on correspondence with existent resources and on a common understanding of project…
Achieving fault-tolerance will require a strong relationship between the hardware and the protocols used. Different approaches will therefore naturally have tailored proof-of-principle experiments to benchmark progress. Nevertheless,…