Related papers: The Ghost of Performance Reproducibility Past
The performance of software systems remains a persistent concern in the field of software engineering. While traditional metrics like binary size and execution time have long been focal points for developers, the power consumption concern…
Reproducibility remains a central challenge in computational social science, where complex workflows, evolving software ecosystems, and inconsistent documentation hinder researchers ability to re-execute published methods. This study…
A quantum computer has now solved a specialized problem believed to be intractable for supercomputers, suggesting that quantum processors may soon outperform supercomputers on scientifically important problems. But flaws in each quantum…
Deep neural networks are in the limelight of machine learning with their excellent performance in many data-driven applications. However, they can lead to inaccurate predictions when queried in out-of-distribution data points, which can…
Reproducibility and comparability of empirical results are at the core tenet of the scientific method in any scientific field. To ease reproducibility of empirical studies, several benchmarks in software engineering research, such as…
Project Management process plays a significant role in effective development of software projects. Key challenges in the project management process are the estimation of time, cost, defect count, and subsequently selection of apt…
The emergence of heterogeneity in high-performance computing, which harnesses under one integrated system several platforms of different architectures, also led to the development of innovative cross-platform programming models. Along with…
We consider unsupervised crowdsourcing performance based on the model wherein the responses of end-users are essentially rated according to how their responses correlate with the majority of other responses to the same subtasks/questions.…
Using multiple-choice questions (MCQs) has become a standard for assessing LLM capabilities efficiently. A variety of metrics can be employed for this task. However, previous research has not conducted a thorough assessment of them. At the…
Machine learning inference should be subject to stringent inference time constraints while ensuring high inference quality, especially in safety-critical (e.g., autonomous driving) and mission-critical (e.g., emotion recognition) contexts.…
Counts of galaxies as a function of apparent magnitude are among the most time-honored observations in cosmology. In this Letter, we focus on some statistical properties of these counts which are fundamental in order to characterize the…
Fluctuation-enhanced sensing comprises the analysis of the stochastic component of the sensor signal and the utilization of the microscopic dynamics of the interaction between the agent and the sensor. We study the relationship between the…
Machine learning models frequently experience performance drops under distribution shifts. The underlying cause of such shifts may be multiple simultaneous factors such as changes in data quality, differences in specific covariate…
Given a collection of computational models that all estimate values of the same natural process, we compare the performance of the average of the collection to the individual member whose estimates are nearest a given set of observations.…
Reproducibility of modeling is a problem that exists for any machine learning practitioner, whether in industry or academia. The consequences of an irreproducible model can include significant financial costs, lost time, and even loss of…
Successful HPC software applications are long-lived. When ported across machines and their compilers, these applications often produce different numerical results, many of which are unacceptable. Such variability is also a concern while…
A key trait of stochastic optimizers is that multiple runs of the same optimizer in attempting to solve the same problem can produce different results. As a result, their performance is evaluated over several repeats, or runs, on the…
Data is a cornerstone of empirical software engineering (ESE) research and practice. Data underpin numerous process and project management activities, including the estimation of development effort and the prediction of the likely location…
Understanding the relationship between complexity and stability in large dynamical systems -- such as ecosystems -- remains a key open question in complexity theory which has inspired a rich body of work developed over more than fifty…
The values of software organizations are crucial for achieving high performance; in particular, agile development approaches emphasize their importance. Researchers have thus far often assumed that a specific set of values, compatible with…