Related papers: Evaluating Heuristics for Iterative Impact Analysi…
Recent graphical interfaces offer direct manipulation mechanisms for authoring visualizations, but are largely restricted to static output. To author interactive visualizations, users must instead turn to textual specification; but, such…
AI has revolutionised decision-making across various fields. Yet human judgement remains paramount for high-stakes decision-making. This has fueled explorations of collaborative decision-making between humans and AI systems, aiming to…
Software behavioral models have proven useful for emulating and testing software systems. Many techniques have been proposed to infer behavioral models of software systems from their interaction traces. The quality of the inferred model is…
CONTEXT: There is growing interest in establishing software engineering as an evidence-based discipline. To that end, replication is often used to gain confidence in empirical findings, as opposed to reproduction where the goal is showing…
Recommender systems have been widely deployed across various domains such as e-commerce and social media, and intelligently suggest items like products and potential friends to users based on their preferences and interaction history, which…
Large language models (LLMs) are now used in multi-turn workflows, but we still lack a clear way to measure when iteration helps and when it hurts. We present an evaluation framework for iterative refinement that spans ideation, code, and…
Obtaining large annotated datasets is critical for training successful machine learning models and it is often a bottleneck in practice. Weak supervision offers a promising alternative for producing labeled datasets without ground truth…
The development of conversational AI assistants is an iterative process with multiple components. As such, the evaluation and continual improvement of these assistants is a complex and multifaceted problem. This paper introduces the…
The experimental evaluation of algorithms results in a large set of data which generally do not follow a normal distribution or are not heteroscedastic. Besides, some of its entries may be missing, due to the inability of an algorithm to…
Accurate estimation of treatment effects is essential for decision-making across various scientific fields. This task, however, becomes challenging in areas like social sciences and online marketplaces, where treating one experimental unit…
The goal of this article is to study fundamental mechanisms behind so-called indirect and direct data-driven control for unknown systems. Specifically, we consider policy iteration applied to the linear quadratic regulator problem. Two…
Using state-level opioid overdose mortality data from 1999-2016, we simulated four time-varying treatment scenarios, which correspond to real-world policy dynamics (ramp up, ramp down, temporary and inconsistent). We then evaluated seven…
According to a study of The Standish Group International, 44% of software projects cost more and last longer than expected. More accurate the effort estimation is; the better the enterprise gets organized and the more the software project…
Software development is an iterative process which includes a set of development steps that transform the initial high level specification of the system into its final, fully specified, implementation. This report discusses the theoretical…
We observe a change in the way users access information, that is, the rise of conversational information access (CIA) agents. However, the automatic evaluation of these agents remains an open challenge. Moreover, the training of CIA agents…
In this work, we consider the fundamental problem of reachability analysis over imperative programs with real variables. The reachability property requires that a program can reach certain target states during its execution. Previous works…
Create an idea, prototype it, evaluate if users like it, then learn. It is the circle of business. If AI can operate in all parts of the circle, it will enable rapid iteration and learning speeds for businesses. Experiment platforms that…
There exist applications of reinforcement learning like medicine where policies need to be ''interpretable'' by humans. User studies have shown that some policy classes might be more interpretable than others. However, it is costly to…
Return on Investment (ROI) is one of the most popular performance measurement and evaluation metrics. ROI analysis (when applied correctly) is a powerful tool in comparing solutions and making informed decisions on the acquisitions of…
Why do biased predictions arise? What interventions can prevent them? We evaluate 8.2 million algorithmic predictions of math performance from $\approx$400 AI engineers, each of whom developed an algorithm under a randomly assigned…