Related papers: Ensure A/B Test Quality at Scale with Automated Ra…
In this paper in which we address the evaluation of measurement process quality, we mainly focus on the evaluation procedure, as far as it is based on the numerical measurement outcomes. We challenge the approach where the "exact" value of…
Business Process Simulation (BPS) is an approach to analyze the performance of business processes under different scenarios. For example, BPS allows us to estimate what would be the cycle time of a process if one or more resources became…
Annotated datasets are an essential ingredient to train, evaluate, compare and productionalize supervised machine learning models. It is therefore imperative that annotations are of high quality. For their creation, good quality management…
In online randomized experiments or A/B tests, accurate predictions of participant inclusion rates are of paramount importance. These predictions not only guide experimenters in optimizing the experiment's duration but also enhance the…
Randomized benchmarking (RB) is a widely used strategy to assess the quality of available quantum gates in a computational context. RB involves applying known random sequences of gates to an initial state and using the statistics of a final…
In industrial engineering and manufacturing, quality control is an essential part of the production process of a product. To ensure proper functionality of a manufactured good, rigorous testing has to be performed to identify defective…
The quality of Artificial Intelligence (AI) algorithms is of significant importance for confidently adopting algorithms in various applications such as cybersecurity, healthcare, and autonomous driving. This work presents a principled…
At eBay, there are thousands of product health metrics for different domain teams to monitor. We built a two-phase alerting system to notify users with actionable alerts based on anomaly detection and alert retrieval. In the first phase, we…
The requirements in automation, digitalization, and fast computations have loaded the IT sector with expectations of highly reliable, efficient, and cost-effective software. Given that the process of testing, verification, and validation of…
Large-scale randomized experiments, sometimes called A/B tests, are increasingly prevalent in many industries. Though such experiments are often analyzed via frequentist $t$-tests, arguably such analyses are deficient: $p$-values are hard…
New testing and development procedures and methods are needed to address topics like power system stability, operation and control in the context of grid integration of rapidly developing smart grid technologies. In this context, individual…
Automated approaches to answer patient-posed health questions are rising, but selecting among systems requires reliable evaluation. The current gold standard for evaluating the free-text artificial intelligence (AI) responses--human expert…
With the rapid development of internet Router, the complexity of its mainboard has been growing dramatically. The high reliability requirement renders the number of testing cases increasing exponentially, which becomes the bottleneck that…
Large technology firms face the problem of moderating content on their online platforms for compliance with laws and policies. To accomplish this at the scale of billions of pieces of content per day, a combination of human and machine…
To improve the efficiency of Monte Carlo estimation, practitioners are turning to biased Markov chain Monte Carlo procedures that trade off asymptotic exactness for computational speed. The reasoning is sound: a reduction in variance due to…
We propose an alternative framework to existing setups for controlling false alarms when multiple A/B tests are run over time. This setup arises in many practical applications, e.g. when pharmaceutical companies test new treatment options…
The prevalence of software systems has become an integral part of modern-day living. Software usage has increased significantly, leading to its growth in both size and complexity. Consequently, software development is becoming a more…
Randomized experiments are the gold standard for evaluating the effects of changes to real-world systems. Data in these tests may be difficult to collect and outcomes may have high variance, resulting in potentially large measurement error.…
A common assumption in the spatial analysis of agricultural field trials is stationarity. In practice, however, this assumption is often violated due to unaccounted field effects. For instance, in plant breeding field trials, this can lead…
Mobile app development involves a unique set of challenges including device fragmentation and rapidly evolving platforms, making testing a difficult task. The design space for a comprehensive mobile testing strategy includes features,…