Related papers: Measuring Diagnostic Test Performance Using Imperf…
We consider a patient risk models which has access to patient features such as vital signs, lab values, and prior history but does not have access to a patient's diagnosis. For example, this occurs in a model deployed at intake time for…
Cross-sectional incidence estimation based on recency testing has become a widely used tool in HIV research. Recently, this method has gained prominence in HIV prevention trials to estimate the "placebo" incidence that participants might…
In group testing, the task is to identify defective items by testing groups of them together using as few tests as possible. We consider the setting where each item is defective with a constant probability $\alpha$, independent of all other…
Organizations use intrusion detection systems (IDSes) to identify harmful activity among millions of computer network events. Cybersecurity analysts review IDS alarms to verify whether malicious activity occurred and to take remedial…
Stress testing poses a causal question: how would portfolio credit losses change if the macroeconomy followed an adverse counterfactual path? Yet standard practice remains predictive and might be therefore vulnerable to omitted-variable…
Benford's law is often used as a support to critical decisions related to data quality or the presence of data manipulations or even fraud. However, many authors argue that conventional statistical tests will reject the null of data…
Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the…
The standard A/B testing approaches are mostly based on t-test in large scale industry applications. These standard approaches however suffers from low statistical power in business settings, due to nature of small sample-size or…
It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out-of-sample performance can be hampered when predictors…
Statistical significance testing of differences in values of metrics like recall, precision and balanced F-score is a necessary part of empirical natural language processing. Unfortunately, we find in a set of experiments that many commonly…
Anecdotally, using an estimated propensity score is superior to the true propensity score in estimating the average treatment effect based on observational data. However, this claim comes with several qualifications: it holds only if…
There is significant interest in developing evaluation metrics which accurately estimate the quality of generated text without the aid of a human-written reference text, which can be time consuming and expensive to collect or entirely…
When testing multiple hypothesis in a survey --e.g. many different source locations, template waveforms, and so on-- the final result consists in a set of confidence intervals, each one at a desired confidence level. But the probability…
Data fusion techniques integrate information from heterogeneous data sources to improve learning, generalization, and decision making across data sciences. In causal inference, these methods leverage rich observational data to improve…
Subjects in clinical studies that investigate paired body parts can carry a disease on either both sides (bilateral) or a single side (unilateral) of the organs. Data in such studies may consist of both bilateral and unilateral records.…
A myriad of measures to illustrate performance of predictive artificial intelligence (AI) models have been proposed in the literature. Selecting appropriate performance measures is essential for predictive AI models that are developed to be…
U.S. state education agencies mark schools displaying achievement gaps between demographic subgroups as needing improvement. Some schools may have few students in these subgroups, such that average end-of-year test scores only noisily…
There are racial disparities in the COVID-19 test positivity rate, suggesting that minorities may be under-tested. Here, drawing on the literature on statistically assessing racial disparities in policing, we 1) illuminate a statistical…
Overconfidence is a common issue for deep neural networks, limiting their deployment in real-world applications. To better estimate confidence, existing methods mostly focus on fully-supervised scenarios and rely on training labels. In this…
There have been reports of correlation between estimates of prevalence and test accuracy across studies included in diagnostic meta-analyses. It has been hypothesized that this unexpected association arises because of certain biases…