Related papers: Caveats for using statistical significance tests i…
We investigate a growing body of work that seeks to improve recommender systems through the use of review text. Generally, these papers argue that since reviews 'explain' users' opinions, they ought to be useful to infer the underlying…
Contemporary statistical publications rely on simulation to evaluate performance of new methods and compare them with established methods. In the context of meta-analysis of log-odds-ratios, we investigate how the ways in which simulations…
We consider the conditional randomization test as a way to account for covariate imbalance in randomized experiments. The test accounts for covariate imbalance by comparing the observed test statistic to the null distribution of the test…
Statistical hypothesis testing, as formalized by 20th Century statisticians and taught in college statistics courses, has been a cornerstone of 100 years of scientific progress. Nevertheless, the methodology is increasingly questioned in…
Given the growing use of impact metrics in the evaluation of scholars, journals, academic institutions, and even countries, there is a critical need for means to compare scientific impact across disciplinary boundaries. Unfortunately,…
Rank-based inference methods are applied in various disciplines, typically when procedures relying on standard normal theory are not justifiable, for example when data are not symmetrically distributed, contain outliers, or responses are…
The tension between qualitative theorizing and quantitative methods is pervasive in the social sciences, and poses a constant challenge to empirical research. But in science studies as an interdisciplinary specialty, there are additional…
Purpose: We address the question of whether differences are statistically significant in the rankings of universities. We propose methods measuring the statistical significance among different universities and illustrate the results by…
Comparisons of different treatments or production processes are the goals of a significant fraction of applied research. Unsurprisingly, two-sample problems play a main role in Statistics through natural questions such as `Is the the new…
Factorial designs are frequently used in different fields of science, e.g. psychological, medical or biometric studies. Standard approaches, as the ANOVA $F$-test, make different assumptions on the distribution of the error terms, the…
Controversy about the significance of underdetermination of theories persists in the philosophy and conduct of science. The issue has practical import when research is used to inform decision making, because scientific uncertainty yields…
Statistical models have seen a significant rise in popularity in recent years. Despite their undeniable success in various industry use cases such as sabermetrics, investment portfolio management, and artificial intelligence, there has been…
Most researchers acknowledge an intrinsic hierarchy in the scholarly journals ('journal rank') that they submit their work to, and adjust not only their submission but also their reading strategies accordingly. On the other hand, much has…
The development of scientometric indicators and methods for evaluative purposes, requires a multitude of assumptions, conventions, limitations, and caveats. Given this, we cannot permit ambiguities in the key concepts forming the basis of…
For testing the statistical significance of a treatment effect, we usually compare between two parts of a population, one is exposed to the treatment, and the other is not exposed to it. Standard parametric and nonparametric two-sample…
Following discussions in 2010 and 2011, scientometric evaluators have increasingly abandoned relative indicators in favor of comparing observed with expected citation ratios. The latter method provides parameters with error values allowing…
The experimental evaluation of the methods and concepts covered in software engineering has been increasingly valued. This value indicates the constant search for new forms of assessment and validation of the results obtained in Software…
Score reliability is necessary for establishing a validity argument for an instrument, and is therefore highly important to investigate. Depending on the proposed instrument use and score interpretations, differing degrees of precision in…
Context: Several researchers have reported their experiences in applying secondary studies (Systematic Literature Reviews - SLRs and Systematic Mappings - SMs) in Software Engineering (SE). However, there is still a lack of studies…
When interpreting A/B tests, we typically focus only on the statistically significant results and take them by face value. This practice, termed post-selection inference in the statistical literature, may negatively affect both point…