Related papers: Systematic Analysis of Experiment Precision Measur…
Evaluation is the foundation of empirical science, yet the evaluation of evaluation itself -- so-called meta-evaluation -- remains strikingly underdeveloped. While methods such as observational studies, design of experiments (DoE), and…
In mobile computing systems, users can access network services anywhere and anytime using mobile devices such as tablets and smart phones. These devices connect to the Internet via network or telecommunications operators. Users usually have…
Fairness is an emerging and challenging topic in recommender systems. In recent years, various ways of evaluating and therefore improving fairness have emerged. In this study, we examine existing evaluation measures of fairness in…
Agreeing suitability for purpose and procurement decisions depend on assessment of real or simulated performances of sonar systems against user requirements for particular scenarios. There may be multiple pertinent aspects of performance…
Interpreting experimental data in high school experiments can be a difficult task for students, especially when there is large variation in the data. At the same time, calculating the standard deviation poses a challenge for students. In…
Matching for causal inference is a well-studied problem, but standard methods fail when the units to match are text documents: the high-dimensional and rich nature of the data renders exact matching infeasible, causes propensity scores to…
Evaluating the performance of Large Language Models (LLMs) is a critical yet challenging task, particularly when aiming to avoid subjective assessments. This paper proposes a framework for leveraging subjective metrics derived from the…
Test automation is important in software industry but self-assessment instruments for assessing its maturity are not sufficient. The two objectives of this study are to synthesize what an organization should focus to assess its test…
In this paper we investigate the question of how much combined measurements can increase the accuracy of additive quantities. Therefore, we consider a set of measurements from a selection of all possible combinations of the $n$ labeled…
Several technological and scientific advances have been achieved recently in the fields of immersive systems, which are offering new possibilities to applications and services in different communication domains, such as entertainment,…
Measurement uncertainty and experimental error are important concepts taught in undergraduate physics laboratories. Although student ideas about error and uncertainty in introductory classical mechanics lab experiments have been studied…
User's perception of product, by essence subjective, is a major topic in marketing and industrial design. Many methods, based on users' tests, are used so as to characterise this perception. We are interested in three main methods:…
Failure to accurately measure the outcomes of an experiment can lead to bias and incorrect conclusions. Online controlled experiments (aka AB tests) are increasingly being used to make decisions to improve websites as well as mobile and…
How do we evaluate experiences in immersive environments? Despite decades of research in immersive technologies such as virtual reality, the field remains fragmented. Studies rely on overlapping constructs, heterogeneous instruments, and…
Explanation is a fundamentally human process. Understanding the goal and audience of the explanation is vital, yet existing work on explainable reinforcement learning (XRL) routinely does not consult humans in their evaluations. Even when…
I would like to share recommendations on how to do performance benchmarks for the purpose of computer science research evaluation. Research in my field (programming language research) often involves performance considerations, but it is…
We study user sentiment (reported via optional surveys) as a metric for fully randomized A/B tests. Both user-level covariates and treatment assignment can impact response propensity. We propose a set of consistent estimators for the…
Measure user experience in MR (i.e., AR/VR) user studies is essential. Researchers apply a wide range of measuring methods using objective (e.g., biosignals, time logging), behavioral (e.g., gaze direction, movement amplitude), and…
This study developed a new statistical model and method for analyzing the precision of binary measurement methods from collaborative studies. The model is based on beta-binomial distributions. In other words, it assumes that the sensitivity…
Planning quality assurance (QA) activities in a systematic way and controlling their execution are challenging tasks for companies that develop software or software-intensive systems. Both require estimation capabilities regarding the…