Related papers: Streamlining Evaluation with ir-measures
This paper introduces LLAssist, an open-source tool designed to streamline literature reviews in academic research. In an era of exponential growth in scientific publications, researchers face mounting challenges in efficiently processing…
The recent increase in the volume of online meetings necessitates automated tools for managing and organizing the material, especially when an attendee has missed the discussion and needs assistance in quickly exploring it. In this work, we…
Bank supervisors face the complex task of ensuring that new measures are consistently aligned with historical precedents. To address this challenge, we introduce a novel Information Retrieval (IR) System tailored to assist supervisors in…
We describe the development and implementation of research-based learning tools such as the Quantum Interactive Learning Tutorials (QuILTs) and peer instruction tools to reduce students' common difficulties with issues related to…
Evaluation of large language model (LLM) outputs requires users to make critical judgments about the best outputs across various configurations. This process is costly and takes time given the large amounts of data. LLMs are increasingly…
Large Language Models (LLMs) have become indispensable for evaluating writing. However, text feedback they provide is often unintelligible, generic, and not specific to user criteria. Inspired by structured rubrics in education and…
Rational Tracer (Ratracer) is a tool to simplify complicated arithmetic expressions using modular arithmetics and rational function reconstruction, with the main idea of separating the construction of expressions (via tracing, i.e.…
Evaluation of information retrieval systems (IRS) is a prominent topic among information retrieval researchers--mainly directed at a general population. Children require unique IRS and by extension different ways to evaluate these systems,…
Hierarchical classification addresses the problem of classifying items into a hierarchy of classes. An important issue in hierarchical classification is the evaluation of different classification algorithms, which is complicated by the…
Allocation of research funding, as well as promotion and tenure decisions, are increasingly made using indicators and impact factors drawn from citations to published work. A debate among scientometricians about proper normalization of…
Information Retrieval (IR) methods aim to identify documents relevant to a query, which have been widely applied in various natural language tasks. However, existing approaches typically consider only the textual content within documents,…
Recently, it was shown that most popular IR measures are not interval-scaled, implying that decades of experimental IR research used potentially improper methods, which may have produced questionable results. However, it was unclear if and…
Information Retrieval (IR) systems are designed to deliver relevant content, but traditional systems may not optimize rankings for fairness, neutrality, or the balance of ideas. Consequently, IR can often introduce indexical biases, or…
Conducting research often involves managing multiple disconnected tools for survey design, data collection, response analysis, and report generation, leading to inefficiencies, increased error risks, and challenges in ensuring…
Recent information retrieval (IR) models are pre-trained and instruction-tuned on massive datasets and tasks, enabling them to perform well on a wide range of tasks and potentially generalize to unseen tasks with instructions. However,…
In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations. We begin with the…
Information retrieval (IR) is a user approach to obtain relevant information which meets needs with the help of a IR system (IRS). However, the IRS shows certain differences between user relevance and system relevance. These gaps are…
Information integration plays a pivotal role in biomedical studies by facilitating the combination and analysis of independent datasets from multiple studies, thereby uncovering valuable insights that might otherwise remain obscured due to…
Citation and publication profiles are gaining importance for the evaluation of top researchers when it comes to the appropriation of funding for excellence programs or career promotion judgments. Indicators like the Normalized Mean Citation…
The rapid advancement in capabilities of large language models (LLMs) raises a pivotal question: How can LLMs accelerate scientific discovery? This work tackles the crucial first stage of research, generating novel hypotheses. While recent…