Related papers: Choose Your Own Adventure: Developing A Values-Ori…
Academic publishing requires solving a collective coordination problem: among thousands of possible publication venues, which deserve a community's attention? A clear consensus helps scholars allocate attention, match submissions to…
Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without arguments, and blurry terminology invites misconceptions. For…
Assigning qualified, unbiased and interested reviewers to paper submissions is vital for maintaining the integrity and quality of the academic publishing system and providing valuable reviews to authors. However, matching thousands of…
Coauthors (CA) of a "lead investigator" (LI) can receive a rank (r) according to their "importance" in having published joint publications with the LI. It is commonly accepted, without any proof, that publications in peer review journals…
Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing papers and making the final acceptance/rejection decisions. Given that human decision making is known to be…
This paper shows how bibliometric assessment can be implemented at individual level. This has been successfully done at the University of Vienna carried out by the Bibliometrics and Publication Strategies Department of the Vienna University…
In recent years, assessing the performance of researchers has become a burden due to the extensive volume of the existing research output. As a result, evaluators often end up relying heavily on a selection of performance indicators like…
Peer reviewing is a central process in modern research and essential for ensuring high quality and reliability of published work. At the same time, it is a time-consuming process and increasing interest in emerging fields often results in a…
Evaluating large language models typically relies on human-authored benchmarks, reference answers, and human or single-model judgments, approaches that scale poorly, become quickly outdated, and mismatch open-world deployments that depend…
Scientific research needs a system that better values rigorous, reusable contributions. Although open knowledge and FAIR (findable, accessible, interoperable, and reusable) principles, along with coalitions and infrastructures, are…
The road to becoming an astronomer is exciting, but often fraught with danger and conflicting messages. A PhD student is inundated with catch-phrases such as "publish or perish" and "it's not about the quantity, but the quality of work".…
Entity rankings (e.g., institutions, journals) are a core component of academia and related industries. Existing approaches to institutional rankings have relied on a variety of data sources, and approaches to computing outcomes, but remain…
University or college rankings have almost become an industry of their own, published by US News \& World Report (USNWR) and similar organizations. Most of the rankings use a similar scheme: Rank universities in decreasing score order,…
Citation recommendation describes the task of recommending citations for a given text. Due to the overload of published scientific works in recent years on the one hand, and the need to cite the most appropriate publications when writing…
This work is a preliminary exploratory study of how we could progress a step towards an AI assisted article classification sys- tem in academia. The proposed system aims to aid the journal editors in their decisions by pinpointing the…
Public opinion on recommender systems has become increasingly wary in recent years. In line with this trend, lawmakers have also started to become more critical of such systems, resulting in the introduction of new laws focusing on aspects…
Much recent work seeks to evaluate values and opinions in large language models (LLMs) using multiple-choice surveys and questionnaires. Most of this work is motivated by concerns around real-world LLM applications. For example,…
The impact factor (IF) of scientific journals has acquired a major role in the evaluations of the output of scholars, departments and whole institutions. Typically papers appearing in journals with large values of the IF receive a high…
Rankings are ubiquitous in the online world today. As we have transitioned from finding books in libraries to ranking products, jobs, job applicants, opinions and potential romantic partners, there is a substantial precedent that ranking…
Evaluating the quality of academic journal is becoming increasing important within the context of research performance evaluation. Traditionally, journals have been ranked by peer review lists such as that of the Association of Business…