English

Can REF output quality scores be assigned by AI? Experimental evidence

Computers and Society 2022-12-16 v1 Computation and Language General Literature Information Retrieval Machine Learning
Authors: Mike Thelwall , Kayvan Kousha , Mahshid Abdoli , Emma Stuart , Meiko Makita , Paul Wilson , Jonathan Levitt
View on arXiv ↗ PDF ↗

Abstract

This document describes strategies for using Artificial Intelligence (AI) to predict some journal article scores in future research assessment exercises. Five strategies have been assessed.

Keywords

peer review artificial intelligence evaluation metrics

Cite

@article{arxiv.2212.08041,
  title  = {Can REF output quality scores be assigned by AI? Experimental evidence},
  author = {Mike Thelwall and Kayvan Kousha and Mahshid Abdoli and Emma Stuart and Meiko Makita and Paul Wilson and Jonathan Levitt},
  journal= {arXiv preprint arXiv:2212.08041},
  year   = {2022}
}

Related papers

View all related →
Digital Libraries · Computer Science
Artificial intelligence technologies to support research assessment: A review
Kayvan Kousha, Mike Thelwall
2022-12-14
Artificial Intelligence · Computer Science
Understanding the Impact of Artificial Intelligence in Academic Writing: Metadata to the Rescue
Javier Conde, Pedro Reviriego, Joaquín Salvachúa, Gonzalo Martínez +2
2025-02-25
Artificial Intelligence · Computer Science
AI Knowledge and Reasoning: Emulating Expert Creativity in Scientific Research
Anirban Mukherjee, Hannah Hanwen Chang
2024-04-09
Digital Libraries · Computer Science
An AI aid to the editors. Exploring the possibility of an AI assisted article classification system
Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, Sriparna Saha +1
2018-02-20
Computers and Society · Computer Science
Reducing research bureaucracy in UK higher education: Can generative AI assist with the internal evaluation of quality?
Gordon Fletcher, Saomai Vu Khan, Aldus Greenhill Fletcher
2025-12-01
Computation and Language · Computer Science
What do writing features tell us about AI papers?
Zining Zhu, Bai Li, Yang Xu, Frank Rudzicz
2021-07-15
Information Retrieval · Computer Science
Artificial intelligence to automate the systematic review of scientific literature
José de la Torre-López, Aurora Ramírez, José Raúl Romero
2024-01-23
Computers and Society · Computer Science
Domain-Specific Evaluation Strategies for AI in Journalism
Sachita Nishal, Charlotte Li, Nicholas Diakopoulos
2024-03-27
Digital Libraries · Computer Science
Predicting article quality scores with machine learning: The UK Research Excellence Framework
Mike Thelwall, Kayvan Kousha, Mahshid Abdoli, Emma Stuart +5
2023-07-31
Digital Libraries · Computer Science
A Machine Learning Based Approach to Categorize Research Journals
Rabia Shabbir Ranjha, Arshad Ali, Shahid Yousaf
2022-10-07
Artificial Intelligence · Computer Science
Artificial Intelligence for Literature Reviews: Opportunities and Challenges
Francisco Bolanos, Angelo Salatino, Francesco Osborne, Enrico Motta
2024-08-07
Artificial Intelligence · Computer Science
Towards AI-assisted Academic Writing
Daniel J. Liebling, Malcolm Kane, Madeleine Grunde-Mclaughlin, Ian J. Lang +2
2025-03-19
Machine Learning · Computer Science
Automated Content Grading Using Machine Learning
Rahul Kr Chauhan, Ravinder Saharan, Siddhartha Singh, Priti Sharma
2020-04-21
Applications · Statistics
How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review
Buxin Su, Natalie Collina, Garrett Wen, Didong Li +4
2025-11-26
Computers and Society · Computer Science
Classification Schemas for Artificial Intelligence Failures
Peter J. Scott, Roman V. Yampolskiy
2019-07-19
Computers and Society · Computer Science
Why can't Epidemiology be automated (yet)?
David Bann, Ed Lowther, Liam Wright, Yevgeniya Kovalchuk
2025-07-22
Computers and Society · Computer Science
Assisting the Grading of a Handwritten General Chemistry Exam with Artificial Intelligence
Jan Cvengros, Gerd Kortemeyer
2025-11-11
Artificial Intelligence · Computer Science
Predicting the Future of AI with AI: High-quality link prediction in an exponentially growing knowledge network
Mario Krenn, Lorenzo Buffoni, Bruno Coutinho, Sagi Eppel +12
2023-12-04
Computation and Language · Computer Science
Exploring AI-Generated Text in Student Writing: How Does AI Help?
David James Woo, Hengky Susanto, Chi Ho Yeung, Kai Guo +1
2024-06-17
Artificial Intelligence · Computer Science
Accelerating science with human-aware artificial intelligence
Jamshid Sourati, James Evans
2023-06-05
Digital Libraries · Computer Science
Textual analysis of artificial intelligence manuscripts reveals features associated with peer review outcome
Philippe Vincent-Lamarre, Vincent Larivière
2020-03-05
Software Engineering · Computer Science
AI-Assisted Requirements Engineering: An Empirical Evaluation Relative to Expert Judgment
Oz Levy, Ilya Dikman, Natan Levy, Michael Winokur
2026-04-17
Digital Libraries · Computer Science
Identifying the Development and Application of Artificial Intelligence in Scientific Text
James Dunham, Jennifer Melot, Dewey Murdick
2020-05-29
Computers and Society · Computer Science
Transforming Student Evaluation with Adaptive Intelligence and Performance Analytics
Pushpalatha K S, Abhishek Mangalur, Ketan Hegde, Chetan Badachi +1
2025-03-10
Astrophysics · Physics
AI Gamma-Ray Burst Classification: Methodology/Preliminary Results
Jon Hakkila, David J. Haglin, Richard J. Roiger, Robert S. Mallozzi +2
2009-10-30
R2 v1 2026-06-28T07:37:24.931Z