Ontology evaluation through functional requirements, such as testing via competency question (CQ) verification, is a well-established yet costly, labour-intensive, and error-prone endeavour, even for ontology engineering experts. In this work, we introduce OE-Assist, a novel framework designed to assist ontology evaluation through automated and semi-automated CQ verification. By presenting and leveraging a dataset of 1,393 CQs paired with corresponding ontologies and ontology stories, our contributions present, to our knowledge, the first systematic investigation into large language model (LLM)-assisted ontology evaluation, and include: (i) evaluating the effectiveness of a LLM-based approach for automatically performing CQ verification against a manually created gold standard, and (ii) developing and assessing an LLM-powered framework to assist CQ verification with Prot\'eg\'e, by providing suggestions. We found that automated LLM-based evaluation with o1-preview and o3-mini perform at a similar level to the average user's performance.
@article{arxiv.2507.14552,
title = {Large Language Models Assisting Ontology Evaluation},
author = {Anna Sofia Lippolis and Mohammad Javad Saeedizade and Robin Keskisärkkä and Aldo Gangemi and Eva Blomqvist and Andrea Giovanni Nuzzolese},
journal= {arXiv preprint arXiv:2507.14552},
year = {2026}
}