MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills
Abstract
Background: Agent skills are increasingly deployed as modular, reusable capability units in AI agent systems. Medical research agent skills require safeguards beyond general-purpose evaluation, including scientific integrity, methodological validity, reproducibility, and boundary safety. This study developed and preliminarily evaluated a domain-specific audit framework for medical research agent skills, with a focus on reliability against expert review. Methods: We developed MedSkillAudit ([email protected]), a layered framework assessing skill release readiness before deployment. We evaluated 75 skills across five medical research categories (15 per category). Two experts independently assigned a quality score (0-100), an ordinal release disposition (Production Ready / Limited Release / Beta Only / Reject), and a high-risk failure flag. System-expert agreement was quantified using ICC(2,1) and linearly weighted Cohen's kappa, benchmarked against the human inter-rater baseline. Results: The mean consensus quality score was 72.4 (SD = 13.0); 57.3% of skills fell below the Limited Release threshold. MedSkillAudit achieved ICC(2,1) = 0.449 (95% CI: 0.250-0.610), exceeding the human inter-rater ICC of 0.300. System-consensus score divergence (SD = 9.5) was smaller than inter-expert divergence (SD = 12.4), with no directional bias (Wilcoxon p = 0.613). Protocol Design showed the strongest category-level agreement (ICC = 0.551); Academic Writing showed a negative ICC (-0.567), reflecting a structural rubric-expert mismatch. Conclusions: Domain-specific pre-deployment audit may provide a practical foundation for governing medical research agent skills, complementing general-purpose quality checks with structured audit workflows tailored to scientific use cases.
Keywords
Cite
@article{arxiv.2604.20441,
title = {MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills},
author = {Yingyong Hou and Xinyuan Lao and Huimei Wang and Qianyu Yao and Wei Chen and Bocheng Huang and Fei Sun and Yuxian Lv and Weiqi Lei and Xueqian Wen and Pengfei Xia and Zhujun Tan and Shengyang Xie},
journal= {arXiv preprint arXiv:2604.20441},
year = {2026}
}
Comments
20 pages, 9 figures, 1 graphic abstract, 4 tables