Improving Methodologies for Agentic Evaluations Across Domains: Leakage of Sensitive Information, Fraud and Cybersecurity Threats
Abstract
The rapid rise of autonomous AI systems and advancements in agent capabilities are introducing new risks due to reduced oversight of real-world interactions. Yet agent testing remains nascent and is still a developing science. As AI agents begin to be deployed globally, it is important that they handle different languages and cultures accurately and securely. To address this, participants from The International Network for Advanced AI Measurement, Evaluation and Science, including representatives from Singapore, Japan, Australia, Canada, the European Commission, France, Kenya, South Korea, and the United Kingdom have come together to align approaches to agentic evaluations. This is the third exercise, building on insights from two earlier joint testing exercises conducted by the Network in November 2024 and February 2025. The objective is to further refine best practices for testing advanced AI systems. The exercise was split into two strands: (1) common risks, including leakage of sensitive information and fraud, led by Singapore AISI; and (2) cybersecurity, led by UK AISI. A mix of open and closed-weight models were evaluated against tasks from various public agentic benchmarks. Given the nascency of agentic testing, our primary focus was on understanding methodological issues in conducting such tests, rather than examining test results or model capabilities. This collaboration marks an important step forward as participants work together to advance the science of agentic evaluations.
Keywords
Cite
@article{arxiv.2601.15679,
title = {Improving Methodologies for Agentic Evaluations Across Domains: Leakage of Sensitive Information, Fraud and Cybersecurity Threats},
author = {Ee Wei Seah and Yongsen Zheng and Naga Nikshith and Mahran Morsidi and Gabriel Waikin Loh Matienzo and Nigel Gay and Akriti Vij and Benjamin Chua and En Qi Ng and Sharmini Johnson and Vanessa Wilfred and Wan Sie Lee and Anna Davidson and Catherine Devine and Erin Zorer and Gareth Holvey and Harry Coppock and James Walpole and Jerome Wynee and Magda Dubois and Michael Schmatz and Patrick Keane and Sam Deverett and Bill Black and Bo Yan and Bushra Sabir and Frank Sun and Hao Zhang and Harriet Farlow and Helen Zhou and Lingming Dong and Qinghua Lu and Seung Jang and Sharif Abuadbba and Simon O'Callaghan and Suyu Ma and Tom Howroyd and Cyrus Fung and Fatemeh Azadi and Isar Nejadgholi and Krishnapriya Vishnubhotla and Pulei Xiong and Saeedeh Lohrasbi and Scott Buffett and Shahrear Iqbal and Sowmya Vajjala and Anna Safont-Andreu and Luca Massarelli and Oskar van der Wal and Simon Möller and Agnes Delaborde and Joris Duguépéroux and Nicolas Rolin and Romane Gallienne and Sarah Behanzin and Tom Seimandi and Akiko Murakami and Takayuki Semitsu and Teresa Tsukiji and Angela Kinuthia and Michael Michie and Stephanie Kasaon and Jean Wangari and Hankyul Baek and Jaewon Noh and Kihyuk Nam and Sang Seo and Sungpil Shin and Taewhi Lee and Yongsu Kim},
journal= {arXiv preprint arXiv:2601.15679},
year = {2026}
}
Comments
The author/contributor list organises contributors by country and alphabetical order within each country. In some places, the order has been altered to match other related publications