Inferring Capabilities from Task Performance with Bayesian Triangulation
Abstract
As machine learning models become more general, we need to characterise them in richer, more meaningful ways. We describe a method to infer the cognitive profile of a system from diverse experimental data. To do so, we introduce measurement layouts that model how task-instance features interact with system capabilities to affect performance. These features must be triangulated in complex ways to be able to infer capabilities from non-populational data -- a challenge for traditional psychometric and inferential tools. Using the Bayesian probabilistic programming library PyMC, we infer different cognitive profiles for agents in two scenarios: 68 actual contestants in the AnimalAI Olympics and 30 synthetic agents for O-PIAAGETS, an object permanence battery. We showcase the potential for capability-oriented evaluation.
Cite
@article{arxiv.2309.11975,
title = {Inferring Capabilities from Task Performance with Bayesian Triangulation},
author = {John Burden and Konstantinos Voudouris and Ryan Burnell and Danaja Rutar and Lucy Cheke and José Hernández-Orallo},
journal= {arXiv preprint arXiv:2309.11975},
year = {2025}
}