We introduce MABe22, a large-scale, multi-agent video and trajectory benchmark to assess the quality of learned behavior representations. This dataset is collected from a variety of biology experiments, and includes triplets of interacting mice (4.7 million frames video+pose tracking data, 10 million frames pose only), symbiotic beetle-ant interactions (10 million frames video data), and groups of interacting flies (4.4 million frames of pose tracking data). Accompanying these data, we introduce a panel of real-life downstream analysis tasks to assess the quality of learned representations by evaluating how well they preserve information about the experimental conditions (e.g. strain, time of day, optogenetic stimulation) and animal behavior. We test multiple state-of-the-art self-supervised video and trajectory representation learning methods to demonstrate the use of our benchmark, revealing that methods developed using human action datasets do not fully translate to animal datasets. We hope that our benchmark and dataset encourage a broader exploration of behavior representation learning methods across species and settings.
@article{arxiv.2207.10553,
title = {MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior},
author = {Jennifer J. Sun and Markus Marks and Andrew Ulmer and Dipam Chakraborty and Brian Geuther and Edward Hayes and Heng Jia and Vivek Kumar and Sebastian Oleszko and Zachary Partridge and Milan Peelman and Alice Robie and Catherine E. Schretter and Keith Sheppard and Chao Sun and Param Uttarwar and Julian M. Wagner and Eric Werner and Joseph Parker and Pietro Perona and Yisong Yue and Kristin Branson and Ann Kennedy},
journal= {arXiv preprint arXiv:2207.10553},
year = {2023}
}
Comments
To appear in ICML 2023, Project website: https://sites.google.com/view/computational-behavior/our-datasets/mabe2022-dataset