Ego-Exo4D：从第一人称与第三人称视角理解熟练人类活动
计算机视觉与模式识别
2024-09-27 v4 人工智能
摘要
我们提出 Ego-Exo4D，一个多样化的大规模多模态多视角视频数据集与基准挑战。Ego-Exo4D 围绕同时采集的熟练人类活动（如体育、音乐、舞蹈、自行车修理）的自我中心与异中心视频展开。来自全球 13 个城市的 740 名参与者在 123 个不同自然场景语境中进行了这些活动，每次产生 1 至 42 分钟的长时段采集，合计 1,286 小时视频。该数据集的多模态程度前所未有：视频配有声道音频、视线注视、3D 点云、相机位姿、IMU 以及多组配对语言描述——包括由教练与教师完成且针对熟练活动领域定制的新型“专家解说”。为推进熟练人类活动的第一人称视频理解前沿，我们还提出一套基准任务及其标注，包括细粒度活动理解、熟练度估计、跨视角转换以及 3D 手/身体姿态。所有资源均已开源以推动社区新研究。项目主页：http://ego-exo4d-data.org/
引用
@article{arxiv.2311.18259,
title = {Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives},
author = {Kristen Grauman and Andrew Westbury and Lorenzo Torresani and Kris Kitani and Jitendra Malik and Triantafyllos Afouras and Kumar Ashutosh and Vijay Baiyya and Siddhant Bansal and Bikram Boote and Eugene Byrne and Zach Chavis and Joya Chen and Feng Cheng and Fu-Jen Chu and Sean Crane and Avijit Dasgupta and Jing Dong and Maria Escobar and Cristhian Forigua and Abrham Gebreselasie and Sanjay Haresh and Jing Huang and Md Mohaiminul Islam and Suyog Jain and Rawal Khirodkar and Devansh Kukreja and Kevin J Liang and Jia-Wei Liu and Sagnik Majumder and Yongsen Mao and Miguel Martin and Effrosyni Mavroudi and Tushar Nagarajan and Francesco Ragusa and Santhosh Kumar Ramakrishnan and Luigi Seminara and Arjun Somayazulu and Yale Song and Shan Su and Zihui Xue and Edward Zhang and Jinxu Zhang and Angela Castillo and Changan Chen and Xinzhu Fu and Ryosuke Furuta and Cristina Gonzalez and Prince Gupta and Jiabo Hu and Yifei Huang and Yiming Huang and Weslie Khoo and Anush Kumar and Robert Kuo and Sach Lakhavani and Miao Liu and Mi Luo and Zhengyi Luo and Brighid Meredith and Austin Miller and Oluwatumininu Oguntola and Xiaqing Pan and Penny Peng and Shraman Pramanick and Merey Ramazanova and Fiona Ryan and Wei Shan and Kiran Somasundaram and Chenan Song and Audrey Southerland and Masatoshi Tateno and Huiyu Wang and Yuchen Wang and Takuma Yagi and Mingfei Yan and Xitong Yang and Zecheng Yu and Shengxin Cindy Zha and Chen Zhao and Ziwei Zhao and Zhifan Zhu and Jeff Zhuo and Pablo Arbelaez and Gedas Bertasius and David Crandall and Dima Damen and Jakob Engel and Giovanni Maria Farinella and Antonino Furnari and Bernard Ghanem and Judy Hoffman and C. V. Jawahar and Richard Newcombe and Hyun Soo Park and James M. Rehg and Yoichi Sato and Manolis Savva and Jianbo Shi and Mike Zheng Shou and Michael Wray},
journal= {arXiv preprint arXiv:2311.18259},
year = {2024}
}
备注
Expanded manuscript (compared to arxiv v1 from Nov 2023 and CVPR 2024 paper from June 2024) for more comprehensive dataset and benchmark presentation, plus new results on v2 data release