无缝交互：二人式音视频运动建模与大规模数据集
计算机视觉与模式识别
2025-07-02 v2 人工智能
摘要
人类交流涉及语言与非语言信号之间的复杂相互作用，这些信号对于传递意义和实现人际目标至关重要。为了发展具有社交智能的 AI 技术, 必须开发能够理解和生成二人式行为动态的模型。为此, 我们引入了无缝交互数据集(Seamless Interaction Dataset), 这是一个包含超过 4000 小时、来自 4000 多名来自多样化情境的面对面互动影像的大规模数据集。该数据集支持开发理解二人式具身动态的 AI 技术, 为虚拟智能体、远程存在体验和多模态内容分析工具的突破提供了可能。我们还开发了一套模型, 利用该数据集生成与人类语言一致的二人式动作手势和面部表情。这些模型可以同时接受来自交谈者的语言和视觉行为作为输入。我们 presenting 的变体使用来自大语言模型(LLM)的语音, 并集成了 2D 和 3D 渲染方法, 为实现交互式虚拟智能体迈进了重要一步。此外, 我们描述了可控变体的运动模型, 能够调整情感反应和表达程度, 以及生成更语义相关的手势。最后, 我们讨论了评估这些二人式运动模型质量的方法, 这些模型正在展示出更直观和响应式的人机交互潜力。
引用
@article{arxiv.2506.22554,
title = {Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset},
author = {Vasu Agrawal and Akinniyi Akinyemi and Kathryn Alvero and Morteza Behrooz and Julia Buffalini and Fabio Maria Carlucci and Joy Chen and Junming Chen and Zhang Chen and Shiyang Cheng and Praveen Chowdary and Joe Chuang and Antony D'Avirro and Jon Daly and Ning Dong and Mark Duppenthaler and Cynthia Gao and Jeff Girard and Martin Gleize and Sahir Gomez and Hongyu Gong and Srivathsan Govindarajan and Brandon Han and Sen He and Denise Hernandez and Yordan Hristov and Rongjie Huang and Hirofumi Inaguma and Somya Jain and Raj Janardhan and Qingyao Jia and Christopher Klaiber and Dejan Kovachev and Moneish Kumar and Hang Li and Yilei Li and Pavel Litvin and Wei Liu and Guangyao Ma and Jing Ma and Martin Ma and Xutai Ma and Lucas Mantovani and Sagar Miglani and Sreyas Mohan and Louis-Philippe Morency and Evonne Ng and Kam-Woh Ng and Tu Anh Nguyen and Amia Oberai and Benjamin Peloquin and Juan Pino and Jovan Popovic and Omid Poursaeed and Fabian Prada and Alice Rakotoarison and Rakesh Ranjan and Alexander Richard and Christophe Ropers and Safiyyah Saleem and Vasu Sharma and Alex Shcherbyna and Jia Shen and Jie Shen and Anastasis Stathopoulos and Anna Sun and Paden Tomasello and Tuan Tran and Arina Turkatenko and Bo Wan and Chao Wang and Jeff Wang and Mary Williamson and Carleigh Wood and Tao Xiang and Yilin Yang and Julien Yao and Chen Zhang and Jiemin Zhang and Xinyue Zhang and Jason Zheng and Pavlo Zhyzheria and Jan Zikes and Michael Zollhoefer},
journal= {arXiv preprint arXiv:2506.22554},
year = {2025}
}