Movie Gen：面向媒体生成的基础模型集合
计算机视觉与模式识别
2025-02-27 v2 人工智能
机器学习
图像与视频处理
摘要
我们 presented Movie Gen，这是一组基础模型，能够生成不同宽高比并同步音频的高质量 1080p HD 视频。我们还展示了额外的能力，包括基于精确指令的视频编辑和根据用户图像生成个性化视频。我们的模型在多个任务上达到了新的状态-of-the-art，包括文本到视频合成、视频个性化、视频编辑、视频到音频生成以及文本到音频生成。我们最大的视频生成模型是一个拥有 300 亿参数的 transformer，训练时最大上下文长度为 73K 视频 token，对应生成时长为 16 秒、帧率为 16 fps 的视频。我们展示了在架构、潜在空间、训练目标和配方、数据 curated、评估协议、并行技术以及推理优化等方面的多项技术创新和简化，使我们能够从放大预训练数据、模型规模和训练计算的优势中获益。我们希望本论文能帮助研究社区加快媒体生成模型的进展和创新。本论文的所有视频均可在 https://go.fb.me/MovieGenResearchVideos 访问。
引用
@article{arxiv.2410.13720,
title = {Movie Gen: A Cast of Media Foundation Models},
author = {Adam Polyak and Amit Zohar and Andrew Brown and Andros Tjandra and Animesh Sinha and Ann Lee and Apoorv Vyas and Bowen Shi and Chih-Yao Ma and Ching-Yao Chuang and David Yan and Dhruv Choudhary and Dingkang Wang and Geet Sethi and Guan Pang and Haoyu Ma and Ishan Misra and Ji Hou and Jialiang Wang and Kiran Jagadeesh and Kunpeng Li and Luxin Zhang and Mannat Singh and Mary Williamson and Matt Le and Matthew Yu and Mitesh Kumar Singh and Peizhao Zhang and Peter Vajda and Quentin Duval and Rohit Girdhar and Roshan Sumbaly and Sai Saketh Rambhatla and Sam Tsai and Samaneh Azadi and Samyak Datta and Sanyuan Chen and Sean Bell and Sharadh Ramaswamy and Shelly Sheynin and Siddharth Bhattacharya and Simran Motwani and Tao Xu and Tianhe Li and Tingbo Hou and Wei-Ning Hsu and Xi Yin and Xiaoliang Dai and Yaniv Taigman and Yaqiao Luo and Yen-Cheng Liu and Yi-Chiao Wu and Yue Zhao and Yuval Kirstain and Zecheng He and Zijian He and Albert Pumarola and Ali Thabet and Artsiom Sanakoyeu and Arun Mallya and Baishan Guo and Boris Araya and Breena Kerr and Carleigh Wood and Ce Liu and Cen Peng and Dimitry Vengertsev and Edgar Schonfeld and Elliot Blanchard and Felix Juefei-Xu and Fraylie Nord and Jeff Liang and John Hoffman and Jonas Kohler and Kaolin Fire and Karthik Sivakumar and Lawrence Chen and Licheng Yu and Luya Gao and Markos Georgopoulos and Rashel Moritz and Sara K. Sampson and Shikai Li and Simone Parmeggiani and Steve Fine and Tara Fowler and Vladan Petrovic and Yuming Du},
journal= {arXiv preprint arXiv:2410.13720},
year = {2025}
}