CamI2V：基于相机控制的图像到视频扩散模型
摘要
近期研究将相机姿态作为用户友好且受物理约束的条件集成到视频扩散模型中，实现了精确的相机控制。本文识别出一个关键挑战，即有效建模跨帧的噪声交互，以增强几何一致性和相机可控性。我们创新性地将条件质量与其降低不确定性能力关联联合，将噪声跨帧特征解释为一种噪声条件。认识到噪声条件在提供确定性信息的同时，也因添加的噪声引入随机性和潜在误导，我们提出仅在对应的单极线上聚合特征，以获取最佳数量的噪声条件。此外，我们针对单极线消失的场景（通常由快速相机运动、动态对象或遮挡引起），确保在多样化环境下的鲁棒性能。 Furthermore, we develop a more robust and reproducible evaluation pipeline to address the inaccuracies and instabilities of existing camera control metrics. Our method achieves a 25.64% improvement in camera controllability on the RealEstate10K dataset without compromising dynamics or generation quality and demonstrates strong generalization to out-of-domain images. Training and inference require only 24GB and 12GB of memory, respectively, for 16-frame sequences at 256x256 resolution. We will release all checkpoints, along with training and evaluation code. Dynamic videos are best viewed at https://zgctroy.github.io/CamI2V.
引用
@article{arxiv.2410.15957,
title = {CamI2V: Camera-Controlled Image-to-Video Diffusion Model},
author = {Guangcong Zheng and Teng Li and Rui Jiang and Yehao Lu and Tao Wu and Xi Li},
journal= {arXiv preprint arXiv:2410.15957},
year = {2024}
}