Video frame interpolation (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video. Existing learning-based VFI methods have achieved great success, but they still suffer from limited generalization ability due to the limited motion distribution of training datasets. In this paper, we propose a novel optimization-based VFI method that can adapt to unseen motions at test time. Our method is based on a cycle-consistency adaptation strategy that leverages the motion characteristics among video frames. We also introduce a lightweight adapter that can be inserted into the motion estimation module of existing pre-trained VFI models to improve the efficiency of adaptation. Extensive experiments on various benchmarks demonstrate that our method can boost the performance of two-frame VFI models, outperforming the existing state-of-the-art methods, even those that use extra input.
@article{arxiv.2306.13933,
title = {Boost Video Frame Interpolation via Motion Adaptation},
author = {Haoning Wu and Xiaoyun Zhang and Weidi Xie and Ya Zhang and Yanfeng Wang},
journal= {arXiv preprint arXiv:2306.13933},
year = {2023}
}
Comments
Accepted by BMVC 2023 (Oral Presentation) Project Page: https://haoningwu3639.github.io/VFI_Adapter_Webpage/