This paper proposes a Generative Face Video Compression (GFVC) approach using Supplemental Enhancement Information (SEI), where a series of compact spatial and temporal representations of a face video signal (e.g., 2D/3D keypoints, facial semantics and compact features) can be coded using SEI messages and inserted into the coded video bitstream. At the time of writing, the proposed GFVC approach using SEI messages has been included into a draft amendment of the Versatile Supplemental Enhancement Information (VSEI) standard by the Joint Video Experts Team (JVET) of ISO/IEC JTC 1/SC 29 and ITU-T SG21, which will be standardized as a new version of ITU-T H.274 | ISO/IEC 23002-7. To the best of the authors' knowledge, the JVET work on the proposed SEI-based GFVC approach is the first standardization activity for generative video compression. The proposed SEI approach has not only advanced the reconstruction quality of early-day Model-Based Coding (MBC) via the state-of-the-art generative technique, but also established a new SEI definition for future GFVC applications and deployment. Experimental results illustrate that the proposed SEI-based GFVC approach can achieve remarkable rate-distortion performance compared with the latest Versatile Video Coding (VVC) standard, whilst also potentially enabling a wide variety of functionalities including user-specified animation/filtering and metaverse-related applications.
Cite
@article{arxiv.2410.15105,
title = {Standardizing Generative Face Video Compression using Supplemental Enhancement Information},
author = {Bolin Chen and Yan Ye and Jie Chen and Ru-Ling Liao and Shanzhi Yin and Shiqi Wang and Kaifa Yang and Yue Li and Yiling Xu and Ye-Kui Wang and Shiv Gehlot and Guan-Ming Su and Peng Yin and Sean McCarthy and Gary J. Sullivan},
journal= {arXiv preprint arXiv:2410.15105},
year = {2025}
}