To unlock video chat for hundreds of millions of people hindered by poor connectivity or unaffordable data costs, we propose to authentically reconstruct faces on the receiver's device using facial landmarks extracted at the sender's side and transmitted over the network. In this context, we discuss and evaluate the benefits and disadvantages of several deep adversarial approaches. In particular, we explore quality and bandwidth trade-offs for approaches based on static landmarks, dynamic landmarks or segmentation maps. We design a mobile-compatible architecture based on the first order animation model of Siarohin et al. In addition, we leverage SPADE blocks to refine results in important areas such as the eyes and lips. We compress the networks down to about 3MB, allowing models to run in real time on iPhone 8 (CPU). This approach enables video calling at a few kbits per second, an order of magnitude lower than currently available alternatives.
Cite
@article{arxiv.2012.00328,
title = {Low Bandwidth Video-Chat Compression using Deep Generative Models},
author = {Maxime Oquab and Pierre Stock and Oran Gafni and Daniel Haziza and Tao Xu and Peizhao Zhang and Onur Celebi and Yana Hasson and Patrick Labatut and Bobo Bose-Kolanu and Thibault Peyronel and Camille Couprie},
journal= {arXiv preprint arXiv:2012.00328},
year = {2020}
}