English

360{\deg} Volumetric Portrait Avatar

Computer Vision and Pattern Recognition 2023-12-12 v1

Abstract

We propose 360{\deg} Volumetric Portrait (3VP) Avatar, a novel method for reconstructing 360{\deg} photo-realistic portrait avatars of human subjects solely based on monocular video inputs. State-of-the-art monocular avatar reconstruction methods rely on stable facial performance capturing. However, the common usage of 3DMM-based facial tracking has its limits; side-views can hardly be captured and it fails, especially, for back-views, as required inputs like facial landmarks or human parsing masks are missing. This results in incomplete avatar reconstructions that only cover the frontal hemisphere. In contrast to this, we propose a template-based tracking of the torso, head and facial expressions which allows us to cover the appearance of a human subject from all sides. Thus, given a sequence of a subject that is rotating in front of a single camera, we train a neural volumetric representation based on neural radiance fields. A key challenge to construct this representation is the modeling of appearance changes, especially, in the mouth region (i.e., lips and teeth). We, therefore, propose a deformation-field-based blend basis which allows us to interpolate between different appearance states. We evaluate our approach on captured real-world data and compare against state-of-the-art monocular reconstruction methods. In contrast to those, our method is the first monocular technique that reconstructs an entire 360{\deg} avatar.

Keywords

Cite

@article{arxiv.2312.05311,
  title  = {360{\deg} Volumetric Portrait Avatar},
  author = {Jalees Nehvi and Berna Kabadayi and Julien Valentin and Justus Thies},
  journal= {arXiv preprint arXiv:2312.05311},
  year   = {2023}
}

Comments

Project page: https://jalees018.github.io/3VP-Avatar/

R2 v1 2026-06-28T13:45:29.704Z