The Phong Surface: Efficient 3D Model Fitting using Lifted Optimization
Abstract
Realtime perceptual and interaction capabilities in mixed reality require a range of 3D tracking problems to be solved at low latency on resource-constrained hardware such as head-mounted devices. Indeed, for devices such as HoloLens 2 where the CPU and GPU are left available for applications, multiple tracking subsystems are required to run on a continuous, real-time basis while sharing a single Digital Signal Processor. To solve model-fitting problems for HoloLens 2 hand tracking, where the computational budget is approximately 100 times smaller than an iPhone 7, we introduce a new surface model: the `Phong surface'. Using ideas from computer graphics, the Phong surface describes the same 3D shape as a triangulated mesh model, but with continuous surface normals which enable the use of lifting-based optimization, providing significant efficiency gains over ICP-based methods. We show that Phong surfaces retain the convergence benefits of smoother surface models, while triangle meshes do not.
Cite
@article{arxiv.2007.04940,
title = {The Phong Surface: Efficient 3D Model Fitting using Lifted Optimization},
author = {Jingjing Shen and Thomas J. Cashman and Qi Ye and Tim Hutton and Toby Sharp and Federica Bogo and Andrew William Fitzgibbon and Jamie Shotton},
journal= {arXiv preprint arXiv:2007.04940},
year = {2020}
}