Missing data poses significant challenges while learning representations of video sequences. We present Disentangled Imputed Video autoEncoder (DIVE), a deep generative model that imputes and predicts future video frames in the presence of missing data. Specifically, DIVE introduces a missingness latent variable, disentangles the hidden video representations into static and dynamic appearance, pose, and missingness factors for each object. DIVE imputes each object's trajectory where data is missing. On a moving MNIST dataset with various missing scenarios, DIVE outperforms the state of the art baselines by a substantial margin. We also present comparisons for real-world MOTSChallenge pedestrian dataset, which demonstrates the practical value of our method in a more realistic setting. Our code and data can be found at https://github.com/Rose-STL-Lab/DIVE.
@article{arxiv.2006.13391,
title = {Learning Disentangled Representations of Video with Missing Data},
author = {Armand Comas-Massagué and Chi Zhang and Zlatan Feric and Octavia Camps and Rose Yu},
journal= {arXiv preprint arXiv:2006.13391},
year = {2020}
}