Robots cannot yet match humans' ability to rapidly learn the shapes of novel 3D objects and recognize them robustly despite clutter and occlusion. We present Bayes3D, an uncertainty-aware perception system for structured 3D scenes, that reports accurate posterior uncertainty over 3D object shape, pose, and scene composition in the presence of clutter and occlusion. Bayes3D delivers these capabilities via a novel hierarchical Bayesian model for 3D scenes and a GPU-accelerated coarse-to-fine sequential Monte Carlo algorithm. Quantitative experiments show that Bayes3D can learn 3D models of novel objects from just a handful of views, recognizing them more robustly and with orders of magnitude less training data than neural baselines, and tracking 3D objects faster than real time on a single GPU. We also demonstrate that Bayes3D learns complex 3D object models and accurately infers 3D scene composition when used on a Panda robot in a tabletop scenario.
@article{arxiv.2312.08715,
title = {Bayes3D: fast learning and inference in structured generative models of 3D objects and scenes},
author = {Nishad Gothoskar and Matin Ghavami and Eric Li and Aidan Curtis and Michael Noseworthy and Karen Chung and Brian Patton and William T. Freeman and Joshua B. Tenenbaum and Mirko Klukas and Vikash K. Mansinghka},
journal= {arXiv preprint arXiv:2312.08715},
year = {2023}
}