Seeing only a tiny part of the whole is not knowing the full circumstance. Bird's-eye-view (BEV) perception, a process of obtaining allocentric maps from egocentric views, is restricted when using a narrow Field of View (FoV) alone. In this work, mapping from 360{\deg} panoramas to BEV semantics, the 360BEV task, is established for the first time to achieve holistic representations of indoor scenes in a top-down view. Instead of relying on narrow-FoV image sequences, a panoramic image with depth information is sufficient to generate a holistic BEV semantic map. To benchmark 360BEV, we present two indoor datasets, 360BEV-Matterport and 360BEV-Stanford, both of which include egocentric panoramic images and semantic segmentation labels, as well as allocentric semantic maps. Besides delving deep into different mapping paradigms, we propose a dedicated solution for panoramic semantic mapping, namely 360Mapper. Through extensive experiments, our methods achieve 44.32% and 45.78% in mIoU on both datasets respectively, surpassing previous counterparts with gains of +7.60% and +9.70% in mIoU. Code and datasets are available at the project page: https://jamycheung.github.io/360BEV.html.
Cite
@article{arxiv.2303.11910,
title = {360BEV: Panoramic Semantic Mapping for Indoor Bird's-Eye View},
author = {Zhifeng Teng and Jiaming Zhang and Kailun Yang and Kunyu Peng and Hao Shi and Simon Reiß and Ke Cao and Rainer Stiefelhagen},
journal= {arXiv preprint arXiv:2303.11910},
year = {2023}
}
Comments
Code and datasets are available at the project page: https://jamycheung.github.io/360BEV.html. Accepted to WACV 2024