QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding
Abstract
Robotic tasks such as planning and navigation require a hierarchical semantic understanding of a scene, which could include multiple floors and rooms. Current methods primarily focus on object segmentation for 3D scene understanding. However, such methods struggle to segment out topological regions like "kitchen" in the scene. In this work, we introduce a two-step pipeline to solve this problem. First, we extract a topological map, i.e., floorplan of the indoor scene using a novel multi-channel occupancy representation. Then, we generate CLIP-aligned features and semantic labels for every room instance based on the objects it contains using a self-attention transformer. Our language-topology alignment supports natural language querying, e.g., a "place to cook" locates the "kitchen". We outperform the current state-of-the-art on room segmentation by ~20% and room classification by ~12%. Our detailed qualitative analysis and ablation studies provide insights into the problem of joint structural and semantic 3D scene understanding. Project Page: quest-maps.github.io
Cite
@article{arxiv.2404.06442,
title = {QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding},
author = {Yash Mehan and Kumaraditya Gupta and Rohit Jayanti and Anirudh Govil and Sourav Garg and Madhava Krishna},
journal= {arXiv preprint arXiv:2404.06442},
year = {2024}
}
Comments
Accepted at 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) as Oral Presentation. Also presented at the 2nd Workshop on Open-Vocabulary 3D Scene Understanding (OpenSUN3D) at CVPR 2024