CLUSTSEG: Clustering for Universal Segmentation
Abstract
We present CLUSTSEG, a general, transformer-based framework that tackles different image segmentation tasks (i.e., superpixel, semantic, instance, and panoptic) through a unified neural clustering scheme. Regarding queries as cluster centers, CLUSTSEG is innovative in two aspects:1) cluster centers are initialized in heterogeneous ways so as to pointedly address task-specific demands (e.g., instance- or category-level distinctiveness), yet without modifying the architecture; and 2) pixel-cluster assignment, formalized in a cross-attention fashion, is alternated with cluster center update, yet without learning additional parameters. These innovations closely link CLUSTSEG to EM clustering and make it a transparent and powerful framework that yields superior results across the above segmentation tasks.
Cite
@article{arxiv.2305.02187,
title = {CLUSTSEG: Clustering for Universal Segmentation},
author = {James Liang and Tianfei Zhou and Dongfang Liu and Wenguan Wang},
journal= {arXiv preprint arXiv:2305.02187},
year = {2023}
}
Comments
Accepted to ICML 2023; Code: https://github.com/JamesLiang819/ClustSeg