LiDAR-based 3D object detectors typically rely on proposal heads with hand-crafted components like anchor assignment and non-maximum suppression (NMS), complicating training and limiting extensibility. We present AutoReg3D, an autoregressive 3D detector that casts detection as sequence generation. Given point-cloud features, AutoReg3D emits objects in a range-causal (near-to-far) order and encodes each object as a short, discrete-token sequence consisting of its center, size, orientation, velocity, and class. This near-to-far ordering mirrors LiDAR geometry--near objects occlude far ones but not vice versa--enabling straightforward teacher forcing during training and autoregressive decoding at test time. AutoReg3D is compatible across diverse point-cloud or backbones and attains competitive nuScenes performance without anchors or NMS. Beyond parity, the sequential formulation unlocks language-model advances for 3D perception, including GRPO-style reinforcement learning for task-aligned objectives. These results position autoregressive decoding as a viable, flexible alternative for LiDAR-based detection and open a path to importing modern sequence-modeling tools into 3D perception.
@article{arxiv.2603.07985,
title = {On the Feasibility and Opportunity of Autoregressive 3D Object Detection},
author = {Zanming Huang and Jinsu Yoo and Sooyoung Jeon and Zhenzhen Liu and Mark Campbell and Kilian Q Weinberger and Bharath Hariharan and Wei-Lun Chao and Katie Z Luo},
journal= {arXiv preprint arXiv:2603.07985},
year = {2026}
}