Visual Geometry Transformer (VGGT) is a strong feed-forward model for multiple 3D tasks, but its Alternating-Attention (AA) stack scales quadratically in the total token count, making long clips expensive. Existing token-reduction accelerators operate inside AA, leaving the patch grid that enters AA uncompressed. We introduce PaceVGGT, a pre-AA token pruning framework that prunes DINO patch tokens before the first AA block of a frozen VGGT. PaceVGGT trains a lightweight Token Scorer that estimates per-token importance from DINO features. The scorer is first distilled against an AA-internal attention target from the unpruned backbone, then refined under downstream camera, depth, and point-map losses. A per-frame keep budget fixes the backbone-visible sequence length, while an importance-adaptive merge/prune assignment preserves residual content from high-saliency frames under a fixed total merge budget. A Feature-guided Restoration module reconstructs the dense spatial grid required by the prediction heads. On ScanNet-50 and 7-Scenes, PaceVGGT remains on the reconstruction quality--latency frontier while reducing inference latency. On ScanNet-50, it reduces latency by 5.1× over unmodified VGGT at N=300 and 1.47× over LiteVGGT at N=1000. These results identify pre-AA pruning as a viable acceleration route for frozen VGGT-style geometry transformers.
@article{arxiv.2605.08371,
title = {PaceVGGT: Pre-Alternating-Attention Token Pruning for Visual Geometry Transformers},
author = {Haotang Li and Zhenyu Qi and Shaohan Henry Wang and Kebin Peng and Zi Wang and Qing Guo and Sen He and Huanrui Yang},
journal= {arXiv preprint arXiv:2605.08371},
year = {2026}
}