English

Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation

Computer Vision and Pattern Recognition 2026-03-11 v2

Abstract

Knowledge distillation between asymmetric architectures often induces severe geometric constraints on the learned representation space. In this work, we investigate the Dimensional Collapse phenomenon when distilling global Vision Transformers (CLIP and DINOv2) into capacity-constrained CNNs. By employing strictly centered SVD and Effective Rank, we first demonstrate a capacity-agnostic phase transition on CIFAR-10 where standard cosine distillation collapses representations to an intrinsic Effective Rank of ~16. To reverse this, we integrate an auxiliary contrastive objective (InfoNCE), expanding the student's manifold by 2.4x (to ~38 effective dimensions). We further demonstrate that while DINOv2's uniform geometry partially prevents collapse, contrastive expansion remains a universal requirement to reach the CNN's topological capacity limit (~82 dimensions). Finally, we reveal a critical capacity-density trade-off: overparameterization within fixed manifolds induces brittleness, while capacity-constrained models act as optimal low-pass semantic filters, successfully recovering inherent noise immunity.

Keywords

Cite

@article{arxiv.2603.06698,
  title  = {Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation},
  author = {Kabir Thayani},
  journal= {arXiv preprint arXiv:2603.06698},
  year   = {2026}
}

Comments

Introduced auxiliary InfoNCE objective to reverse dimensional collapse. Expanded experiments to DINOv2 teacher and CIFAR-100 dataset. 3 pages, 3 figures, 2 tables