English

The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers

Computer Vision and Pattern Recognition 2025-12-09 v1

Abstract

Vision Transformers (ViTs) lack the hierarchical inductive biases inherent to Convolutional Neural Networks (CNNs), theoretically allowing them to maintain high-dimensional representations throughout all layers. However, recent observations suggest ViTs often spontaneously manifest a "U-shaped" entropy profile-compressing information in middle layers before expanding it for the final classification. In this work, we demonstrate that this "Inductive Bottleneck" is not an architectural artifact, but a data-dependent adaptation. By analyzing the layer-wise Effective Encoding Dimension (EED) of DINO-trained ViTs across datasets of varying compositional complexity (UC Merced, Tiny ImageNet, and CIFAR-100), we show that the depth of the bottleneck correlates strongly with the semantic abstraction required by the task. We find that while texture-heavy datasets preserve high-rank representations throughout, object-centric datasets drive the network to dampen high-frequency information in middle layers, effectively "learning" a bottleneck to isolate semantic features.

Keywords

Cite

@article{arxiv.2512.07331,
  title  = {The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers},
  author = {Kanishk Awadhiya},
  journal= {arXiv preprint arXiv:2512.07331},
  year   = {2025}
}
R2 v1 2026-07-01T08:14:30.042Z