English

Causality $\neq$ Decodability, and Vice Versa: Lessons from Interpreting Counting ViTs

Machine Learning 2025-10-14 v1 Computer Vision and Pattern Recognition

Abstract

Mechanistic interpretability seeks to uncover how internal components of neural networks give rise to predictions. A persistent challenge, however, is disentangling two often conflated notions: decodability--the recoverability of information from hidden states--and causality--the extent to which those states functionally influence outputs. In this work, we investigate their relationship in vision transformers (ViTs) fine-tuned for object counting. Using activation patching, we test the causal role of spatial and CLS tokens by transplanting activations across clean-corrupted image pairs. In parallel, we train linear probes to assess the decodability of count information at different depths. Our results reveal systematic mismatches: middle-layer object tokens exert strong causal influence despite being weakly decodable, whereas final-layer object tokens support accurate decoding yet are functionally inert. Similarly, the CLS token becomes decodable in mid-layers but only acquires causal power in the final layers. These findings highlight that decodability and causality reflect complementary dimensions of representation--what information is present versus what is used--and that their divergence can expose hidden computational circuits.

Keywords

Cite

@article{arxiv.2510.09794,
  title  = {Causality $\neq$ Decodability, and Vice Versa: Lessons from Interpreting Counting ViTs},
  author = {Lianghuan Huang and Yingshan Chang},
  journal= {arXiv preprint arXiv:2510.09794},
  year   = {2025}
}
R2 v1 2026-07-01T06:30:21.671Z