English

Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m

Machine Learning 2026-05-26 v1 Artificial Intelligence Computation and Language

Abstract

Independently trained transformers compute the same function in residual-stream bases that differ by a uniform random rotation on SO(dmodel)\mathrm{SO}(d_{\mathrm{model}}). We call this phenomenon polymorphism: same function, mutually unintelligible interior coordinates. One matrix multiplication per model pair removes it: an orthogonal Procrustes fit on a single batch of activations transfers sparse-autoencoder feature dictionaries and steering vectors between independently trained models, with no retraining. The phenomenon is invisible to the standard SAE universality metric. Decoder-column cosine similarity matches across seeds at 98%, the SAE-universality headline number, while an SAE trained on one seed reconstructs another seed's activations at negative explained variance, worse than predicting the constant mean. The decoder columns align; the encoder reads from a rotated frame. A single Procrustes rotation RR restores reconstruction to within 0.025 EV of the within-seed ceiling at every internal site. RR is Haar-distributed: RIF\|R - I\|_F matches the random-orthogonal prediction 2dmodel\sqrt{2 d_{\mathrm{model}}} to 0.1% at dmodel=512d_{\mathrm{model}} = 512, and a Kolmogorov-Smirnov test of RR's eigenvalue spectrum against Haar SO(dmodel)\mathrm{SO}(d_{\mathrm{model}}) returns p1.000p \approx 1.000 pooled and per-pair. Diff-of-means steering vectors transfer in three regimes by alignment with RR's invariant subspace: clean when pinned by shared output weights, partial when overlapping the rotated subspace, inverted otherwise. With no shared I/O (Pythia), all three collapse to universally inverted. The same rotation account holds across training checkpoints within a single run. Validated on a 104k-parameter Dyck-3 transformer and nine independently-trained Pythia-70m seeds on The Pile, via a pre-registered four-bar operational framework. Frontier-scale (10B+) replication remains open.

Keywords

Cite

@article{arxiv.2605.24577,
  title  = {Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m},
  author = {Jordan F. McCann},
  journal= {arXiv preprint arXiv:2605.24577},
  year   = {2026}
}

Comments

26 pages, 4 figures, 40 references. Pre-registered four-bar framework; all numerical claims reproducible

R2 v1 2026-07-22T07:30:03.401Z