Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
Abstract
Independently trained transformers compute the same function in residual-stream bases that differ by a uniform random rotation on . We call this phenomenon polymorphism: same function, mutually unintelligible interior coordinates. One matrix multiplication per model pair removes it: an orthogonal Procrustes fit on a single batch of activations transfers sparse-autoencoder feature dictionaries and steering vectors between independently trained models, with no retraining. The phenomenon is invisible to the standard SAE universality metric. Decoder-column cosine similarity matches across seeds at 98%, the SAE-universality headline number, while an SAE trained on one seed reconstructs another seed's activations at negative explained variance, worse than predicting the constant mean. The decoder columns align; the encoder reads from a rotated frame. A single Procrustes rotation restores reconstruction to within 0.025 EV of the within-seed ceiling at every internal site. is Haar-distributed: matches the random-orthogonal prediction to 0.1% at , and a Kolmogorov-Smirnov test of 's eigenvalue spectrum against Haar returns pooled and per-pair. Diff-of-means steering vectors transfer in three regimes by alignment with 's invariant subspace: clean when pinned by shared output weights, partial when overlapping the rotated subspace, inverted otherwise. With no shared I/O (Pythia), all three collapse to universally inverted. The same rotation account holds across training checkpoints within a single run. Validated on a 104k-parameter Dyck-3 transformer and nine independently-trained Pythia-70m seeds on The Pile, via a pre-registered four-bar operational framework. Frontier-scale (10B+) replication remains open.
Cite
@article{arxiv.2605.24577,
title = {Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m},
author = {Jordan F. McCann},
journal= {arXiv preprint arXiv:2605.24577},
year = {2026}
}
Comments
26 pages, 4 figures, 40 references. Pre-registered four-bar framework; all numerical claims reproducible