Sparse Autoencoders for transformer-based language models are typically defined independently per layer. In this work we analyze statistical relationships between features in adjacent layers to understand how features evolve through a forward pass. We provide a graph visualization interface for features and their most similar next-layer neighbors (https://stefanhex.com/spar-2024/feature-browser/), and build communities of related features across layers. We find that a considerable amount of features are passed through from a previous layer, some features can be expressed as quasi-boolean combinations of previous features, and some features become more specialized in later layers.
@article{arxiv.2410.08869,
title = {Evolution of SAE Features Across Layers in LLMs},
author = {Daniel Balcells and Benjamin Lerner and Michael Oesterle and Ediz Ucar and Stefan Heimersheim},
journal= {arXiv preprint arXiv:2410.08869},
year = {2024}
}
Comments
Presented at the Attributing Model Behavior at Scale (ATTRIB) workshop at NeurIPS 2024