IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Abstract
Circuit discovery has recently attracted attention as a potential research direction to explain the non-trivial behaviors of language models. It aims to find the computational subgraphs, also known as circuits, within the model that are responsible for solving specific tasks. However, most existing studies overlook the holistic nature of these circuits and require designing specific corrupted activations for different tasks, which is inaccurate and inefficient. In this work, we propose an end-to-end approach based on the principle of Information Bottleneck, called IBCircuit, to identify informative circuits holistically. IBCircuit is an optimization framework for holistic circuit discovery and can be applied to any given task without tediously corrupted activation design. In both the Indirect Object Identification (IOI) and Greater-Than tasks, IBCircuit identifies more faithful and minimal circuits in terms of critical node components and edge components compared to recent related work.
Cite
@article{arxiv.2602.22581,
title = {IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck},
author = {Tian Bian and Yifan Niu and Chaohao Yuan and Chengzhi Piao and Bingzhe Wu and Long-Kai Huang and Yu Rong and Tingyang Xu and Hong Cheng and Jia Li},
journal= {arXiv preprint arXiv:2602.22581},
year = {2026}
}