Who Does What in Deep Learning? Multidimensional Game-Theoretic Attribution of Function of Neural Units
Abstract
Neural networks now generate text, images, and speech with billions of parameters, producing a need to know how each neural unit contributes to these high-dimensional outputs. Existing explainable-AI methods, such as SHAP, attribute importance to inputs, but cannot quantify the contributions of neural units across thousands of output pixels, tokens, or logits. Here we close that gap with Multiperturbation Shapley-value Analysis (MSA), a model-agnostic game-theoretic framework. By systematically lesioning combinations of units, MSA yields Shapley Modes, unit-wise contribution maps that share the exact dimensionality of the model's output. We apply MSA across scales, from multi-layer perceptrons to the 56-billion-parameter Mixtral-8x7B and Generative Adversarial Networks (GAN). The approach demonstrates how regularisation concentrates computation in a few hubs, exposes language-specific experts inside the LLM, and reveals an inverted pixel-generation hierarchy in GANs. Together, these results showcase MSA as a powerful approach for interpreting, editing, and compressing deep neural networks.
Cite
@article{arxiv.2506.19732,
title = {Who Does What in Deep Learning? Multidimensional Game-Theoretic Attribution of Function of Neural Units},
author = {Shrey Dixit and Kayson Fakhar and Fatemeh Hadaeghi and Patrick Mineault and Konrad P. Kording and Claus C. Hilgetag},
journal= {arXiv preprint arXiv:2506.19732},
year = {2025}
}