Probing Information Distribution in Transformer Architectures through Entropy Analysis
Abstract
This work explores entropy analysis as a tool for probing information distribution within Transformer-based architectures. By quantifying token-level uncertainty and examining entropy patterns across different stages of processing, we aim to investigate how information is managed and transformed within these models. As a case study, we apply the methodology to a GPT-based large language model, illustrating its potential to reveal insights into model behavior and internal representations. This approach may offer insights into model behavior and contribute to the development of interpretability and evaluation frameworks for transformer-based models
Keywords
Cite
@article{arxiv.2507.15347,
title = {Probing Information Distribution in Transformer Architectures through Entropy Analysis},
author = {Amedeo Buonanno and Alessandro Rivetti and Francesco A. N. Palmieri and Giovanni Di Gennaro and Gianmarco Romano},
journal= {arXiv preprint arXiv:2507.15347},
year = {2025}
}
Comments
Presented to the Italian Workshop on Neural Networks (WIRN2025) and it will appear in a Springer Chapter