Relational inductive biases on attention mechanisms
Abstract
Inductive learning aims to construct general models from specific examples, guided by biases that influence hypothesis selection and determine generalization capacity. In this work, we focus on characterizing the relational inductive biases present in attention mechanisms, understood as assumptions about the underlying relationships between data elements. From the perspective of geometric deep learning, we analyze the most common attention mechanisms in terms of their equivariance properties with respect to permutation subgroups, which allows us to propose a classification based on their relational biases. Under this perspective, we show that different attention layers are characterized by the underlying relationships they assume on the input data.
Cite
@article{arxiv.2507.04117,
title = {Relational inductive biases on attention mechanisms},
author = {Víctor Mijangos and Ximena Gutierrez-Vasques and Verónica E. Arriola and Ulises Rodríguez-Domínguez and Alexis Cervantes and José Luis Almanzara},
journal= {arXiv preprint arXiv:2507.04117},
year = {2025}
}
Comments
This paper was originally published in Spanish in the journal Research in Computing Science (https://www.rcs.cic.ipn.mx/)