This work introduces a highly efficient implementation of the transformer architecture on a Field-Programmable Gate Array (FPGA) by using the \texttt{hls4ml} tool. Given the demonstrated effectiveness of transformer models in addressing a wide range of problems, their application in experimental triggers within particle physics becomes a subject of significant interest. In this work, we have implemented critical components of a transformer model, such as multi-head attention and softmax layers. To evaluate the effectiveness of our implementation, we have focused on a particle physics jet flavor tagging problem, employing a public dataset. We recorded latency under 2 μs on the Xilinx UltraScale+ FPGA, which is compatible with hardware trigger requirements at the CERN Large Hadron Collider experiments.
@article{arxiv.2402.01047,
title = {Ultra Fast Transformers on FPGAs for Particle Physics Experiments},
author = {Zhixing Jiang and Dennis Yin and Elham E Khoda and Vladimir Loncar and Ekaterina Govorkova and Eric Moreno and Philip Harris and Scott Hauck and Shih-Chieh Hsu},
journal= {arXiv preprint arXiv:2402.01047},
year = {2024}
}