TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices
Abstract
Developing deep learning models on tiny devices (e.g. Microcontroller units, MCUs) has attracted much attention in various embedded IoT applications. However, it is challenging to efficiently design and deploy recent advanced models (e.g. transformers) on tiny devices due to their severe hardware resource constraints. In this work, we propose TinyFormer, a framework specifically designed to develop and deploy resource-efficient transformer models on MCUs. TinyFormer consists of SuperNAS, SparseNAS, and SparseEngine. Separately, SuperNAS aims to search for an appropriate supernet from a vast search space. SparseNAS evaluates the best sparse single-path transformer model from the identified supernet. Finally, SparseEngine efficiently deploys the searched sparse models onto MCUs. To the best of our knowledge, SparseEngine is the first deployment framework capable of performing inference of sparse transformer models on MCUs. Evaluation results on the CIFAR-10 dataset demonstrate that TinyFormer can design efficient transformers with an accuracy of 96.1% while adhering to hardware constraints of 1MB storage and 320KB memory. Additionally, TinyFormer achieves significant speedups in sparse inference, up to 12.2x comparing to the CMSIS-NN library. TinyFormer is believed to bring powerful transformers into TinyML scenarios and to greatly expand the scope of deep learning applications
Cite
@article{arxiv.2311.01759,
title = {TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices},
author = {Jianlei Yang and Jiacheng Liao and Fanding Lei and Meichen Liu and Lingkun Long and Junyi Chen and Han Wan and Bei Yu and Weisheng Zhao},
journal= {arXiv preprint arXiv:2311.01759},
year = {2025}
}
Comments
This paper is accepted by IEEE Transactions on Circuits and Systems I: Regular Papers