Automated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipeline

Jayeeta Mondal; Swarnava Dey; Arijit Mukherjee

Automated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipeline

Computer Vision and Pattern Recognition 2023-11-13 v1 Optimization and Control

Authors: Jayeeta Mondal , Swarnava Dey , Arijit Mukherjee

Abstract

Multiple Deep Neural Networks (DNNs) integrated into single Deep Learning (DL) inference pipelines e.g. Multi-Task Learning (MTL) or Ensemble Learning (EL), etc., albeit very accurate, pose challenges for edge deployment. In these systems, models vary in their quantization tolerance and resource demands, requiring meticulous tuning for accuracy-latency balance. This paper introduces an automated heterogeneous quantization approach for DL inference pipelines with multiple DNNs.

Keywords

deep neural network pipeline optimization deep neural network acceleration

Cite

@article{arxiv.2311.05870,
  title  = {Automated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipeline},
  author = {Jayeeta Mondal and Swarnava Dey and Arijit Mukherjee},
  journal= {arXiv preprint arXiv:2311.05870},
  year   = {2023}
}

Related papers

View all related →

Instrumentation and Detectors · Physics

Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors