First-Generation Inference Accelerator Deployment at Facebook
Abstract
In this paper, we provide a deep dive into the deployment of inference accelerators at Facebook. Many of our ML workloads have unique characteristics, such as sparse memory accesses, large model sizes, as well as high compute, memory and network bandwidth requirements. We co-designed a high-performance, energy-efficient inference accelerator platform based on these requirements. We describe the inference accelerator platform ecosystem we developed and deployed at Facebook: both hardware, through Open Compute Platform (OCP), and software framework and tooling, through Pytorch/Caffe2/Glow. A characteristic of this ecosystem from the start is its openness to enable a variety of AI accelerators from different vendors. This platform, with six low-power accelerator cards alongside a single-socket host CPU, allows us to serve models of high complexity that cannot be easily or efficiently run on CPUs. We describe various performance optimizations, at both platform and accelerator level, which enables this platform to serve production traffic at Facebook. We also share deployment challenges, lessons learned during performance optimization, as well as provide guidance for future inference hardware co-design.
Cite
@article{arxiv.2107.04140,
title = {First-Generation Inference Accelerator Deployment at Facebook},
author = {Michael Anderson and Benny Chen and Stephen Chen and Summer Deng and Jordan Fix and Michael Gschwind and Aravind Kalaiah and Changkyu Kim and Jaewon Lee and Jason Liang and Haixin Liu and Yinghai Lu and Jack Montgomery and Arun Moorthy and Satish Nadathur and Sam Naghshineh and Avinash Nayak and Jongsoo Park and Chris Petersen and Martin Schatz and Narayanan Sundaram and Bangsheng Tang and Peter Tang and Amy Yang and Jiecao Yu and Hector Yuen and Ying Zhang and Aravind Anbudurai and Vandana Balan and Harsha Bojja and Joe Boyd and Matthew Breitbach and Claudio Caldato and Anna Calvo and Garret Catron and Sneh Chandwani and Panos Christeas and Brad Cottel and Brian Coutinho and Arun Dalli and Abhishek Dhanotia and Oniel Duncan and Roman Dzhabarov and Simon Elmir and Chunli Fu and Wenyin Fu and Michael Fulthorp and Adi Gangidi and Nick Gibson and Sean Gordon and Beatriz Padilla Hernandez and Daniel Ho and Yu-Cheng Huang and Olof Johansson and Shishir Juluri and Shobhit Kanaujia and Manali Kesarkar and Jonathan Killinger and Ben Kim and Rohan Kulkarni and Meghan Lele and Huayu Li and Huamin Li and Yueming Li and Cynthia Liu and Jerry Liu and Bert Maher and Chandra Mallipedi and Seema Mangla and Kiran Kumar Matam and Jubin Mehta and Shobhit Mehta and Christopher Mitchell and Bharath Muthiah and Nitin Nagarkatte and Ashwin Narasimha and Bernard Nguyen and Thiara Ortiz and Soumya Padmanabha and Deng Pan and Ashwin Poojary and Ye and Qi and Olivier Raginel and Dwarak Rajagopal and Tristan Rice and Craig Ross and Nadav Rotem and Scott Russ and Kushal Shah and Baohua Shan and Hao Shen and Pavan Shetty and Krish Skandakumaran and Kutta Srinivasan and Roshan Sumbaly and Michael Tauberg and Mor Tzur and Sidharth Verma and Hao Wang and Man Wang and Ben Wei and Alex Xia and Chenyu Xu and Martin Yang and Kai Zhang and Ruoxi Zhang and Ming Zhao and Whitney Zhao and Rui Zhu and Ajit Mathews and Lin Qiao and Misha Smelyanskiy and Bill Jia and Vijay Rao},
journal= {arXiv preprint arXiv:2107.04140},
year = {2021}
}