SEIFER: Scalable Edge Inference for Deep Neural Networks

Arjun Parthasarathy; Bhaskar Krishnamachari

SEIFER: Scalable Edge Inference for Deep Neural Networks

Networking and Internet Architecture 2022-11-21 v2 Distributed, Parallel, and Cluster Computing

Authors: Arjun Parthasarathy , Bhaskar Krishnamachari

Abstract

Edge inference is becoming ever prevalent through its applications from retail to wearable technology. Clusters of networked resource-constrained edge devices are becoming common, yet there is no production-ready orchestration system for deploying deep learning models over such edge networks which adopts the robustness and scalability of the cloud. We present SEIFER, a framework utilizing a standalone Kubernetes cluster to partition a given DNN and place these partitions in a distributed manner across an edge network, with the goal of maximizing inference throughput. The system is node fault-tolerant and automatically updates deployments based on updates to the model's version. We provide a preliminary evaluation of a partitioning and placement algorithm that works within this framework, and show that we can improve the inference pipeline throughput by 200% by utilizing sufficient numbers of resource-constrained nodes. We have implemented SEIFER in open-source software that is publicly available to the research community.

Keywords

deep neural network acceleration deep neural network deep learning

Cite

@article{arxiv.2210.12218,
  title  = {SEIFER: Scalable Edge Inference for Deep Neural Networks},
  author = {Arjun Parthasarathy and Bhaskar Krishnamachari},
  journal= {arXiv preprint arXiv:2210.12218},
  year   = {2022}
}

Comments

Accepted to the "Challenges in Deploying and Monitoring ML Systems" Workshop of NeurIPS 2022

SEIFER: Scalable Edge Inference for Deep Neural Networks

Abstract

Keywords

Cite

Comments

Related papers