This paper presents BigDL (a distributed deep learning framework for Apache Spark), which has been used by a variety of users in the industry for building deep learning applications on production big data platforms. It allows deep learning applications to run on the Apache Hadoop/Spark cluster so as to directly process the production data, and as a part of the end-to-end data analysis pipeline for deployment and management. Unlike existing deep learning frameworks, BigDL implements distributed, data parallel training directly on top of the functional compute model (with copy-on-write and coarse-grained operations) of Spark. We also share real-world experience and "war stories" of users that have adopted BigDL to address their challenges(i.e., how to easily build end-to-end data analysis and deep learning pipelines for their production data).
@article{arxiv.1804.05839,
title = {BigDL: A Distributed Deep Learning Framework for Big Data},
author = {Jason Dai and Yiheng Wang and Xin Qiu and Ding Ding and Yao Zhang and Yanzhang Wang and Xianyan Jia and Cherry Zhang and Yan Wan and Zhichao Li and Jiao Wang and Shengsheng Huang and Zhongyuan Wu and Yang Wang and Yuhao Yang and Bowen She and Dongjie Shi and Qi Lu and Kai Huang and Guoqiong Song},
journal= {arXiv preprint arXiv:1804.05839},
year = {2021}
}
Comments
In ACM Symposium of Cloud Computing conference (SoCC) 2019