This paper presents a new fast, highly scalable distributed matrix multiplication algorithm on Apache Spark, called Stark, based on Strassen's matrix multiplication algorithm. Stark preserves Strassen's 7 multiplications scheme in a distributed environment and thus achieves faster execution. It is based on two new ideas; it creates a recursion tree of computation where each level of such tree corresponds to division and combination of distributed matrix blocks in the form of Resilient Distributed Datasets(RDDs); It processes each divide and combine step in parallel and memorize the sub-matrices by intelligently tagging matrix blocks in it. To the best of our knowledge, Stark is the first Strassen's implementation in Spark platform. We show experimentally that Stark has a strong scalability with increasing matrix size enabling us to multiply two (16384 x 16384) matrices with 28% and 36% less wall clock time than Marlin and MLLib respectively, state-of-the-art matrix multiplication approaches based on Spark.
@article{arxiv.1811.07325,
title = {Stark: Fast and Scalable Strassen's Matrix Multiplication using Apache Spark},
author = {Chandan Misra and Sourangshu Bhattacharya and Soumya K. Ghosh},
journal= {arXiv preprint arXiv:1811.07325},
year = {2018}
}