English

Comparing Spark vs MPI/OpenMP On Word Count MapReduce

Distributed, Parallel, and Cluster Computing 2019-02-05 v2 Performance
Authors: Junhao Li
View on arXiv ↗ PDF ↗

Abstract

Spark provides an in-memory implementation of MapReduce that is widely used in the big data industry. MPI/OpenMP is a popular framework for high performance parallel computing. This paper presents a high performance MapReduce design in MPI/OpenMP and uses that to compare with Spark on the classic word count MapReduce task. My result shows that the MPI/OpenMP MapReduce outperforms Apache Spark by about 300%.

Cite

@article{arxiv.1811.04875,
  title  = {Comparing Spark vs MPI/OpenMP On Word Count MapReduce},
  author = {Junhao Li},
  journal= {arXiv preprint arXiv:1811.04875},
  year   = {2019}
}

Related papers

View all related →
Distributed, Parallel, and Cluster Computing · Computer Science
MPIgnite: An MPI-Like Language and Prototype Implementation for Apache Spark
Brandon L. Morris, Anthony Skjellum
2017-07-18
Performance · Computer Science
Performance Benefits of DataMPI: A Case Study with BigDataBench
Fan Liang, Chen Feng, Xiaoyi Lu, Zhiwei Xu
2014-03-17
Distributed, Parallel, and Cluster Computing · Computer Science
Benchmarking Apache Spark and Hadoop MapReduce on Big Data Classification
Taha Tekdogan, Ali Cakmak
2022-09-23
Databases · Computer Science
Evaluating Hive and Spark SQL with BigBench
Todor Ivanov, Max-Georg Beer
2016-01-14
Distributed, Parallel, and Cluster Computing · Computer Science
A Benchmarking Study to Evaluate Apache Spark on Large-Scale Supercomputers
George K. Thiruvathukal, Cameron Christensen, Xiaoyong Jin, François Tessier +1
2019-10-01
Distributed, Parallel, and Cluster Computing · Computer Science
Performance Evaluation of Parallel Message Passing and Thread Programming Model on Multicore Architectures
D. T. Hasta, A. B. Mutiara
2010-12-13
Databases · Computer Science
Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark
Elham Azhir, Mehdi Hosseinzadeh, Faheem Khan, Amir Mosavi
2022-10-14
Distributed, Parallel, and Cluster Computing · Computer Science
Understanding and Optimizing the Performance of Distributed Machine Learning Applications on Apache Spark
Celestine Dünner, Thomas Parnell, Kubilay Atasu, Manolis Sifalakis +1
2018-06-21
Distributed, Parallel, and Cluster Computing · Computer Science
Technical Report: On the Usability of Hadoop MapReduce, Apache Spark & Apache Flink for Data Science
Bilal Akil, Ying Zhou, Uwe Röhm
2018-03-30
Distributed, Parallel, and Cluster Computing · Computer Science
BSP vs MapReduce
Matthew Felice Pace
2012-06-19
Distributed, Parallel, and Cluster Computing · Computer Science
Decoupled Strategy for Imbalanced Workloads in MapReduce Frameworks
Sergio Rivas-Gomez, Sai Narasimhamurthy, Keeran Brabazon, Oliver Perks +2
2018-10-10
Distributed, Parallel, and Cluster Computing · Computer Science
Reproducible Experiments for Comparing Apache Flink and Apache Spark on Public Clouds
Shelan Perera, Ashansa Perera, Kamal Hakimzadeh
2016-10-17
Distributed, Parallel, and Cluster Computing · Computer Science
Accelerating Large-Scale Data Analysis by Offloading to High-Performance Computing Libraries using Alchemist
Alex Gittens, Kai Rothauge, Shusen Wang, Michael W. Mahoney +5
2018-05-31
Distributed, Parallel, and Cluster Computing · Computer Science
Alchemist: An Apache Spark <=> MPI Interface
Alex Gittens, Kai Rothauge, Shusen Wang, Michael W. Mahoney +5
2018-06-06
Distributed, Parallel, and Cluster Computing · Computer Science
Frustrated with MPI+Threads? Try MPIxThreads!
Hui Zhou, Ken Raffenetti, Junchao Zhang, Yanfei Guo +1
2024-01-31
Distributed, Parallel, and Cluster Computing · Computer Science
Achieving Efficient Strong Scaling with PETSc using Hybrid MPI/OpenMP Optimisation
Michael Lange, Gerard Gorman, Michele Weiland, Lawrence Mitchell +1
2015-01-14
Distributed, Parallel, and Cluster Computing · Computer Science
Solving All-Pairs Shortest-Paths Problem in Large Graphs Using Apache Spark
Frank Schoeneman, Jaroslaw Zola
2019-08-08
Machine Learning · Computer Science
MLlib: Machine Learning in Apache Spark
Xiangrui Meng, Joseph Bradley, Burak Yavuz, Evan Sparks +12
2015-05-27
Distributed, Parallel, and Cluster Computing · Computer Science
Performance comparison of Dask and Apache Spark on HPC systems for Neuroimaging
Mathieu Dugré, Valérie Hayot-Sasson, Tristan Glatard
2024-06-04
High Energy Physics - Lattice · Physics
Benchmarking MILC code with OpenMP and MPI
Steven Gottlieb, Sonali Tamhankar
2009-10-31
Distributed, Parallel, and Cluster Computing · Computer Science
Parallel Paradigms in Modern HPC: A Comparative Analysis of MPI, OpenMP, and CUDA
Nizar ALHafez, Ahmad Kurdi
2025-06-19
Distributed, Parallel, and Cluster Computing · Computer Science
The Efficiency of MapReduce in Parallel External Memory
Gero Greiner, Riko Jacob
2011-12-19
Distributed, Parallel, and Cluster Computing · Computer Science
An efficient MPI/OpenMP parallelization of the Hartree-Fock method for the second generation of Intel Xeon Phi processor
Vladimir Mironov, Yuri Alexeev, Kristopher Keipert, Michael D'mello +2
2017-08-15
Distributed, Parallel, and Cluster Computing · Computer Science
A unified framework to improve the interoperability between HPC and Big Data languages and programming models
César Piñeiro, Juan C. Pichel
2024-11-25
Distributed, Parallel, and Cluster Computing · Computer Science
Collectives in hybrid MPI+MPI code: design, practice and performance
Huan Zhou, Jose Gracia, Naweiluo Zhou, Ralf Schneider
2020-07-23
R2 v1 2026-06-23T05:12:57.302Z