English

Hadoop Performance Models

Distributed, Parallel, and Cluster Computing 2011-06-07 v1
Authors: Herodotos Herodotou
View on arXiv ↗ PDF ↗

Abstract

Hadoop MapReduce is now a popular choice for performing large-scale data analytics. This technical report describes a detailed set of mathematical performance models for describing the execution of a MapReduce job on Hadoop. The models describe dataflow and cost information at the fine granularity of phases within the map and reduce tasks of a job execution. The models can be used to estimate the performance of MapReduce jobs as well as to find the optimal configuration settings to use when running the jobs.

Keywords

distributed systems data processing machine learning

Cite

@article{arxiv.1106.0940,
  title  = {Hadoop Performance Models},
  author = {Herodotos Herodotou},
  journal= {arXiv preprint arXiv:1106.0940},
  year   = {2011}
}

Comments

16 pages, 0 figures, Duke University Technical Report

Related papers

View all related →
Distributed, Parallel, and Cluster Computing · Computer Science
Optimizing MapReduce for Highly Distributed Environments
Benjamin Heintz, Abhishek Chandra, Ramesh K. Sitaraman
2012-07-31
Distributed, Parallel, and Cluster Computing · Computer Science
Analyzing Web Application Log Files to Find Hit Count Through the Utilization of Hadoop MapReduce in Cloud Computing Environment
Sayalee Narkhede, Trupti Baraskar, Debajyoti Mukhopadhyay
2014-12-01
Distributed, Parallel, and Cluster Computing · Computer Science
An Alternative C++ based HPC system for Hadoop MapReduce
Vignesh S., Muthumanikandan V., Siddarth S., Sainath G
2020-06-29
Distributed, Parallel, and Cluster Computing · Computer Science
Benchmarking and Performance Modelling of MapReduce Communication Pattern
Sheriffo Ceesay, Adam Barker, Yuhui Lin
2020-05-26
Distributed, Parallel, and Cluster Computing · Computer Science
Survey on Improved Scheduling in Hadoop MapReduce in Cloud Environments
B. Thirumala Rao, L. S. S. Reddy
2012-07-04
Distributed, Parallel, and Cluster Computing · Computer Science
An Open-Source Project for MapReduce Performance Self-Tuning
Donghua Chen
2020-01-01
Distributed, Parallel, and Cluster Computing · Computer Science
Beyond Batch Processing: Towards Real-Time and Streaming Big Data
Saeed Shahrivari, Saeed Jalili
2014-08-04
Networking and Internet Architecture · Computer Science
A Survey of Big Data Machine Learning Applications Optimization in Cloud Data Centers and Networks
Sanaa Hamid Mohamed, Taisir E. H. El-Gorashi, Jaafar M. H. Elmirghani
2019-10-03
Distributed, Parallel, and Cluster Computing · Computer Science
High Performance Risk Aggregation: Addressing the Data Processing Challenge the Hadoop MapReduce Way
Zhimin Yao, Blesson Varghese, Andrew Rau-Chaplin
2013-11-25
Distributed, Parallel, and Cluster Computing · Computer Science
Performance Tuning of Hadoop MapReduce: A Noisy Gradient Approach
Sandeep Kumar, Sindhu Padakandla, Chandrashekar L, Priyank Parihar +2
2019-08-28
Distributed, Parallel, and Cluster Computing · Computer Science
An Experimental Evaluation of Performance of A Hadoop Cluster on Replica Management
Muralikrishnan Ramane, Sharmila Krishnamoorthy, Sasikala Gowtham
2014-11-10
Distributed, Parallel, and Cluster Computing · Computer Science
Iterative MapReduce for Large Scale Machine Learning
Joshua Rosen, Neoklis Polyzotis, Vinayak Borkar, Yingyi Bu +4
2013-03-15
Databases · Computer Science
Building Wavelet Histograms on Large Data in MapReduce
Jeffrey Jestes, Ke Yi, Feifei Li
2011-11-01
Databases · Computer Science
Column-Oriented Storage Techniques for MapReduce
Avrilia Floratou, Jignesh Patel, Eugene Shekita, Sandeep Tata
2011-05-24
Distributed, Parallel, and Cluster Computing · Computer Science
On Modelling and Prediction of Total CPU Usage for Applications in MapReduce Environments
Nikzad Babaii Rizvandi, Javid Taheri, Reza Moraveji, Albert Y. Zomaya
2012-07-30
Databases · Computer Science
Analyzing Large-Scale, Distributed and Uncertain Data
Yaron Gonen
2017-12-06
Distributed, Parallel, and Cluster Computing · Computer Science
Optimization and analysis of large scale data sorting algorithm based on Hadoop
Zhuo Wang, Longlong Tian, Dianjie Guo, Xiaoming Jiang
2015-06-02
Distributed, Parallel, and Cluster Computing · Computer Science
Running genetic algorithms on Hadoop for solving high dimensional optimization problems
Güngör Yildirim, İbrahim R Hallac, Galip Aydin, Yetkin Tatar
2018-02-13
Distributed, Parallel, and Cluster Computing · Computer Science
Measuring the Optimality of Hadoop Optimization
Woo-Cheol Kim, Changryong Baek, Dongwon Lee
2013-07-12
Databases · Computer Science
ReStore: Reusing Results of MapReduce Jobs
Iman Elghandour, Ashraf Aboulnaga
2012-03-02
Distributed, Parallel, and Cluster Computing · Computer Science
Complexity Measures for Map-Reduce, and Comparison to Parallel Computing
Ashish Goel, Kamesh Munagala
2012-11-29
Distributed, Parallel, and Cluster Computing · Computer Science
Thesis Report: Resource Utilization Provisioning in MapReduce
Hamidreza Barati, Nasrin Jaberi
2012-03-21
Distributed, Parallel, and Cluster Computing · Computer Science
Hadoop Mapreduce Performance Enhancement Using In-node Combiners
Woo-Hyun Lee, Hee-Gook Jun, Hyoung-Joo Kim
2015-11-17
Distributed, Parallel, and Cluster Computing · Computer Science
Statistical Regression to Predict Total Cumulative CPU Usage of MapReduce Jobs
Nikzad Babaii Rizvandi, Javid Taheri, Reza Moraveji, Albert Y. Zomaya
2013-03-18
Databases · Computer Science
Observations on Factors Affecting Performance of MapReduce based Apriori on Hadoop Cluster
Sudhakar Singh, Rakhi Garg, P. K. Mishra
2017-01-24
R2 v1 2026-06-21T18:18:02.446Z