Npiy : A Novel Partitioner For Improving Mapreduce Performance

Keywords

Data skew; Data transmission amount; Hadoop; Heterogeneous; Load balance; MapReduce; Parallel image processing

Abstract

MapReduce is an effective and widely-used framework for processing large datasets in parallel over a cluster of computers. Data skew, cluster heterogeneity, and network traffic are three issues that significantly affect the performance of MapReduce applications. However, the hash-based partitioner in the native Hadoop does not consider these factors. This paper proposes a new partitioner for Yarn (Hadoop 2.6.0), namely, NPIY, which adopts an innovative parallel sampling method to distribute intermediate data. The paper makes the following major contributions: (1) NPIY mitigates data skew in MapReduce applications; (2) NPIY considers the heterogeneity of computing resources to balance the loads among Reducers; (3) NPIY reduces the network traffic in the shuffle phase by trying to retain intermediate data on those nodes running both map and reduce tasks. Compared with the native Hadoop and other popular strategies, NPIY can reduce execution time by up to 41.66% and 58.68% in homogeneous and heterogeneous clusters, respectively. We further customize NPIY for parallel image processing, and the execution time has been improved by 28.8% compared with the native Hadoop.

Publication Date

6-1-2018

Publication Title

Journal of Visual Languages and Computing

Volume

46

Number of Pages

1-11

Document Type

Article

Personal Identifier

scopus

DOI Link

https://doi.org/10.1016/j.jvlc.2018.04.001

Socpus ID

85046466118 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/85046466118

This document is currently not available here.

Share

COinS