Tuning Performance Of Spark Programs

Keywords

Apache spark; Performance model

Abstract

Along with the explosive growth of data, there is a great demand to speedup the ability to process them. Although there are several platforms such as Spark that have made analysis easier to developers, the performance tuning for such platforms meanwhile becomes complex. In this paper, we propose an efficient performance optimization engine called Hedgehog to evaluate the performance based on 'Law of Diminishing Marginal Utility' and give an optimal configuration setting. The initial experiments show that our optimization can gain 19.6% performance improvement compared to the naive configuration by tuning only 3 parameters.

Publication Date

5-16-2018

Publication Title

Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018

Number of Pages

282-285

Document Type

Article; Proceedings Paper

Personal Identifier

scopus

DOI Link

https://doi.org/10.1109/IC2E.2018.00057

Socpus ID

85048350509 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/85048350509

This document is currently not available here.

Share

COinS