Title

Concentric Layout, A New Scientific Data Layout For Matrix Data-Set In Hadoop File System

Keywords

data access pattern; data layout; Hadoop distributed file system; matrix file

Abstract

Due to the explosive growth in the size of scientific data-sets, data-intensive computing and analysing are an emerging trend in computational science. In these applications, data pre-processing is widely adopted because it can optimise the data layout or format beforehand to facilitate the future data access. On the other hand, current research shows an increasing popularity of MapReduce framework for large-scale data processing. However, the data access patterns which are generally applied to scientific data-set are not supported by current MapReduce framework directly. This gap motivates us to provide support for these scientific data access patterns in MapReduce framework. In our work, we study the data access patterns in matrix files and propose a new concentric data layout solution to facilitate matrix data access and analysis in MapReduce framework. Concentric data layout is a data layout which maintains the dimensional property in chunk level. Contrary to the continuous data layout adopted in the current Hadoop framework, concentric data layout stores the data from the same sub-matrix into one chunk. This layout can guarantee that the average performance of data access is optimal regardless of the various access patterns. The concentric data layout requires reorganising the data before it is being analysed or processed. Our experiments are launched on a real-world halo-finding application; the results indicate that the concentric data layout improves the overall performance by up to 38%. © 2013 Copyright Taylor and Francis Group, LLC.

Publication Date

10-1-2013

Publication Title

International Journal of Parallel, Emergent and Distributed Systems

Volume

28

Issue

5

Number of Pages

407-433

Document Type

Article

Personal Identifier

scopus

DOI Link

https://doi.org/10.1080/17445760.2012.720982

Socpus ID

84882312169 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/84882312169

This document is currently not available here.

Share

COinS