Title

Introducing Map-Reduce To High End Computing

Abstract

In this work we present an scientific application that has been given a Hadoop MapReduce implementation. We also discuss other scientific fields of supercomputing that could benefit from a MapReduce implementation. We recognize in this work that Hadoop has potential benefit for more applications than simply data mining, but that it is not a panacea for all data intensive applications. We provide an example of how the halo finding application, when applied to large astrophysics datasets, benefits from the model of the Hadoop architecture. The halo finding application uses a friends of friends algorithm to quickly cluster together large sets of particles to output files which a visualization software can interpret. The current implementation requires that large datasets be moved from storage to computation resources for every simulation of astronomy data. Our Hadoop implementation allows for an in-place halo finding application on the datasets, which removes the time consuming process of transferring data between resources. © 2008 IEEE.

Publication Date

12-1-2008

Publication Title

Proceedings of the 2008 3rd Petascale Data Storage Workshop, PDSW 2008

Number of Pages

-

Document Type

Article; Proceedings Paper

Personal Identifier

scopus

DOI Link

https://doi.org/10.1109/PDSW.2008.4811889

Socpus ID

70149124333 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/70149124333

This document is currently not available here.

Share

COinS