Enhancing UNICORE Storage Management Using Hadoop Distributed File System

  • Wasim Bari
  • Ahmed Shiraz Memon
  • Bernd Schuller
Conference paper

DOI: 10.1007/978-3-642-14122-5_39

Volume 6043 of the book series Lecture Notes in Computer Science (LNCS)
Cite this paper as:
Bari W., Memon A.S., Schuller B. (2010) Enhancing UNICORE Storage Management Using Hadoop Distributed File System. In: Lin HX. et al. (eds) Euro-Par 2009 – Parallel Processing Workshops. Euro-Par 2009. Lecture Notes in Computer Science, vol 6043. Springer, Berlin, Heidelberg

Abstract

UNICORE is a state of the art and well tested Grid middleware, designed for seamless and secure access to distributed resources, applications and data, in an easy to use fashion. A wide variety of UNICORE applications for example in bio-informatics generate and compute huge amounts of data. These large amounts of data are not easy to manage reliably and efficiently with the default UNICORE storage system which is using a standard file system. Hence, the current UNICORE does not support a scalable distributed storage system so far. We have integrated Apache Hadoop and its supported distributed storage/file systems into the UNICORE storage management service. Thus allows to build a UNICORE storage system providing data replication, disaster recovery, durability and elasticity. In this paper we will present the architecture and operation of a prototype called UniHadoop, which provides the integration of UNICORE and the distributed storage systems (DSS) supported by Hadoop and highlight its potential in usage scenarios.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Wasim Bari
    • 1
  • Ahmed Shiraz Memon
    • 2
  • Bernd Schuller
    • 2
  1. 1.Rheinisch-Westfälische Technische Hochschule (RWTH)AachenGermany
  2. 2.Institute of Advanced SimulationJülich Supercomputing Centre Forschungszentrum Jülich GmbHJülichGermany