Managing Very-Large Distributed Datasets

  • Miguel Branco
  • Ed Zaluska
  • David de Roure
  • Pedro Salgado
  • Vincent Garonne
  • Mario Lassnig
  • Ricardo Rocha
Conference paper

DOI: 10.1007/978-3-540-88871-0_54

Part of the Lecture Notes in Computer Science book series (LNCS, volume 5331)
Cite this paper as:
Branco M. et al. (2008) Managing Very-Large Distributed Datasets. In: Meersman R., Tari Z. (eds) On the Move to Meaningful Internet Systems: OTM 2008. OTM 2008. Lecture Notes in Computer Science, vol 5331. Springer, Berlin, Heidelberg

Abstract

In this paper, we introduce a system for handling very large datasets, which need to be stored across multiple computing sites. Data distribution introduces complex management issues, particularly as computing sites may make use of different storage systems with different internal organizations. The motivation for our work is the ATLAS Experiment for the Large Hadron Collider (LHC) at CERN, where the authors are involved in developing the data management middleware. This middleware, called DQ2, is charged with shipping petabytes of data every month to research centers and universities worldwide and has achieved aggregate throughputs in excess of 1.5 Gbytes/sec over the wide-area network. We describe DQ2’s design and implementation, which builds upon previous work on distributed file systems, peer-to-peer systems and Data Grids. We discuss its fault tolerance and scalability properties and briefly describe results from its daily usage for the ATLAS Experiment.

Keywords

Data Management Data Grids Distributed Systems Grid Computing Datasets 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Miguel Branco
    • 1
  • Ed Zaluska
    • 1
  • David de Roure
    • 1
  • Pedro Salgado
    • 1
  • Vincent Garonne
    • 1
  • Mario Lassnig
    • 1
  • Ricardo Rocha
    • 1
  1. 1.CERN - European Organization for Nuclear Research, University of Southampton,UK,University of InnsbruckAustria

Personalised recommendations