Abstract
This paper addresses the problem of building a grid file system for applications that need to manipulate huge data, distributed and concurrently accessed at a very large scale. In this paper we explore how this goal could be reached through a cooperation between the Gfarm grid file system and BlobSeer, a distributed object management system specifically designed for huge data management under heavy concurrency. The resulting BLOB-based grid file system exhibits scalable file access performance in scenarios where huge files are subject to massive, concurrent, fine-grain accesses. This is demonstrated through preliminary experiments of our prototype, conducted on the Grid’5000 testbed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
1 The Grid Security Infrastructure Working Group. http://www.gridforum.org/ security/gsi/.
TheGrid’5000Project. http://www.grid5000.fr/.
Bill Allccck, Joe Bester, John Bresnahan, Ann L. Chervenak, Ian Foster, Carl Kessehilan, Sam Meder, Veronika Nefedova, Dawy Quesnel, and Steven Tuecke. Data management and transfer in high-peiformance computational grid envimnments. Parallel Comput., 28(5):749— 771, 2002.
Alessandro Bassi, Micah Beck, Graham Fagg, Terry Moore, James S. Plank, Martin Swany, and Rich Wolski. The Internet Backplane Protocol: A study in resource sharing. In Proc. 2nd IEEE/ACM Intl. Symp. on Cluster Computing and the Grid (CCGRJD ‘02), page 194, Washington, DC, USA, 2002. IEEE Computer Society.
Philip H. Cans, Walter B. Ligon, Robert B. Ross, and Rajeev Thakur. PVFS: A parallel file system for linux clusters. In Proceedings of the 4th Annual Linux Showcase and Conference, pages 317—327, Atlanta, GA, 2000. USENIX Association.
Ananth Devulapalli, Dennis Dalessandro, Pete Wyckoff, Nawab AR, and P. Sadayappan. Integrating parallel file systems with object-based stomge devices. In SC ‘07: Proceedings of the 2007ACM/IEEE conference on Supercomputing, pages 1—10, New York, NY, USA, 2007. ACM.
M. Factor, K. Meth, D. Naor, 0. Rodeh, and J. Satran. Object stomge: the future building block for storage systems. In Local to Global Data Interoperability - Challenges and Technologies, 2005, pages 119—123, 2005.
Sanjay Ghemawat, Howard Gobiofi’, and Shun-Tak Leung. The Google file system. In SOSP ‘03: Proceedings of the nineteenth ACM symposium on Operating systems principles, pages 29-43, New York, NY, USA, 2003. ACM Press.
HDFS. The Hadoop Distributed File System. http: / / hadoop. apache. org/ conmon/docs/rO.2O. 1/hdfsdesign.html.
Bogdan Nicolae, Gabriel Antoniu, and Luc Bougé. Distributed management of massive data. an efficient fine grain data access scheme. In International Workslwp on High-Pc rformance Data Management in Grid Environment (HPDGrid 2008), Toulouse, 2008. Held in conjunction with VECPAR’08. Electronic prcceedings.
Bogdan Nicolae, Gabriel Antoniu, and Luc Bougé. Blobseer: How to enable efficient versioning for large object storage under heavy access concurrency. In EDBT ‘09: 2ndlnteniational Workshop on Data Management in P2P Systems (DaMaP ‘09), St Petersburg, Russia, 2009.
Bogdan Nicolae, Gabriel Antoniu, and Luc Boug. Enabling high data throughput in desktop grids through decentralized data and metadata management: The BlobSeer approach. In Proceedings of the 15th Euro-Par Conference on Parallel Processing (Euro-Par d9), Lect. Notes in Comp. Science, Delft, The Netherlands, 2009. Springer-Verlag. To appear.
P. Schwan. Lustre: Building a file system for 1000-node clusters. In Proceedings of the Linux Symposium, 2003.
Osamu Tatebe and Satoshi Sekiguchi. Gfarm v2: A grid file system that supports highperfomance distributed and parallel data computing. In Proceedings of the 2004 Computing in High Energy and Nuclear Physics, 2004.
Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn. Ceph: a scalable, high-performance distributed file system. In OSDI ‘06: Proceedings of the 7th symposium on Operating systems design and implementation, pages 307—320, Berkeley, CA, USA, 2006. USENIX Association.
Brian S. White, Michael Walker, Marty Humphrey, and Andrew S. Grimshaw. LegionFS: a secure and scalable file system supporting cross-domain high-performance applications. In Proc. 2001 ACM/IEEE Conf on Supercomputing (SC ‘01), pages 59—59, New York, NY, USA, 2001. ACM Press.
FUSE. http://fuse.sourceforge.net/.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer US
About this paper
Cite this paper
Tran, VT., Antoniu, G., Nicolae, B., Bougé, L., Tatebe, O. (2010). Towards a Grid File System Based on a Large-Scale BLOB Management Service. In: Desprez, F., Getov, V., Priol, T., Yahyapour, R. (eds) Grids, P2P and Services Computing. Springer, Boston, MA. https://doi.org/10.1007/978-1-4419-6794-7_2
Download citation
DOI: https://doi.org/10.1007/978-1-4419-6794-7_2
Published:
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4419-6793-0
Online ISBN: 978-1-4419-6794-7
eBook Packages: Computer ScienceComputer Science (R0)