Abstract
While cluster file systems exploit data striping scheme to boost large file I/O throughput, small file performance is impaired and neglected. Common metadata-based optimizations introduce obstacles such as metadata server overload and migration latency. In this paper, a novel adaptive migration strategy is incorporated into metadata-based optimization to alleviate these side effects by migrating file dynamically. Guided by proposed adaptive migration threshold model, two types of file migration are applied to reduce metadata server load without degrading current performance of file system obviously. Schemes of latency hiding and migration consistency are also introduced to reduce overhead induced by small file optimization. Our results indicate that proposed optimization can substantially improve file creation and deletion performance, and boost small file I/O throughput by more than 20%. Moreover, side effects on overall performance produced by file migration are slight and can be absorbed by improvements.
This paper is supported by Hi-tech Research and Development Program of China (863 Program, No. 2007AA01A127).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Hildebrand, D., Ward, L., Honeyman, P.: Large Files, Small Writes, and pNFS. In: Proceedings of the 20th Annual International Conference on Supercomputing, Queensland, Australia, pp. 116–124 (2006)
Shaikh, F., Chainani, M.: A Case for Small File Packing in Parallel Virtual File System (PVFS2). In: Advanced and Distributed Operating Sytems (2007)
Kuhn, M., Kunkel, J., Ludwig, T.: Directory-based Metadata Optimizations for Small Files in PVFS2. In: Luque, E., Margalef, T., Benítez, D. (eds.) Euro-Par 2008. LNCS, vol. 5168, pp. 90–99. Springer, Heidelberg (2008)
Wang, F., Xin, Q., Hong, B., Brandt, S.A., Miller, S.L.: File System Workload Analysis For Large Scale Scientific Computing Applications. In: 12th NASA Goddard Conference on Mass Storage Systems and Technologies, USA, pp. 139–152 (2004)
Roselli, D., Lorch, A.T.E.: A Comparison of File System Workloads. In: Proceedings of the 2000 USENIX Annual Technical Conference, Berkeley, CA, USA, pp. 41–54 (2000)
A. N. Laboratory. PVFS2 Online Document (2008), http://www.pvfs.org/
Devulapalli, A., Wyckoff, P.: File Creation Strategies in a Distributed Metadata file System. In: Parallel and Distributed Processing Symposium, USA, pp. 1–10 (2007)
Sebepou1, Z., Magoutis, K., Marazakis, M., Bilas, A.: A Comparative Experimental Study of Parallel File Systems for Large-Scale Data Processing. In: First USENIX Workshop on Large-Scale Computing, Berkeley, CA, USA (2008)
Katcher, J.: PostMark: A New File System Benchmark (2008), http://www.netapp.com
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Li, X., Dong, B., Xiao, L., Ruan, L. (2010). Performance Optimization of Small File I/O with Adaptive Migration Strategy in Cluster File System. In: Zhang, W., Chen, Z., Douglas, C.C., Tong, W. (eds) High Performance Computing and Applications. Lecture Notes in Computer Science, vol 5938. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11842-5_33
Download citation
DOI: https://doi.org/10.1007/978-3-642-11842-5_33
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-11841-8
Online ISBN: 978-3-642-11842-5
eBook Packages: Computer ScienceComputer Science (R0)