Abstract
In the bigdata era, metadata performance is critical in modern distributed file systems. Traditionally, the metadata management strategies like the subtree partitioning method focus on keeping namespace locality, while the other ones like the hash-based mapping method aim to offer good load balance. Nevertheless, none of these methods achieve the two desirable properties simultaneously. To close this gap, in this paper, we propose a novel metadata management scheme, HSM\(^{2}\), which combines the subtree partitioning and hash-based mapping method together. We implemented HSM\(^{2}\) in CephFS, a widely deployed distributed file systems, and conducted a comprehensive set of metadata-intensive experiments. Experimental results show that HSM\(^{2}\) can achieve better namespace locality and load balance simultaneously. Compared with CephFS, HSM\(^{2}\) can reduce the completion time by 70% and achieve 3.9\(\times \) overall throughput speedup for a file-scanning workload.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abad, C.L., Luu, H., Roberts, N., Lee, K., Lu, Y., Campbell, R.H.: Metadata traces and workload models for evaluating big storage systems. In: 2012 IEEE Fifth International Conference on Utility and Cloud Computing, pp. 125–132. IEEE (2012)
Abbasi, Z., et al.: Scalable performance of the panasas parallel file system. In: FAST 2008 Proceedings of the 6th USENIX Conference on File and Storage Technologies, pp. 17–33 (2008)
Alam, S.R., El-Harake, H.N., Howard, K., Stringfellow, N., Verzelloni, F.: Parallel I/O and the metadata wall. In: Proceedings of the Sixth Workshop on Parallel Data Storage, pp. 13–18. ACM (2011)
Anderson, E.: Capture, conversion, and analysis of an intense NFS workload. In: FAST, vol. 9, pp. 139–152 (2009)
Beaver, D., Kumar, S., Li, H.C., Sobel, J., Vajgel, P., et al.: Finding a needle in haystack: Facebook’s photo storage. In: OSDI, vol. 10, pp. 1–8 (2010)
Braam, P.: The lustre storage architecture. arXiv preprint arXiv:1903.01955 (2019)
Braam, P., Callahan, M., Schwan, P., et al.: The intermezzo file system. In: Proceedings of the 3rd of the Perl Conference, O’Reilly Open Source Convention (1999)
Carns, P., Lang, S., Ross, R., Vilayannur, M., Kunkel, J., Ludwig, T.: Small-file access in parallel file systems. In: 2009 IEEE International Symposium on Parallel & Distributed Processing, IPDPS 2009, pp. 1–11. IEEE (2009)
Corbett, P.F., Feitelson, D.G.: The Vesta parallel file system. ACM Trans. Comput. Syst. (TOCS) 14(3), 225–264 (1996)
Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. In: ACM SIGOPS Operating Systems Review, vol. 37, pp. 29–43. ACM (2003)
Harter, T., et al.: Analysis of HDFS under HBase: a Facebook messages case study. In: FAST, vol. 14, p. 12 (2014)
Hertel, C.R.: Implementing CIFS: The Common Internet File System. Prentice Hall Professional, Upper Saddle River (2004)
Leung, A.W., Pasupathy, S., Goodson, G.R., Miller, E.L.: Measurement and analysis of large-scale network file system workloads. In: USENIX Annual Technical Conference, vol. 1, pp. 2–5 (2008)
Li, S., Lu, Y., Shu, J., Hu, Y., Li, T.: LocoFS: a loosely-coupled metadata service for distributed file systems. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, Denver, CO, USA, 12–17 November 2017, pp. 4:1–4:12 (2017)
Miller, E.L., Katz, R.H.: RAMA: an easy-to-use, high-performance parallel file system. Parallel Comput. 23(4–5), 419–446 (1997)
Morris, J.H., Satyanarayanan, M., Conner, M.H., Howard, J.H., Rosenthal, D.S., Smith, F.D.: Andrew: a distributed personal computing environment. Commun. ACM 29(3), 184–201 (1986)
Ousterhout, J.K., Cherenson, A.R., Douglis, F., Nelson, M.N., Welch, B.B.: The sprite network operating system. Computer 21(2), 23–36 (1988)
Pawlowski, B., Juszczak, C., Staubach, P., Smith, C., Lebel, D., Hitz, D.: NFS version 3: Design and implementation. In: USENIX Summer, Boston, MA, pp. 137–152 (1994)
Rodeh, O., Teperman, A.: zFS-a scalable distributed file system using object disks. In: 2003 Proceedings of 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2003), pp. 207–218. IEEE (2003)
Roselli, D.S., Lorch, J.R., Anderson, T.E., et al.: A comparison of file system workloads. In: USENIX Annual Technical Conference, General Track, pp. 41–54 (2000)
Satyanarayanan, M.: Coda: a highly available file system for a distributed workstation environment. In: Proceedings of the Second Workshop on Workstation Operating Systems, pp. 114–116. IEEE (1989)
Schwan, P., et al.: Lustre: building a file system for 1000-node clusters. In: Proceedings of the 2003 Linux Symposium, vol. 2003, pp. 380–386 (2003)
Shen, Z., Shu, J., Lee, P.P.: Reconsidering single failure recovery in clustered file systems. In: 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 323–334. IEEE (2016)
Shvachko, K., Kuang, H., Radia, S., Chansler, R., et al.: The hadoop distributed file system. In: MSST, vol. 10, pp. 1–10 (2010)
Tarasov, V.: Filebench (2018). https://github.com/filebench/filebench
Thomson, A., Abadi, D.J.: CalvinFS: consistent WAN replication and scalable metadata management for distributed file systems. In: 13th USENIX Conference on File and Storage Technologies (FAST 2015), pp. 1–14 (2015)
Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D., Maltzahn, C.: Ceph: a scalable, high-performance distributed file system. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation, pp. 307–320. USENIX Association (2006)
Weil, S.A., Pollack, K.T., Brandt, S.A., Miller, E.L.: Dynamic metadata management for petabyte-scale file systems. In: Proceedings of the 2004 ACM/IEEE Conference on Supercomputing, p. 4. IEEE Computer Society (2004)
Xiao, L., Ren, K., Zheng, Q., Gibson, G.A.: ShardFS vs. indexFS: replication vs. caching strategies for distributed metadata management in cloud storage systems. In: Proceedings of the Sixth ACM Symposium on Cloud Computing, pp. 236–249. ACM (2015)
Xing, J., Xiong, J., Sun, N., Ma, J.: Adaptive and scalable metadata management to support a trillion files. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, p. 26. ACM (2009)
Xue, L., Brandt, S.A., Miller, E.L., Long, D.D.: Efficient metadata management in large distributed file systems. In: Twentieth IEEE/Eleventh NASA Goddard Conference on Mass Storage Systems and Technologies (2003)
Zhang, S., Catanese, H., Wang, A.I.A.: The composite-file file system: decoupling the one-to-one mapping of files and metadata for better performance. In: FAST, pp. 15–22 (2016)
Acknowledgement
This work is supported in part by National Key R&D Program of China under Grant No. 2018YFB1003204, NSFC under Grant No. 61772484, and the Joint Funds of CETC under Grant No. 20166141B08080101.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, Y., Chen, Y., Shao, X., Chen, J., Yuan, L., Xu, Y. (2020). HSM\(^{2}\): A Hybrid and Scalable Metadata Management Method in Distributed File Systems. In: Shen, H., Sang, Y. (eds) Parallel Architectures, Algorithms and Programming. PAAP 2019. Communications in Computer and Information Science, vol 1163. Springer, Singapore. https://doi.org/10.1007/978-981-15-2767-8_19
Download citation
DOI: https://doi.org/10.1007/978-981-15-2767-8_19
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-2766-1
Online ISBN: 978-981-15-2767-8
eBook Packages: Computer ScienceComputer Science (R0)