Advertisement

Reducing Rename Overhead in Full-Path-Indexed File System

  • Longhua Wang
  • Youyou Lu
  • Siyang Li
  • Fan Yang
  • Jiwu ShuEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11719)

Abstract

Full-path-indexed file systems use a key-value database to store the full path names of files and their metadata. With this pattern, the I/O efficiency can be improved because data is placed on persistent storage in scan order. However, it introduces intolerable overhead on renaming a directory because of the modification on the full path names of files under that directory. In this paper, we introduce prefix replacement mechanism on B+-tree to accelerate renaming directories on full-path-indexed file systems. It consists of three steps: pre-scan prefix deletion, key replacement and floating-split bulk insertion. Unnecessary searches and compares are reduced in these mechanisms. We use Kyoto Cabinet as the key-value database, and implement prefix replacement mechanism on it. We run tests on two benchmarks, the first is generated by Mdtest [18], and the second is the source code of Linux [19]. Compared with LocoFS [4], one kind of full-path-indexed file system, our design is about 5\(\times \) faster to rename large directories, and the performance is basically same on small directories.

Keywords

File system management Full-path-indexed file system Key-value store B+-tree 

References

  1. 1.
    McKusick, M.K., Joy, W.N., Leffler, S.J., Fabry, R.S.: A fast file system for UNIX. ACM Trans. Comput. Syst. 2(3), 181–197 (1984)CrossRefGoogle Scholar
  2. 2.
    Tao, X., Alei, L.: Small file access optimization based on GlusterFS. In: Proceedings of 2014 International Conference on Cloud Computing and Internet of Things, pp. 101–104 (2014)Google Scholar
  3. 3.
    Jannen, W., Yuan, J., Yang, Z., Esmet, J., Esmet, J., Jiao, Y.: BetrFS: a right-optimized write-optimized file system. In: 13th Conference on File and Storage Technologies, pp. 301–315 (2015)CrossRefGoogle Scholar
  4. 4.
    Li, S., Lu, Y., Shu, J., Hu, Y., Li, T.: LocoFS: a loosely-coupled metadata service for distributed file systems. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 4–15 (2017)Google Scholar
  5. 5.
    Esmet, J., Bender, M.A., Farach-Colton, M., Kuszmaul, B.C.: The TokuFS streaming file system. In: USENIX Conference on Hot Topics in Storage & File Systems (2012)Google Scholar
  6. 6.
    Jannen, W., Yuan, J., Zhan, Y., Akshintala, A., Esmet, J., Jiao, Y., et al.: BetrFS: write-optimization in a kernel file system. ACM Trans. Storage 11(4), 1–29 (2015)CrossRefGoogle Scholar
  7. 7.
    Ren, K., Gibson, G.A.: TABLEFS: enhancing metadata efficiency in the local file system. In: USENIX Annual Technical Conference, pp. 145–156 (2013)Google Scholar
  8. 8.
    Yuan, J., Zhan, Y., Jannen, W., Pandey, P., Akshintala, A., Chandnani, K., et al.: Optimizing every operation in a write-optimized file system. In: USENIX Conference on File & Storage Technologies (2016)Google Scholar
  9. 9.
    Zhan, Y., et al.: The full path to full-path indexing. In: Proceedings of the 16th USENIX Conference on File and Storage Technologies, pp. 123–138 (2018)Google Scholar
  10. 10.
    Ren, K., Zheng, Q., Patil, S., Gibson, G.: IndexFS: scaling file system metadata performance with stateless caching and bulk insertion. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 237–248 (2014)Google Scholar
  11. 11.
    O’Neil, P., Cheng, E., Gawlick, D., O’Neil, E.: The log-structured merge-tree (LSM-tree). Acta Informatica 33(4), 351–385 (1996)CrossRefGoogle Scholar
  12. 12.
    Shetty, P.J., Spillane, R.P., Malpani, R.R., et al.: Building workload-independent storage with VT-trees. Presented as Part of the 11th USENIX Conference on File and Storage Technologies, pp. 17–30 (2013)Google Scholar
  13. 13.
    Sears, R., Ramakrishnan, R.: bLSM: a general purpose log structured merge tree. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp. 217–228 (2012)Google Scholar
  14. 14.
    Brodal, G.S., Fagerberg, R.: Lower bounds for external memory dictionaries. In: Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 546–554 (2003)Google Scholar
  15. 15.
    Davies, A., Orsaria, A.: Scale out with GlusterFS. Linux J. 2013(235), 1 (2013)Google Scholar
  16. 16.
    Patil, S., Gibson, G.A.: Scale and concurrency of giga+: file system directories with millions of files. In: USENIX Conference on File and Storage Technologies, vol. 11, p. 13 (2011)Google Scholar
  17. 17.
    Brandt, S.A., Miller, E.L., et al.: Efficient metadata management in large distributed storage systems. In: Mass Storage Systems and Technologies, pp. 290–298 (2003)Google Scholar
  18. 18.
    MDTEST Benchmark. https://github.com/MDTEST-LANL/mdtest. Accessed 16 Apr 2019
  19. 19.
  20. 20.
    Lensing, P.H., Cortes, T., Hughes, J., Brinkmann, A.: File system scalability with highly decentralized metadata on independent storage devices. In: 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 366–375 (2016)Google Scholar
  21. 21.
    Shi, X., Lin, H., Jin, H., et al.: Giraffe: a scalable distributed coordination service for large-scale systems. In: 2014 IEEE International Conference on Cluster Computing, pp. 38–47 (2014)Google Scholar
  22. 22.
    Zheng, Q., Ren, K., Gibson, G., et al.: DeltaFS: exascale file systems scale better without dedicated servers. In: Proceedings of the 10th Parallel Data Storage Workshop, pp. 1–6 (2015)Google Scholar
  23. 23.
    Lensing, P.H., Cortes, T., Brinkmann, A.: Direct lookup and hash-based metadata placement for local file systems. In: Proceedings of the 6th International Systems and Storage Conference, p. 5 (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Longhua Wang
    • 1
    • 2
  • Youyou Lu
    • 1
    • 2
  • Siyang Li
    • 1
    • 2
  • Fan Yang
    • 1
    • 2
  • Jiwu Shu
    • 1
    • 2
    Email author
  1. 1.Department of Computer Science and TechnologyTsinghua UniversityBeijingChina
  2. 2.Tsinghua National Laboratory for Information Science and TechnologyBeijingChina

Personalised recommendations