Frontiers of Computer Science

, Volume 8, Issue 3, pp 367–377

Hybrid hierarchy storage system in MilkyWay-2 supercomputer

  • Weixia Xu
  • Yutong Lu
  • Qiong Li
  • Enqiang Zhou
  • Zhenlong Song
  • Yong Dong
  • Wei Zhang
  • Dengping Wei
  • Xiaoming Zhang
  • Haitao Chen
  • Jianying Xing
  • Yuan Yuan
Research Article
  • 167 Downloads

Abstract

With the rapid improvement of computation capability in high performance supercomputer system, the imbalance of performance between computation subsystem and storage subsystem has become more and more serious, especially when various big data are produced ranging from tens of gigabytes up to terabytes. To reduce this gap, large-scale storage systems need to be designed and implemented with high performance and scalability.MilkyWay-2 (TH-2) supercomputer system with peak performance 54.9 Pflops, definitely has this kind of requirement for storage system. This paper mainly introduces the storage system in MilkyWay-2 supercomputer, including the hardware architecture and the parallel file system. The storage system in MilkyWay-2 supercomputer exploits a novel hybrid hierarchy storage architecture to enable high scalability of I/O clients, I/O bandwidth and storage capacity. To fit this architecture, a user level virtualized file system, named H2FS, is designed and implemented which can cooperate local storage and shared storage together into a dynamic single namespace to optimize I/O performance in IO-intensive applications. The evaluation results show that the storage system in MilkyWay-2 supercomputer can satisfy the critical requirements in large scale supercomputer, such as performance and scalability.

Keywords

supercomputer storage system file system MilkyWay-2 hybrid hierarchy 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Franks B. Taming the Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics. www.wiley.com, 2012Google Scholar
  2. 2.
    Verta O, Mastroianni C, Talia D. A super-peer model for resource discovery services in large-scale grids. Future Generation Computer Systems, 2005, 21(8): 1235–1248CrossRefGoogle Scholar
  3. 3.
    Bent J, Grider G, Kettering Br, Manzanares A, McClelland M, Torres A, Torrez A. Storage challenges at Los Alamos National Lab. In: Proceedings of the 2012 Symposium on Massive Storage Systems and Technologies. 2012: 1–5Google Scholar
  4. 4.
    Watson R W, Coyne R A. The parallel I/O architecture of the high-performance storage system. In: Proceedings of the 14th IEEE Symposium on Mass Storage Systems. 1995, 27–44CrossRefGoogle Scholar
  5. 5.
    Lofstead J, Zheng F, Liu Q, Klasky S, Oldfield R, Kordenbrock T, Schwan K, Wolf M. Managing variability in the IO performance of petascale storage system. IEEE Computer Society, 2010: 1–12Google Scholar
  6. 6.
    Zhuge H. The Knowledge Grid. Singapore: World Scientific, 2004CrossRefGoogle Scholar
  7. 7.
    Oldfield R A, Maccabe A B, Arunagiri S, Kordenbrock T, Riesen R, Ward L, Widener P. Lightweight I/O for scientific applications. Technical Report of Sandia National Laboratories, 2006, 1–11Google Scholar
  8. 8.
    Liu N, Cope J, Carns P H, Carothers C D, Ross R B, Grider G, Crume A, Maltzahn C. On the role of burst buffers in leadership-class storage systems. In: Proceedings of the 2012 Symposium on Massive Storage Systems and Technologies. 2012: 1–11Google Scholar
  9. 9.
    Zhou E Q, Lu Y T, Zhang W, Dong Y. H2FS: a hybrid hierarchy file-system for scalable data-intensive computing for HPC systems. Poster paper in International Supercomputing Conference. 2013Google Scholar
  10. 10.
    Lustre: A scalable, high-performance file system. Cluster File Systems Inc. Whitepaper, Version 1.0, November 2002. http://www.lustre.org/docs/whitepaper.pdf
  11. 11.
    Xie M, Lu Y T, Liu L, Cao H J, Yang X J. Implementation and evaluation of network interface and message passing services for TianHe-1A supercomputer. In: Proceedings of the 19th Annual IEEE Symposium on High Performance Interconnects. 2011, 78–86Google Scholar
  12. 12.
    Welch B, Unangst M, Abbasi Z, Gibson G, Mueller B, Small J, Zelenka J, Zhou B. Scalable performance of the panasas parallel file system. FAST, 2008, 8: 1–17Google Scholar
  13. 13.
  14. 14.
    Ryu K D, Inglett T A, Bellofatto R, Blocksome, M. A, Gooding T, Kumar S, Mamidala A R, Megerian, M G, Miller S, Nelson M T, Rosenburg B, Smith B, Van O J, Wang A, Wisniewski R W. IBM Blue Gene/Q system software stack. IBM Journal of Research and Development, 2013, 57(1/2): 1–12CrossRefGoogle Scholar
  15. 15.
    Rogers J. Power efficiency and performance with ORNL’s cray XK7 Titan. Star Craft Companion, 2012: 1040–1050Google Scholar
  16. 16.
    Yu W, Vetter J S, Oral H S. Performance characterization and optimization of parallel i/o on the cray XT. In: Proceedings of the IEEE International Symposium on Parallel and Distributed Processing. 2008, 1–11Google Scholar
  17. 17.
    Yu W, Oral S, Vetter J, Barrett R. Efficiency evaluation of Ccray XT parallel IO stack. Cray User Group Meeting, 2007, 1–9Google Scholar
  18. 18.
    Miyazaki H, Kusano Y, Shinjou N, et al. Overview of the K computer system. Fujitsu Scientific and Technical Journal, 2012, 48Google Scholar
  19. 19.
    Xing J, Xiong J, Sun N, Jie M. Adaptive and scalable metadata management to support a trillion files. In: Proceedings of the ACM Conference on High Performance Computing Networking, Storage and Analysis. 2009, 26: 1–11CrossRefGoogle Scholar
  20. 20.
    Surendra B, Chou J, Rübel O, Prabhat, Karimabadi H, Daughton W S, Roytershteyn V, Bethel E W, Howison M, Hsu K J, Lin KW, Shoshani A, Uselton A, Wu K. Parallel I/O, analysis, and visualization of a trillion particle simulation. Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society Press, 2012, 59: 1–12Google Scholar
  21. 21.
    Payne M, Widener P, Wolf M, Abbasi H, McManus S, Bridges P G, Schwan K. Exploiting latent I/O asynchrony in petascale science applications. In: Proceedings of the 4th IEEE International Conference on EScience. 2008: 410–411Google Scholar
  22. 22.
    Ali N, Carns P, Iskra K, Kimpe D, Lang S, Latham R, Ross R, Ward L, Sadayappan P. Scalable I/O forwarding framework for high-performance computing systems. In: Proceedings of the 2009 IEEE International Conference on Cluster Computing and Workshops. 2009, 1–10CrossRefGoogle Scholar
  23. 23.
    Lu Y Y, Shu J W, Li S, Yi L T. Accelerating distributed updates with asynchronous ordered writes in a parallel file system. In: Proceedings of the 2012 IEEE International Conference on Cluster Computing. 2012, 302–310CrossRefGoogle Scholar
  24. 24.
    Shepler S, Callaghan B, Robinson D, Thurlow R, Sun Microsystems Inc., Beame C, Hummingbird Ltd., Eisler M, Doveck D, Network Appliance Inc. Network file system version 4 protocol. Network, 2003, 3530Google Scholar
  25. 25.
    Goodson G, Welch B, Halevy B, Black D, Adamson A. NFSv4 pNFS extensions. Technical Report, 2005Google Scholar
  26. 26.
    Carns P H, Settlemyer B W, Ligon W B III. Using server-to-server communication in parallel file systems to simplify consistency and improve performance. In: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing. 2008, 6Google Scholar
  27. 27.
    Devulapalli A, Ohio P W. File creation strategies in a distributed metadata file system. In: Proceedings of the 2007 IEEE International on Parallel and Distributed Processing Symposium. 2007, 1–10CrossRefGoogle Scholar
  28. 28.
    Carns P, Lang S, Ross R, Vilayannur M, Kunkel J, Ludwig T. Small-file access in parallel file systems. In: Proceedings of the 2009 IEEE International Symposium on Parallel and Distributed Processing. 2009, 1–11CrossRefGoogle Scholar
  29. 29.
    Sakai K, Sumimoto S, Kurokawa M. High-performance and highly reliable file system for the K computer. FUJITSU Science Technology, 2012, 48(3): 302–209Google Scholar
  30. 30.
    Liu Q, Klasky S, Podhorszki N, Lofstead H, Abbasi C S, Chang J, Cummings D, Dinakar C, Docan S, Ethier R, Grout T, Kordenbrock Z, Lin X, Ma R, Oldfield M, Parashar A, Romosan N, Samatova K, Schwan A, Shoshani Y, Tian M, Wolf W, Yu F, Zhang F, Zheng F. ADIOS: powering I/O to extreme scale computing. 1–6Google Scholar
  31. 31.
    Lofstead J, Zheng F, Klasky S, Schwan K. Adaptable, metadata rich IO methods for portable high performance IO. In: Proceedings of the 2009 International Parallel and Distributed Processing. 2009, 1–10Google Scholar
  32. 32.
    Li J, Liao W, Choudhary A, Ross R, Thakur R, Gropp W, Latham R, Siegel A, Gallagher B, Zingale M. Parallel netCDF: a high-performance scientific I/O interface. In: Proceedings of the 2003 ACM/IEEE Conference on Supercomputing. 2003, 39CrossRefGoogle Scholar

Copyright information

© Higher Education Press and Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Weixia Xu
    • 1
    • 2
  • Yutong Lu
    • 1
    • 2
  • Qiong Li
    • 2
  • Enqiang Zhou
    • 1
    • 2
  • Zhenlong Song
    • 2
  • Yong Dong
    • 1
    • 2
  • Wei Zhang
    • 1
    • 2
  • Dengping Wei
    • 2
  • Xiaoming Zhang
    • 2
  • Haitao Chen
    • 1
    • 2
  • Jianying Xing
    • 2
  • Yuan Yuan
    • 2
  1. 1.State Key Laboratory of High Performance ComputingChangshaChina
  2. 2.College of ComputerNational University of Defense TechnologyChangshaChina

Personalised recommendations