Cluster Computing

, Volume 20, Issue 3, pp 2119–2131 | Cite as

Adaptive hybrid storage systems leveraging SSDs and HDDs in HPC cloud environments

  • Donghun Koo
  • Jik-Soo Kim
  • Soonwook Hwang
  • Hyeonsang Eom
  • Jaehwan LeeEmail author


Cloud computing should inherently support various types of data-intensive workloads with different storage access patterns. This makes a high-performance storage system in the Cloud an important component. Emerging flash device technologies such as solid state drives (SSDs) are a viable choice for building high performance computing (HPC) cloud storage systems to address more fine-grained data access patterns. However, the bit-per-dollar SSD price is still higher than the prices of HDDs. This study proposes an optimized progressive file layout (PFL) method to leverage the advantages of SSDs in a parallel file system such as Lustre so that small file I/O performance can be significantly improved. A PFL can dynamically adjust chunk sizes and stripe patterns according to various I/O traffics. Extensive experimental results show that this approach (i.e. building a hybrid storage system based on a combination of SSDs and HDDs) can actually achieve balanced throughput over mixed I/O workloads consisting of large and small file access patterns.


Progressive file layout SSD Lustre file system HPC cloud 



This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (No. NRF-2015R1C1A1A02036524), and by Institute for Information & Communications Technology Promotion (IITP) Grant Funded by the Korean government (MSIP) (No. R0190-16-2012, High Performance Big Data Analytics Platform Performance Acceleration Technologies Development).


  1. 1.
    High Performance Computing in the AWS Cloud.
  2. 2.
    Sun Oracle. Lustre Software Release 2.x Operation Manual. Accessed January 2011
  3. 3.
    Paciucci, G., Paper, S., Meyers, I., Ballantyne, D.: Developing High-Performance, Scalable, cost effective storage solutions with Intel Cloud Edition Lustre and Amazon Web Services. Reference Architecture: Developing Storage Solutions with Intel Cloud Edition for Lustre and Amazon Web Services (2015)Google Scholar
  4. 4.
    Raicu, I., Foster, I.T., Zhao, Y.: Many-task computing for grids and supercomputers 2008. In: Workshop on Many-Task Computing on Grids and Supercomputers (2008)Google Scholar
  5. 5.
    The Apache Hadoop project: open-source software for reliable, scalable, distributed computing.
  6. 6.
    Hammond, J.L.: Intel high performance data division. Progressive file layouts prototype. LUG (2015)Google Scholar
  7. 7.
    Mohr, R., Brim, M., Oral, S., Dilger, A.: Evaluating progressive file layouts for Lustre. LUG (2016)Google Scholar
  8. 8.
    Koo, D., Kim, J.-S., Hwang, S., Eom, H., Lee, J.: Utilizing Progressive File Layout Leveraging SSDs in HPC Cloud Environments. In: Proceedings of the IEEE International Workshops on Foundations and Applications of Self* Systems, September 2016Google Scholar
  9. 9.
    Benchmarking Working Group, OpenSFS. I/O Characterization of Large-Scale HPC Centers. Reference Architecture: the Supercomputing Conference (2012)Google Scholar
  10. 10.
    Lee, J., Koo, D., Park, K., Kim, J., Hwang, S.: Performance analysis of Lustre file system using high performance storage devices. KIISE Trans. Comput. Pract. 22(4), 163–169 (2016)CrossRefGoogle Scholar
  11. 11.
  12. 12.
    Prabhakar, R., Vazhkudai, S.S., Kim, Y., Butt, A.R., Li, M., Kandemir, M.: Provisioning a multi-tiered data staging area for extreme-scale machines. In: Proceedings of the 2011 31st International Conference on Distributed Computing Systems, ICDCS’11, pp. 1–12, IEEE Computer Society, Washington, DC, USA (2011)Google Scholar
  13. 13.
  14. 14.
    Liu, N., Cope, J., Carns, P., Carothers, C., Ross, R., Grider, G., Crume, A., Maltzahn, C.: On the role of burst buffers in leadership-class storage systems. In: IEEE 28th Symposium on MSST/SNAPI. IEEE (2012)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.School of Electronics and Information EngineeringKorea Aerospace UniversityGoyangRepublic of Korea
  2. 2.Department of Computer EngineeringMyongji UniversityYonginRepublic of Korea
  3. 3.Korea Institute of Science and Technology InformationDaejeonRepublic of Korea
  4. 4.Department of Computer Science and EngineeringSeoul National UniversitySeoulRepublic of Korea

Personalised recommendations