Multiple-Level MPI File Write-Back and Prefetching for Blue Gene Systems

  • Javier García Blas
  • Florin Isailă
  • J. Carretero
  • Robert Latham
  • Robert Ross
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5759)


This paper presents the design and implementation of an asynchronous data-staging strategy for file accesses based on ROMIO, the most popular MPI-IO distribution, and ZeptoOS, an open source operating system solution for Blue Gene systems. We describe and evaluate a two-level file write-back implementation and a one-level prefetching solution. The experimental results demonstrate that both solutions achieve high performance through a high degree of overlap between computation, communication, and file I/O.


MPI-IO Parallel I/O Parallel File Systems Supercomputers 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Top 500 list,
  2. 2.
    Isaila, F., Garcia Blas, J., Carretero, J., Latham, R., Lang, S., Ross, R.: Latency hiding file I/O for Blue Gene systems. In: CCGRID 2009 (2009)Google Scholar
  3. 3.
    Ma, X., Winslett, M., Lee, J., Yu, S.: Improving MPI-IO Output Performance with Active Buffering Plus Threads. In: IPDPS, pp. 22–26 (2003)Google Scholar
  4. 4.
    Liao, W.-k., Coloma, K., Choudhary, A.K., Ward, L.: Cooperative Write-Behind Data Buffering for MPI I/O. In: Di Martino, B., Kranzlmüller, D., Dongarra, J. (eds.) EuroPVM/MPI 2005. LNCS, vol. 3666, pp. 102–109. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  5. 5.
    Patterson, R.H., Gibson, G.A., Ginting, E., Stodolsky, D., Zelenka, J.: Informed prefetching and caching. SIGOPS Oper. Syst. Rev., 79–95 (1995)Google Scholar
  6. 6.
    Byna, S., Chen, Y., Sun, X.H., Thakur, R., Gropp, W.: Parallel I/O prefetching using MPI file caching and I/O signatures. In: SC 2008, pp. 1–12 (2008)Google Scholar
  7. 7.
    Chang, F., Gibson, G.: Automatic I/O Hint Generation Through Speculative Execution. In: Proceedings of OSDI (1999)Google Scholar
  8. 8.
    Chen, Y., Byna, S., Sun, X.H., Thakur, R., Gropp, W.: Hiding I/O latency with pre-execution prefetching for parallel applications. In: SC 2008, pp. 1–10 (2008)Google Scholar
  9. 9.
    Yu, W., Vetter, J.S., Canon, R.S.: OPAL: An Open-Source MPI-IO Library over Cray XT. In: SNAPI 2007, pp. 41–46 (2007)Google Scholar
  10. 10.
    Yu, W., Vetter, J.: ParColl: Partitioned Collective I/O on the Cray XT. In: ICPP, pp. 562–569 (2008)Google Scholar
  11. 11.
    Sahoo, Y.H., Howson, R., et al: High performance file I/O for the Blue Gene/L supercomputer. In: HPCA, pp. 187–196 (2006)Google Scholar
  12. 12.
    Moreira, J., et al.: Designing a highly-scalable operating system: the Blue Gene/L story. In: SC 2006, p. 118 (2006)Google Scholar
  13. 13.
    ZeptoOs Project (2008),
  14. 14.
    Iskra, K., Romein, J.W., Yoshii, K., Beckman, P.: ZOID: I/O-forwarding infrastructure for petascale architectures. In: PPoPP 2008, pp. 153–162 (2008)Google Scholar
  15. 15.
    Thakur, R., Gropp, W., Lusk, E.: On Implementing MPI-IO Portably and with High Performance. In: Proc. of IOPADS, May 1999, pp. 23–32 (1999)Google Scholar
  16. 16.
    Thakur, R., Lusk, E.: An Abstract-Device Interface for Implementing Portable Parallel-I/O Interfaces. In: Proc. of The 6th Symposium on the Frontiers of Massively Parallel Computation, pp. 180–187 (1996)Google Scholar
  17. 17.
    Blas, J.G., Isaila, F., Singh, D.E., Carretero, J.: View-Based Collective I/O for MPI-IO. In: CCGRID, pp. 409–416 (2008)Google Scholar
  18. 18.
    Wong, P., der Wijngaart, R.: NAS Parallel Benchmarks I/O Version 2.4. Technical report, NASA Ames Research Center (2003)Google Scholar
  19. 19.
    Isaila, F., Blas, J.G., Carretero, J., Liao, W.K., Choudhary, A.: AHPIOS: An MPI-based ad-hoc parallel I/O system. In: Proceedings of IEEE ICPADS (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Javier García Blas
    • 1
  • Florin Isailă
    • 1
  • J. Carretero
    • 1
  • Robert Latham
    • 2
  • Robert Ross
    • 2
  1. 1.University Carlos IIISpain
  2. 2.Argonne National LaboratoryUSA

Personalised recommendations