The Blue Gene/L Supercomputer: A Hardware and Software Story

  • José E. Moreira
  • Valentina Salapura
  • George Almasi
  • Charles Archer
  • Ralph Bellofatto
  • Peter Bergner
  • Randy Bickford
  • Mathias Blumrich
  • José R. Brunheroto
  • Arthur A. Bright
  • Michael Brutman
  • José G. Castaños
  • Dong Chen
  • Paul Coteus
  • Paul Crumley
  • Sam Ellis
  • Thomas Engelsiepen
  • Alan Gara
  • Mark Giampapa
  • Tom Gooding
  • Shawn Hall
  • Ruud A. Haring
  • Roger Haskin
  • Philip Heidelberger
  • Dirk Hoenicke
  • Todd Inglett
  • Gerrard V. Kopcsay
  • Derek Lieber
  • David Limpert
  • Pat McCarthy
  • Mark Megerian
  • Mike Mundy
  • Martin Ohmacht
  • Jeff Parker
  • Rick A. Rand
  • Don Reed
  • Ramendra Sahoo
  • Alda Sanomiya
  • Richard Shok
  • Brian Smith
  • Gordon G. Stewart
  • Todd Takken
  • Pavlos Vranas
  • Brian Wallenfelt
  • Michael Blocksome
  • Joe Ratterman
Special Issue on High-End Computing

The Blue Gene/L system at the Department of Energy Lawrence Livermore National Laboratory in Livermore, California is the world’s most powerful supercomputer. It has achieved groundbreaking performance in both standard benchmarks as well as real scientific applications. In that process, it has enabled new science that simply could not be done before. Blue Gene/L was developed by a relatively small team of dedicated scientists and engineers. This article is both a description of the Blue Gene/L supercomputer as well as an account of how that system was designed, developed, and delivered. It reports on the technical characteristics of the system that made it possible to build such a powerful supercomputer. It also reports on how teams across the world worked around the clock to accomplish this milestone of high-performance computing.

Keywords

Blue Gene system-on-a-chip hardware/software co-design ultra-scale computing parallel processing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    G. Almasi, S. Chatterjee, A. Gara, J. Gunnels, M. Gupta, A. Henning, J. E. Moreira and B. Walkup, Unlocking the Performance of the BlueGene/L Supercomputer, In Proc IEEE/ACM SC04, Pittsburgh, PA (November 2004).Google Scholar
  2. 2.
    ASC/Alliances Center for Astrophysical Thermonuclear Flashes, University of Chicago; see http://flash.uchicago.edu/website/home/.Google Scholar
  3. 3.
    A. Bright, M. Ellavsky, A. Gara, R. Haring, G. Kopcsay, R. Lembach, J. Marcella, M. Ohmacht, and V. Salapura. Creating the BlueGene/L Supercomputer from Low Power SoC ASICs, In Proc ISSCC 2005 – IEEE International Solid-State Circuits Conference (February 2005).Google Scholar
  4. 4.
    V. Bulatov, W. Cai, J. Fier, M. Hiratani, G. Hommes, T. Pierce, M. Tang, M. Rhee, R. K. Yates, and T. Arsenlis, Scalable Line Dynamics in ParaDiS, In Proc IEEE/ACM SC04, Pittsburgh, PA (November 2004).Google Scholar
  5. 5.
    A. W. Cook, W. H. Cabot, M. L. Welcome, P. L. Williams, B. J. Miller, B. R. de Supinski, and R. K. Yates, Tera-Scalable Algorithms for Variable-Density Elliptic Hydrodynamics with Spectral Accuracy, In Proc IEEE/ACM SC05, Seattle, WA (November 2005).Google Scholar
  6. 6.
    K. Davis, A. Hoisie, G. Johnson, D. J. Kerbyson, M. Lang, S. Pakin, and F. Petrini, A Performance and Scalability Analysis of the BlueGene/L Architecture, In Proc IEEE/ACM SC04, Pittsburgh, PA (November 2004).Google Scholar
  7. 7.
    B. G. Fitch, A. Rayshubskiy, M. Eleftheriou, T. J. Christopher Ward, M. E. Giampapa M. C. Pitman, and Robert S. Germain. Blue Matter: Approaching the Limits of Concurrency for Classical Molecular Dynamics, In Proc IEEE/ACM SC06, Tampa, FL (November 2006).Google Scholar
  8. 8.
    Fryxell B., Olson K., Ricker P., Timmes F.X., Zingale M., Lamb D.Q., MacNeice P., Rosner R., Tufo H. (2000). FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes. Astrophysical J. Supplement 131:273CrossRefGoogle Scholar
  9. 9.
    Gara A., Blumrich M.A., Chen D., Chiu G.L.-T., Coteus P., Giampapa M.E., Haring R.A., Heidelberger P., Hoenicke D., Kopcsay G.V., Liebsch T.A., Ohmacht M., Steinmacher-Burow B.D., Takken T., Vranas P., (March/May 2005). Overview of the Blue Gene/L System Architecture. IBM J. Res. Dev. 49(2/3):195–212Google Scholar
  10. 10.
    Germain R.S., Zhestkov Y., Eleftheriou M., Rayshubskiy A., Suits F., Ward T.J.C., Fitch B.G., (March/May 2005). Early Performance Data on the Blue Matter Molecular Simulation Framework. IBM J. Res. Dev. 49(2/3):447–456Google Scholar
  11. 11.
    D. S. Greenberg, R. Brightwell, L. A. Fisk, A. B. Maccabe, and R. E. Riesen, A System Software Architecture for High-end Computing, IEEE/ACM SC97, San Jose, CA, pp. 1–15 (November 1997).Google Scholar
  12. 12.
    F. Gygi, E. W. Draeger, M. Schulz, B. R. De Supinski, J. A. Gunnels, V. Austel, J. C. Sexton, F. Franchetti, S. Kral, C. Ueberhuber, and J. Lorenz, Large-Scale Electronic Structure Calculations of High-Z Metals on the BlueGene/L Platform, In Proc IEEE/ACM SC06 Tampa, FL (November 2006).Google Scholar
  13. 13.
    F. Gygi, E. W. Draeger, B. R. de Supinski, R. K. Yates, F. Franchetti, S. Kral, J. Lorenz, C. W. Ueberhuber, J. A. Gunnels, and J. C. Sexton. Large-scale First-Principles Molecular Dynamics Simulations on the BlueGene/L Platform using the Qbox code, IEEE/ACM SC05, Seattle, WA (November 2005).Google Scholar
  14. 14.
    A. Henning. BlueGene/L: Improving Application Memory Performance on a Massively Parallel Machine. M.E. Thesis. Cornell University (2005).Google Scholar
  15. 15.
    IBM Blue Gene Team, Blue Gene: A Vision for Protein Science using a Petaflop Supercomputer, IBM Syst. J., 40(2): (2001).Google Scholar
  16. 16.
    IBM Corporation, Blue Gene/L: Application development. 2006. http://www.redbooks. ibm.com/abstracts/sg247179.html?OpenGoogle Scholar
  17. 17.
    E. Ipek, B. R. de Supinski, M. Schulz, and S. A. McKee, An Approach to Performance Prediction for Parallel Applications, In Proc 2005 Euro-Par, Lisbon, Portugal (August 2005).Google Scholar
  18. 18.
    S. Louis, B. R. de Supinski, BlueGene/L: Early application scaling results. BlueGene System Software Workshop February 23–24, 2005, Salt Lake City, Utah. http://www-unix.mcs.anl.gov/~beckman/bluegene/SSW-Utah-2005/BGL-SSW22-LLNL-Apps.pdfGoogle Scholar
  19. 19.
    J. E. Moreira, G. Almási, C. Archer, R. Bellofatto, P. Bergner, J. R. Brunheroto, M. Brutman, J. G. Castaños, P. G. Crumley, M. Gupta, T. Inglett, D. Lieber, D. Limpert, P. McCarthy, M. Megerian, M. Mendell, M. Mundy, D. Reed, R. K. Sahoo, A. Sanomiya, R. Shok, B. Smith, and G. G. Stewart, Blue Gene/L Programming and Operating environment, IBM J. Res. Dev., 49(2/3): (March/May 2005).Google Scholar
  20. 20.
    F. Petrini, D. Kerbyson, and S. Pakin, The case of the missing supercomputer performance: achieving optimal performance on the 8,192 processors of ASCI Q. IEEE/ACM SC03, Phoenix, AZ, November 2003.Google Scholar
  21. 21.
    R. Ross, J. E. Moreira, K. Cupps, W. Pfeiffer. Parallel I/O on the IBM Blue Gene/L system. Blue Gene/L Consortium Quarterly Newsletter. Argonne National Laboratory. 1st quarter 2006. http://www-fp.mcs.anl.gov/bgconsortium/file%20system%20newsletter2.pdf.Google Scholar
  22. 22.
    V. Salapura, R. Bickford, M. Blumrich, A. Bright, D. Chen, P. Coteus, A. Gara, M. Giampapa, M. Gschwind, M. Gupta, S. Hall, R. A. Haring, P. Heidelberger, D. Hoenicke, G. V. Kopcsay, M. Ohmacht, R. A. Rand, T. Takken, and P. Vranas, Power and Performance Optimization at the System Level, In Proc ACM Computing Frontiers 2005, Ischia, Italy (May 2005).Google Scholar
  23. 23.
    V. Salapura, R. Walkup, and A. Gara, Exploiting Workload Parallelism for Performance and Power Optimization in Blue Gene, IEEE Micro (September/October 2006).Google Scholar
  24. 24.
    V. Salapura, J. R. Brunheroto, F. Redìgolo, D. Hoenicke, and A. Gara, Exploiting eDRAM Bandwidth with Data Prefetching: Simulation and Measurements, IBM internal document.Google Scholar
  25. 25.
    K. van der Schaaf, Blue Gene in the Heart of a Wide Area Sensor Network, In Proc QCDOC and Blue Gene: Next Generation of HPC Architecture Workshop. Edinburgh, UK (October 2005).Google Scholar
  26. 26.
    M. Seager, The BlueGene/L Computing Environment. Lawrence Livermore National Laboratory. October 2003. http://www.llnl.gov/asci/platforms/bluegene/papers/16seager.pdf.Google Scholar
  27. 27.
    K. Singh, E. Ipek, S. A. McKee, B. R. de Supinski, M. Schulz, and R. Caruana, Predicting Parallel Application Performance via Machine Learning Approaches, In Proc Concurrency and Computation: Practice and Experience, 2006 (To appear).Google Scholar
  28. 28.
    F. H. Streitz, J. N. Glosli, M. V. Patel, B. Chan, R. K. Yates, B. R. de Supinski, J. Sexton, J. A. Gunnels, 100+ TFlop solidification simulations on BlueGene/L, In Proc IEEE/ACM SC05, Seattle, WA (November 2005).Google Scholar
  29. 29.
    University of Mannheim, University of Tennessee, and NERSC/LBNL. TOP500 Supercomputer sites. http://www.top500.org/.Google Scholar
  30. 30.
    University of Tennessee. HPC Challenge Benchmark. http://icl.cs.utk.edu/hpcc/.Google Scholar
  31. 31.
    P. Vranas, G. Bhanot, M. Blumrich, D. Chen, A. Gara, P. Heidelberger, V. Salapura, and J. Sexton, The BlueGene/L Supercomputer and Quantum ChromoDynamics. 2006 Gordon Bell Prize. In Proc IEEE/ACM SC06, Tampa, FL (November 2006).Google Scholar
  32. 32.
    S. R. Wheat, A. B. Maccabe, R. Riesen, D. W. van Dresser, and T. M. Stallcup, PUMA: An Operating System for Massively Parallel Systems, In Proc Proceedings of the 27th Hawaii International Conference on System Sciences, pp. 56–65 (1994).Google Scholar
  33. 33.
    Wheat S.R., Maccabe A.B., Riesen R., van Dresser D.W., Stallcup T.M., (1994). PUMA: An Operating System for Massively Parallel Systems. Sci. Programming 3:275–288Google Scholar
  34. 34.
    A. Yoo, E. Chow, K. Henderson, W. McLendon, B. Hendrickson, U. Catalyurek. A scalable distributed parallel breadth-first search algorithm on BlueGene/L, In Proc IEEE/ACM SC05, Seattle, WA (November 2005).Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  • José E. Moreira
    • 1
  • Valentina Salapura
    • 1
  • George Almasi
    • 1
  • Charles Archer
    • 2
  • Ralph Bellofatto
    • 1
  • Peter Bergner
    • 2
  • Randy Bickford
    • 1
  • Mathias Blumrich
    • 1
  • José R. Brunheroto
    • 1
  • Arthur A. Bright
    • 1
  • Michael Brutman
    • 2
  • José G. Castaños
    • 1
  • Dong Chen
    • 1
  • Paul Coteus
    • 1
  • Paul Crumley
    • 1
  • Sam Ellis
    • 2
  • Thomas Engelsiepen
    • 3
  • Alan Gara
    • 1
  • Mark Giampapa
    • 1
  • Tom Gooding
    • 2
  • Shawn Hall
    • 1
  • Ruud A. Haring
    • 1
  • Roger Haskin
    • 3
  • Philip Heidelberger
    • 1
  • Dirk Hoenicke
    • 1
  • Todd Inglett
    • 2
  • Gerrard V. Kopcsay
    • 1
  • Derek Lieber
    • 1
  • David Limpert
    • 2
  • Pat McCarthy
    • 2
  • Mark Megerian
    • 2
  • Mike Mundy
    • 2
  • Martin Ohmacht
    • 1
  • Jeff Parker
    • 2
  • Rick A. Rand
    • 1
  • Don Reed
    • 2
  • Ramendra Sahoo
    • 1
  • Alda Sanomiya
    • 1
  • Richard Shok
    • 1
  • Brian Smith
    • 2
  • Gordon G. Stewart
    • 2
  • Todd Takken
    • 1
  • Pavlos Vranas
    • 1
  • Brian Wallenfelt
    • 2
  • Michael Blocksome
    • 2
  • Joe Ratterman
    • 2
  1. 1.IBM Thomas J. Watson Research CenterYorktown HeightsUSA
  2. 2.IBM Systems and Technology GroupRochesterUSA
  3. 3.IBM Almaden Research CenterSan JoseUSA

Personalised recommendations