Acta Geotechnica

, Volume 4, Issue 2, pp 79–93 | Cite as

Toward petascale earthquake simulations

  • Yifeng Cui
  • Reagan Moore
  • Kim Olsen
  • Amit Chourasia
  • Philip Maechling
  • Bernard Minster
  • Steven Day
  • Yuanfang Hu
  • Jing Zhu
  • Thomas Jordan
Research Paper


Earthquakes are among the most complex terrestrial phenomena, and modeling of earthquake dynamics is one of the most challenging computational problems in science. Computational capabilities have advanced to a state where we can perform wavefield simulations for realistic three-dimensional earth models, and gain more insights into the earthquakes that threaten California and many areas of the world. The Southern California Earthquake Center initiated a major earthquake research program called TeraShake to perform physics-based numerical simulations of earthquake processes for large geographical regions, at high resolution, and for high frequencies. For a large scale simulation such as TeraShake, optimization problems tend to emerge that are not significant in smaller scale simulations. This involves both large parallel computation and also massive data management and visualization coordination. In this paper, we describe how we performed single-processor optimization of the TeraShake AWM application, optimization of the I/O handling, and optimization of initialization. We also look at the challenges presented by run-time data archive management and visualization. The improvements made to the TeraShake AWM code enabled execution on the 40k IBM Blue Gene processors and have created a community code that can be used by seismologists to perform petascale earthquake simulations.


Data management Earthquake simulation Optimization Parallel computing TeraShake Visualization 



We would like to acknowledge the contribution of Marcio Faerman, and support of the following SDSC staff members: Bryan Banister, Leesa Brieger, Sheau-Yen Chen, Dong Ju Choi, Giridhar Chukkapalli, Steve Cutchin, Larry Diegel, Nancy Wilkins-Diehr, Robert Harkness, Christopher Jordan, Tim Kaiser, George Kremenek, Yi Li, Amitava Majumdar, Jon Meyer, Arcot Rajasekar, Richard Moore, Donald Thorp, Paul Tooby, Michael Wan and Tony Vu. Dan Lapine and Jim Glasgow from NCSA supported the TeraShake job submissions and data storage at NCSA. Rajeev Thakur from ANL provided helpful recommendations for improving the application MPI-IO. SDSC’s computational collaboration effort was supported through the NSF-funded SDSC Strategic Applications Collaborations (SAC) and Strategic Community Collaborations (SCC) programs (award SCI 0438741). The TeraShake Digital Library effort is funded by the National Science Foundation Information Technology Research program through the award EAR 0122464. SDSC’s visualization effort was supported through the NSF-funded SDSC core program. This project was also funded through NSF Grant EAR-0122464 and SCI 0438741: The SCEC Community Modeling Environment—An Information Infrastructure for System-Level Earthquake Research. The simulations used NSF TeraGrid resources including SDSC Datastar, IA-64, BG/L, NCSA IA-64 and PSC XT3. Special Thanks to IBM TJ Watson Research Center for providing access and technical supports to their BG/L resource during Blue Gene Consortium Days in 2006 and 2007. We are also grateful for the editor and reviewers who spent time and effort in reviewing and commenting on this paper, in particular comments by Martin Mai to improve the manuscript.


  1. 1.
    Akcelik V, Bielak J, Biros G, Epanomeritakis I, Fernandez A, Ghattas O, Kim EJ, Lopez J, O’Hallaron D, Tu T, Urbanic J (2003) High resolution forward and inverse earthquake modeling on terascale computers. SC’03, November 15–21, Phoenix, ArizonaGoogle Scholar
  2. 2.
    Andrews DJ (1999) Test of two methods for faulting in finite-difference calculations. Bull Seismol Soc Am 89:931–937Google Scholar
  3. 3.
    Chen P, Zhao L, Jordan T (2007) Full 3D tomography for crustal structure of the Los Angeles region. Bull Seismol Soc Am 97(4):1094–1120CrossRefGoogle Scholar
  4. 4.
    Chaljub E, Komatitsch D, Vilotte JP, Capdeville Y, Festa G (2006) Spectral element analysis in seismology. Advances in wave propagation in heterogeneous earth, Ru-Shan Wu and Valerie Maupin, etds., in the series advances. In: Dmowska R (ed) Geophysics. Elsevier Academic PressGoogle Scholar
  5. 5.
    Chourasia A, Cutchin SM, Olsen KB, Minster B, Day S, Cui Y, Maechling P, Moore R, Jordan T (2007) Insights gained through visualization for large earthquake simulations. IEEE Comput Graph Appl (Discovering Unexpected) 27(5):28–34CrossRefGoogle Scholar
  6. 6.
    Cui Y, Olsen K, Hu Y, Day S, Dalguer L, Minster B, Moore R, Zhu J, Maechling P, Jordan T (2006) Optimization and scalability of a large-scale earthquake simulation application. Eos Trans AGU 87(52), Fall Meet. Suppl., Abstract: S41C-1351Google Scholar
  7. 7.
    Cui Y, Moore R, Olsen, K, Zhu J, Dalguer L, Day S, Cruz-Atienza V, Maechling P, Jordan T (2007) Mapping PetaShake applications to TeraGrid architectures. Eos Trans AGU Monograph 88(52), Fall Meet. Suppl., Abstract: IN21B-0483Google Scholar
  8. 8.
    Dalguer LA, Day S (2007) Staggered-grid split-node method for spontaneous rupture simulation. J Geophys Res 112:B02302. doi: 10.1029/2006JB004467 CrossRefGoogle Scholar
  9. 9.
    Day SM, Bielak J, Dreger D, Graves RW, Larsen S, Olsen KB, Pitarka A (2003) Tests of 3D elastodynamic codes: final report for Lifelines Project 1A02, Pacific Earthquake Engineering Research CenterGoogle Scholar
  10. 10.
    Day SM, Bradley C (2001) Memory-efficient simulation of an-elastic wave propagation. Bull Seis Soc Am 91:520–531CrossRefGoogle Scholar
  11. 11.
    Faerman M, Moore R, Cui Y, Hu Y, Zhu J, Minister B, Maechling P (2007) Managing large scale data for earthquake simulations, Journal of Grid Computing. Springer, Berlin. doi: 10.1007/s10723-007-9072-x Google Scholar
  12. 12.
    Furumura T, Chen L (2004) Large scale parallel simulation and visualization of 3D seismic wavefield using the earth simulator. Comput Model Eng Sci 6:153–168zbMATHGoogle Scholar
  13. 13.
    Furumura T, Chen L (2005) Parallel simulation of strong ground motions during recent and historical damaging earthquakes in Tokyo, Japan. Parallel Comput 31(2):149–165CrossRefGoogle Scholar
  14. 14.
    Graves RW, Somerville PG (2006) Broadband ground motion simulations for scenario ruptures of the Puente hills fault. In: Proceedings, 8th national conference on earthquake engineering, Mira, San Francisco, CA, paper #1052Google Scholar
  15. 15.
    Komatitsh D, Tromp J (1999) Introduction to the spectral element method for three-dimensional seismic wave propagation. Geophys J Int 139:806–822CrossRefGoogle Scholar
  16. 16.
    Komatitsh D, Tromp J (2003) A 14.6 billion degrees of freedom, 5 terabytes, 2.5 terabyte earthquake simulation on the Earth Simulator. In: Proceedings of the supercomputing SC0003 conferenceGoogle Scholar
  17. 17.
    Komatitsh D, Liu Q, Tromp J, Suess P, Stidham C, Shaw J (2004). Simulations of strong ground motion in the Los Angles basis based upon the spectral-element method. Bull Seismol Soc Am 94:187–206CrossRefGoogle Scholar
  18. 18.
    Levander AR (1988) Fourth-order finite-different P-SV seismograms. Geophysics 53:1425–1436CrossRefGoogle Scholar
  19. 19.
    Marcinkovich C, Olsen KB (2003) On the implementation of perfectly matched layers in a 3D fourth-order velocity-stress finite-difference scheme. J Geophys Res. 2002JB002235Google Scholar
  20. 20.
    Moczo P, Kristek J, Galis M, Pazak P, Balazovjech M (2007) The finite-difference and finite-element modeling of seismic wave propagation and earthquake motion. Acta Phys Alovaca 57(2):177–406Google Scholar
  21. 21.
    Moore R, Rajasekar A, Wan M (2005) Data grids, digital libraries and persistent archives: an integrated approach to publishing, sharing and archiving data. Spec Issue Proc IEEE Grid Comput 93(3):578–588Google Scholar
  22. 22.
    Olsen KB, Day SM, Bradley CR (2003) Estimation of Q for long-period (>2 s) waves in the Los Angeles basin. Bull Seismol Soc Am 93:627–638CrossRefGoogle Scholar
  23. 23.
    Olsen K, Day SM, Minster JB, Cui Y, Chourasia A, Faerman M, Moore R, Maechling P, Jordan T (2006) Strong shaking in Los Angeles expected from southern San Andreas earthquake. Geophys Res Lett 33:1–4Google Scholar
  24. 24.
    Olsen KB, Zhu J, Talley J (2006) Dynamic user interface for cross-plot, filtering and upload/download of time series data. Eos Trans AGU 87(52), Fall Meet. Suppl., Abstract: IN51B-0814Google Scholar
  25. 25.
    Olsen KB, Day SM, Minster JB, Cui Y, Chourasia A, Okaya D, Maechling P, Jordan T (2008) Simulation of Mw7.7 earthquakes on the southern San Andreas fault with spontaneous rupture description. Bull Seismol Soc Am (in press)Google Scholar
  26. 26.
    Olsen K, Stephenson W, Geisselmeyer A (2008) 3D Crustal structure and long-period ground motions from a M9.0 megathrust earthquake in the Pacific Northwest region. J Seismol. doi: 10.1007/s10950-007-9082-y
  27. 27.
    Peyrat S, Olsen KB, Madariaga R (2001) Dynamic modeling of the 1992 Landers earthquake. J Geophys Res 106:26467–26482CrossRefGoogle Scholar
  28. 28.
    Richardo T, Ramirez-Guzman L, Tu T, Kim EJ, Lopez J, Bielak J, Ghattas O, O’Hallaron D (2006) Scaling up TeraShake: a 1-Hz case study. Eos Trans AGU 87(52), Fall Meet. Suppl., Abstract: S51E-07Google Scholar
  29. 29.
    SCEC/CME Web Site:
  30. 30.
    TeraGrid Website:
  31. 31.
    Tu T, Yu H, Ramirez-Guzman L, Bielak J, Ghattas O, Ma KL, O’Hallaron D (2006) From mesh generation to scientific visualization: an end-to-end approach to parallel supercomputing. In: Proceedings of SC’06Google Scholar
  32. 32.
    Weldon R, Scharer K, Furnal T, Biasi G (2004) Wrightwood and the earthquake cycle: what a long recurrence record tells us about how faults work. Geol Seismol Am Today 14:4–10Google Scholar

Copyright information

© Springer-Verlag 2008

Authors and Affiliations

  • Yifeng Cui
    • 1
  • Reagan Moore
    • 1
  • Kim Olsen
    • 2
  • Amit Chourasia
    • 1
  • Philip Maechling
    • 3
  • Bernard Minster
    • 4
  • Steven Day
    • 2
  • Yuanfang Hu
    • 1
  • Jing Zhu
    • 1
  • Thomas Jordan
    • 3
  1. 1.San Diego Supercomputer CenterLa JollaUSA
  2. 2.San Diego State UniversitySan DiegoUSA
  3. 3.University of Southern CaliforniaLos AngelesUSA
  4. 4.Scripps Institution of OceanographyLa JollaUSA

Personalised recommendations