Parallel Structured Adaptive Mesh Refinement

  • Jarmo RantakokkoEmail author
  • Michael Thuné

Parallel structured adaptive mesh refinement is a technique for efficient utilization of computational resources. It reduces the computational effort and memory requirements needed for numerical simulation of complex phenomena, described by partial differential equations. Structured adaptive mesh refinement (SAMR) is applied in simulations where the domain is divided into logically rectangular patches, where each patch is discretized with a structured mesh. The purpose of adaptive mesh refinement is to automatically adapt the mesh to the resolution required to epresent important features of the simulated phenomenon in different subdomains. In a parallel computing context, an important consequence of the adaptation is that the dynamically changing resolution leads to a dynamically changing work load, data volume, and communication pattern at run-time. This calls for dynamic load balancing and has implications for data placement as well as parallelization granularity.

This chapter gives an overview of structured adaptive mesh refinement approaches. After a brief introductory survey of SAMR techniques and software packages, the main part of the chapter addresses various issues related to implementation of SAMR on parallel computers. In particular programming models, data placement and load balancing are discussed, for shared memory as well as distributed memory platforms. Various approaches and algorithms are presented. The appropriate choice of dynamic load balancing algorithm, data placement strategy, programming model, etc., depends on both the application state and the computer platform. There is no single best alternative under all circumstances. Consequently, the chapter ends with an account of ongoing research where the objective is to equip SAMR-based simulation software with additional adaptivity, e.g., automatic selection of load balancing algorithms and automatic decision about level of parallelization granularity using a hybrid MPI/OpenMP programming model.


Load Balance Mesh Point Adaptive Mesh Base Grid Dynamic Load Balance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    E. Steinthorsson, D. Modiano, Advanced methodology for simulation of complex flows using structured grid systems, Tech. Rep. 95-28, ICOMP, NASA Lewis Research Center, Cleveland, OH (1995).Google Scholar
  2. 2.
    L. Ferm, P. Lötstedt, P. Sjöberg, Conservative solution of the Fokker–Planck equation for stochastic chemical reactions, BIT 46 (2006) 561–583.CrossRefGoogle Scholar
  3. 3.
    M. J. Berger, J. Oliger, Adaptive mesh refinement for hyperbolic partial differential equations, Journal of Computational Physics 53 (1984) 484–512.zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    M. J. Berger, P. Colella, Local adaptive mesh refinement for shock hydrodynamics, Journal of Computational Physics 82 (1989) 64–84.zbMATHCrossRefGoogle Scholar
  5. 5.
    M. J. Berger, Data structures for adaptive grid generation, SIAM Journal on Scientific and Statistical Computing 7 (1986) 904–916.zbMATHCrossRefGoogle Scholar
  6. 6.
    J. Dreher, R. Grauer, Racoon: A parallel mesh-adaptive framework for hyperbolic conservation laws, Parallel Computing 31 (2005) 913–932.CrossRefMathSciNetGoogle Scholar
  7. 7.
    L. Ferm, P. Lötstedt, Blockwise adaptive grids with multigrid acceleration for compressible flow, AIAA J. 37 (1999) 121–123.CrossRefGoogle Scholar
  8. 8.
    P. Lötstedt, S. Söderberg, Parallel solution of hyperbolic pdes with space-time adaptivity, in: D. H. R. Vilsmeier, F. Benkhaldour (Ed.), Finite Volumes for Complex Applications II, Hermes Science, Paris, 1999, pp. 769–776.Google Scholar
  9. 9.
    P. Lötstedt, S. Söderberg, A. Ramage, L. Hemmingsson-Frändén, Implicit solution of hyperbolic equations with space-time adaptivity, BIT 42 (2002) 128–153.CrossRefGoogle Scholar
  10. 10.
    P. MacNeice et al, PARAMESH: A parallel adaptive mesh refinement community toolkit, Computer Physics Communications 126 (2000) 330–354.zbMATHCrossRefGoogle Scholar
  11. 11.
    K. G. Powell et al., A solution-adaptive upwind scheme for ideal magnetohydrodynamics, Journal of Computational Physics 154 (1999) 284–309.zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    R. A. Trompert, Local uniform grid refinement for time-dependent partial differential equations, Ph.D. thesis, University of Amsterdam (1994).Google Scholar
  13. 13.
    Z. Lan, V. E. Taylor, G. Bryan, A novel dynamic load balancing scheme for parallel systems, Journal of Parallel and Distributed Computing 62 (2002) 1763–1781.zbMATHCrossRefGoogle Scholar
  14. 14.
    P. Colella, D. T. Graves, N. D. Keen, T. J. Ligocki, D. F.Martin, P.W. McCorquodale, D. Modiano, P. O. Schwartz, T. D. Sternberg, B. V. Straalen, Chombo software package for AMR applications: Design document, Available at the Chombo website: (September 2008).
  15. 15.
    R. Deiterding, Parallel adaptive simulation of multi-dimensional detonation structures, Ph.D. thesis, Brandenburgische Technische Universität Cottbus (2003).Google Scholar
  16. 16.
    R. Deiterding, Detonation structure simulation with AMROC, in: L. Y. et. al. (Ed.), High Performance Computing and Communications, No. 3726 in Lecture Notes in Computer Science, Springer, Berlin Heidelberg, (2005), pp. 916–927.Google Scholar
  17. 17.
    M. Parashar, J. Browne, System engineering for high performance computing software: The HDDA/DAGH infrastructure for implementation of parallel structured adaptive mesh refinement, in: Structured Adaptive Mesh Refinement Grid Methods, Volume 117 of IMA Volumes in Mathematics and its Applications, Springer-Verlag, Berlin (2000), pp. 1–18.Google Scholar
  18. 18.
    C. Rendleman, V. Beckner, M. Lijewski, W. Crutchfield, J. Bell, Parallelization of structured, hierarchical adaptive mesh refinement algorithms, Computing and Visualization in Science 3 (2000) 147–157.zbMATHCrossRefGoogle Scholar
  19. 19.
    A. Wissink, R. Hornung, S. Kohn, S. Smith, N. Elliott, Large scale parallel structured AMR calculations using the SAMRAI framework, in: Proceedings of Supercomputing 2001, Denver, USA, (2001).Google Scholar
  20. 20.
    M. Berger, R. LeVeque, Adaptive mesh refinement using wave-propagation algorithms for hyperbolic systems, SIAM Journal of Numerical Analysis 35 (1998) 2298–2316.zbMATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    R. Blikberg, T. Sørevik, Load balancing and OpenMP implementation of nested parallelism, Parallel Computing 31 (2005) 984–998.CrossRefGoogle Scholar
  22. 22.
    L. Bebreu, C. Vouland, E. Blayo, AGRIF: Adaptive grid refinement in Fortran, Computers and Geosciences 34 (2008) 8–13.CrossRefGoogle Scholar
  23. 23.
    M. J. Berger, I. Rigoutsos, An algorithm for point clustering and grid generation, IEEE Transactions on Systems, Man and Cybernetics 21 (1991) 1278–1286.CrossRefGoogle Scholar
  24. 24.
    J. Pilkington, S. Baden, Dynamic partitioning of non-uniform structured workloads with spacefilling curves, IEEE Transactions on Parallel and Distributed Systems 7 (3) (1996) 288– 300.CrossRefGoogle Scholar
  25. 25.
    J. Rantakokko, Partitioning strategies for structured multiblock grids, Parallel Computing 26 (12) (2000) 1161–1680.CrossRefMathSciNetGoogle Scholar
  26. 26.
    M. Thuné, Partitioning strategies for composite grids, Parallel Algorithms and Applications 11 (1997) 325–348.MathSciNetGoogle Scholar
  27. 27.
    D. Balsara, C. Norton, Highly parallel structured adaptive mesh refinement using languagebased approaches, Journal of parallel computing 27 (2001) 37–70.zbMATHCrossRefGoogle Scholar
  28. 28.
    R. D. Hornung, S. Kohn, The SAMRAI homepage, structured adaptive mesh refinement applications infrastructure,
  29. 29.
    J. J. Quirk, A parallel adaptive grid algorithm for computational shock hydrodynamics, Applied Numerical Mathematics 20 (1996) 427–453.zbMATHCrossRefMathSciNetGoogle Scholar
  30. 30.
    M. Parashar, J. C. Browne, On partitioning dynamic adaptive grid hierarchies, presented at HICSS-29 (1996).Google Scholar
  31. 31.
    J. Steensland, Efficient partitioning of dynamic structured grid hierarchies, Ph.D. thesis, Uppsala University (2002).Google Scholar
  32. 32.
    Z. Lan, V. Taylor, G. Bryan, Dynamic load balancing of SAMR applications on distributed systems, in: Proceedings of Supercomputing 2001, (2001).Google Scholar
  33. 33.
    M. Thuné, Straightforward partitioning of composite grids for explicit difference methods, Parallel Computing 17 (1991) 665–672.zbMATHCrossRefMathSciNetGoogle Scholar
  34. 34.
    H. Johansson, A. Vakili, A patch-based partitioner for parallel SAMR applications, accepted for publication in the proceedings of the IASTED International Conference on Parallel and Distributed Computing and Systems, November 2008.Google Scholar
  35. 35.
    J. Berger, S. Bokhari, A partitioning strategy for non-uniform problems onmultiprocessors, IEEE Transactions on Computers 85 (1987) 570–580.CrossRefGoogle Scholar
  36. 36.
    S. H. Bokhari, T. W. Crockett, D. M. Nicol, Binary dissection: Variants & applications, Tech. Rep. ICASE Report No. 97-29, NASA Langley Research Center, Hampton, VA (1997).Google Scholar
  37. 37.
    J. Rantakokko, Strategies for parallel variational data assimilation, Parallel Computing 23 (1997) 2017–2039.zbMATHCrossRefGoogle Scholar
  38. 38.
    J. Rantakokko, A framework for partitioning structured grids with inhomogeneous workload, Parallel Algorithms and Applications 13 (1998) 135–152.zbMATHMathSciNetGoogle Scholar
  39. 39.
    C.-W. Ou, S. Ranka, Parallel remapping algorithms for adaptive problems, Journal of Parallel and Distributed Computing 42 (1997) 109–121.CrossRefGoogle Scholar
  40. 40.
    J. Steensland, Dynamic structured grid hierarchy partitioners using inverse space-filling curves, Tech. Rep. 2001-002, Uppsala University, Department of Information Technology, Uppsala, Sweden (2001).Google Scholar
  41. 41.
    J. Steensland, M. Thuné, S. Chandra, M. Parashar, Towards an adaptive meta-partitioner for parallel SAMR applications, in: Proceedings of the IASTED International Conference on Parallel and Distributed Computing Systems, Las Vegas, (2000), pp. 425–430.Google Scholar
  42. 42.
    J. Steensland, S. Söderberg, M. Thuné, Comparison of dynamic load balancing techniques for a parallel SAMR algorithm, in: T. Sørevik, F. Manne, R. Moe, A. H. Gebremedhin (Eds.), Applied Parallel Computing—New Paradigms for HPC in Industry and Academia, Springer-Verlag, Heidelberg, (2001), pp. 160–169, (Lecture Notes in Computer Science, Vol. 1947).CrossRefGoogle Scholar
  43. 43.
    E. N. Houstis et al., PYTHIA-II: A knowledge/database system for managing performance data and recommending scientific software, ACM TOMS 26 (2000) 227–253.zbMATHCrossRefGoogle Scholar
  44. 44.
    N. Ramakrishnan, C. J. Ribbens, Mining and visualizing recommendation spaces for elliptic PDEs with continuous attributes, ACM TOMS 26 (2000) 254–273.zbMATHCrossRefGoogle Scholar
  45. 45.
    S. Chandra, Armada: A framework for adaptive application-sensitive runtime management of dynamic applications, Master’s Thesis, Graduate School, Rutgers University, NJ (2002).Google Scholar
  46. 46.
    J. Rantakokko, Comparison of parallelization models for structured adaptive mesh refinement, in: M. Danelutto, D. Laforcena, M. Vanneschi (Eds.), Lecture Notes in Computer Science 3149, Springer-Verlag, Heidelberg (2004), pp. 615–623.Google Scholar
  47. 47.
    M. Nordén, H. Löf, J. Rantakokko, S. Holmgren, Geographical locality and dynamic data migration for OpenMP implementations of adaptive PDE solvers, in: Lecture Notes in Computer Science 4315, (2008), pp. 382–393.CrossRefGoogle Scholar
  48. 48.
    J. Rantakokko, An integrated decomposition and partitioning approach for irregular blockstructured applications, in: J. Romlin et al. (Ed.), Proceedings of the IEEE International Parallel and Distributed Processing Symposium, IPDPS 2000, Springer–Verlag, Berlin, (2000), pp. 485–496, Lecture Notes in Computer Science, Vol. 1800.Google Scholar
  49. 49.
    T. Wilhelmsson et al., Increasing resolution and forecast length with a parallel ocean model, in: Proceedings of the Second EuroGOOS International Conference, (1999).Google Scholar
  50. 50.
    T. Wilhelmsson, J. Schüle, Running an operational baltic sea model on the T3E, in: Proceedings of the Fifth European SGI/Cray MPP Workshop, Cineca, Bologna, (1999).Google Scholar
  51. 51.
    J. Rantakokko, A dynamic MPI-OpenMP model for structured adaptive mesh refinement, Parallel Processing Letters 15 (2005) 37–47.CrossRefMathSciNetGoogle Scholar
  52. 52.
    H. Johansson, Performance characterization and evaluation of parallel PDE solvers, Licentiate Thesis 2006-010, Department of Information Technology, Uppsala University (2006).Google Scholar
  53. 53.
    H. Johansson, Design and implementation of a dynamic and adaptive meta-partitioner for parallel SAMR grid hierarchies;, Technical Report 2008-017, Department of Information Technology, Uppsala University (2008).Google Scholar
  54. 54.
    The Common Component Architechture,
  55. 55.
    L. Li, B. Norris, H. Johansson, L. C. McInnes, J. Ray, Component infrastructure for managing performance data and runtime adaptation of parallel applications, accepted for publication in the Proceedings of PARA2008, Trondheim, Norway, 2008.Google Scholar
  56. 56.
    J. O. Kephart, D.M. Chess, The vision of autonomic computing, IEEE Computer 36 (1) (2003) 41–50.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Department of Information TechnologyUppsala UniversitySweden

Personalised recommendations