Massively Parallel FDF Simulation of Turbulent Reacting Flows

  • P. H. PisciuneriEmail author
  • S. L. Yilmaz
  • P. A. Strakey
  • P. Givi
Part of the Mathematical Engineering book series (MATHENGIN)


A review is presented of the evolution of a massively parallel solver for large eddy simulation (LES) of turbulent reacting flows via the filtered density function (FDF). Development of an efficient parallel implementation is particularly challenging due to the hybrid Eulerian/Lagrangian structure of typical FDF simulators. The performance of a novel parallel simulator is assessed at each of the major steps of its development. Subsequent efforts to improve scaling at each of these stages are discussed along with the prospects for further enhancements.


Monte Carlo Large Eddy Simulation Message Passing Interface Turbulent Combustion Message Passing Interface Process 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The work at the University of Pittsburgh is sponsored by AFOSR under Grant FA9550-12-1-0057, by NSF under Grant CBET-1250171, and by the NSF Extreme Science and Engineering Discovery Environment (XSEDE) under Grants TG-CTS070055N& TG-CTS120015. We are thankful to members of the Center for Simulation and Modeling at the University of Pittsburgh for their help with numerous computational issues.


  1. 1.
    Hawthorne WR, Weddell DS, Hottel HC (1948) Third Symp Combust Flame Explos Phenom 3(1):266. doi: 10.1016/S1062-2896(49)80035-3 CrossRefGoogle Scholar
  2. 2.
    Kuo KK, Acharya R (2012) Fundamentals of turbulent and multiphase combustion. Wiley, HobokenCrossRefGoogle Scholar
  3. 3.
    Pope SB (2013) Proc Combust Inst 34(1):1. doi: 10.1016/j.proci.2012.09.009 CrossRefGoogle Scholar
  4. 4.
    Holden C (1991) Science 252:1110. doi: 10.1126/science.252.5009.1107 Google Scholar
  5. 5.
    Chen JH (2011) Proc Combust Inst 33(1):99. doi: 10.1016/j.proci.2010.09.012 CrossRefGoogle Scholar
  6. 6.
    Poinsot T, Veynante D (2011) Theoretical and numerical combustion, 3rd edn. R.T. Edwards Inc, PhiladelphiaGoogle Scholar
  7. 7.
    Givi P (2006) AIAA J 44(1):16. doi: 10.2514/1.15514 CrossRefGoogle Scholar
  8. 8.
    Pope SB (2000) Turbulent flows. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
  9. 9.
    Bilger RW (2000) Prog Energy Combust 26(4–6):367. doi: 10.1016/S0360-1285(00)00015-0 CrossRefGoogle Scholar
  10. 10.
    Peters N (2000) Turbulent combustion. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
  11. 11.
    Minkowycz WJ, Sparrow EM, Murthy JY (eds) (2006) Handbook of numerical heat transfer, 2nd edn. Wiley, New YorkGoogle Scholar
  12. 12.
    Fox RO (2003) Computational models for turbulent reacting flows. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  13. 13.
    Heinz S (2003) Flow Turbul Combust 70(1–4):115. doi: 10.1023/B:APPL.0000004933.17800.46 CrossRefzbMATHGoogle Scholar
  14. 14.
    Haworth DC (2010) Prog Energy Combust 36(2):168. doi: 10.1016/j.pecs.2009.09.003 CrossRefGoogle Scholar
  15. 15.
    Haworth DC (2011) In: Vervisch L, Veynante D, van Beeck JPAJ (eds) Turbulent combustion. von Karman institute for fluid dynamics lecture series. Rhode-Saint-Genèse, BelgiumGoogle Scholar
  16. 16.
    Haworth DC, Pope SB (2011) In: Echekki T, Mastorakos E (eds) Turbulent combustion modeling, fluid mechanics and its applications, vol 95. Springer, Netherlands, pp 119–142. doi: 10.1007/978-94-007-0412-1_6
  17. 17.
    Ansari N, Jaberi FA, Sheikhi MRH, Givi P (2011) In: Maher ARS (ed) Engineering applications of computational fluid dynamics: volume 1. International energy and environment foundation, Chap 1, pp 1–22Google Scholar
  18. 18.
    Sagaut P (2010) Large eddy simulation for incompressible flows, 3rd edn. Springer, New YorkGoogle Scholar
  19. 19.
    Geurts BJ (2004) Elements of direct and large-eddy simulation. R.T. Edwards Inc, PhiladelphiaGoogle Scholar
  20. 20.
    Ghosal S, Moin P (1995) J Comput Phys 118(1):24. doi: 10.1006/jcph.1995.1077 CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    Vreman B, Geurts B, Kuerten H (1994) J Fluid Mech 278:351. doi: 10.1017/S0022112094003745 CrossRefzbMATHGoogle Scholar
  22. 22.
    Williams FA (1985) Combustion theory, 2nd edn. The Benjamin/Cummings Publishing Company, Menlo ParkGoogle Scholar
  23. 23.
    Sheikhi MRH, Drozda TG, Givi P, Pope SB (2003) Phys Fluids 15(8):2321. doi: 10.1063/1.1584678 CrossRefGoogle Scholar
  24. 24.
    Sheikhi MRH, Givi P, Pope SB (2007) Phys Fluids 19(9):095106. doi: 10.1063/1.2768953 CrossRefGoogle Scholar
  25. 25.
    Sheikhi MRH, Givi P, Pope SB (2009) Phys Fluids 21(7):075102. doi: 10.1063/1.3153907 CrossRefGoogle Scholar
  26. 26.
    Colucci PJ, Jaberi FA, Givi P, Pope SB (1998) Phys Fluids 10(2):499. doi: 10.1063/1.869537 CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Jaberi FA, Colucci PJ, James S, Givi P, Pope SB (1999) J Fluid Mech 401:85. doi: 10.1017/S0022112099006643 CrossRefzbMATHGoogle Scholar
  28. 28.
    Zhou XY, Pereira JCF (2000) Flow Turbul Combust 64(4):279. doi: 10.1023/A:1026595626129 CrossRefzbMATHGoogle Scholar
  29. 29.
    Heinz S (2003) Flow Turbul Combust 70(1–4):153. doi: 10.1023/B:APPL.0000004934.22265.74 CrossRefzbMATHGoogle Scholar
  30. 30.
    Raman V, Pitsch H, Fox RO (2005) Combust Flame 143(1–2):56. doi: 10.1016/j.combustflame.2005.05.002 CrossRefGoogle Scholar
  31. 31.
    Sheikhi MRH, Drozda TG, Givi P, Jaberi FA, Pope SB (2005) Proc Combust Inst 30(1):549. doi: 10.1016/j.proci.2004.08.028 CrossRefGoogle Scholar
  32. 32.
    Raman V, Pitsch H (2005) Combust Flame 142(4):329. doi: 10.1016/j.combustflame.2005.03.014 CrossRefGoogle Scholar
  33. 33.
    van Vliet E, Derksen JJ, van den Akker HEA (2005) AIChE J 51(3):725. doi: 10.1002/aic.10365 CrossRefGoogle Scholar
  34. 34.
    Carrara MD, DesJardin PE (2006) Int J Multiph Flow 32(3):365. doi: 10.1016/j.ijmultiphaseflow.2005.11.003 CrossRefzbMATHGoogle Scholar
  35. 35.
    Mustata R, Valiéo L, Jiménez C, Jones W, Bondi S (2006) Combust Flame 145(1–2):88. doi: 10.1016/j.combustflame.2005.12.002 CrossRefGoogle Scholar
  36. 36.
    Jones WP, Navarro-Martinez S, Röhl O (2007) Proc Combust Inst 31(2):1765. doi: 10.1016/j.proci.2006.07.041 CrossRefGoogle Scholar
  37. 37.
    Jones WP, Navarro-Martinez S (2007) Combust Flame 150(3):170. doi: 10.1016/j.combustflame.2007.04.003 CrossRefGoogle Scholar
  38. 38.
    James S, Zhu J, Anand MS (2007) Proc Combust Inst 31(2):1737. doi: 10.1016/j.proci.2006.07.160 CrossRefGoogle Scholar
  39. 39.
    Chen JY (2007) Combust Theory Model 11(5):675. doi: 10.1080/13647830601091723 CrossRefzbMATHGoogle Scholar
  40. 40.
    McDermott R, Pope SB (2007) J Comput Phys 226(1):947. doi: 10.1016/ CrossRefzbMATHMathSciNetGoogle Scholar
  41. 41.
    Raman V, Pitsch H (2007) Proc Combust Inst 31(2):1711. doi: 10.1016/j.proci.2006.07.152 CrossRefGoogle Scholar
  42. 42.
    Drozda TG, Sheikhi MRH, Madnia CK, Givi P (2007) Flow Turbul Combust 78(1):35. doi: 10.1007/s10494-006-9052-4 CrossRefzbMATHGoogle Scholar
  43. 43.
    Réveillon J, Vervisch L (1998) AIAA J 36(3):336. doi: 10.2514/2.401 CrossRefGoogle Scholar
  44. 44.
    Cha CM, Trouillet P (2003) Phys Fluids 15(6):1496. doi: 10.1063/1.1569920 CrossRefGoogle Scholar
  45. 45.
    Yilmaz SL, Nik MB, Givi P, Strakey PA (2010) J Propuls Power 26(1):84. doi: 10.2514/1.44600 CrossRefGoogle Scholar
  46. 46.
    Ansari N, Goldin GM, Sheikhi MRH, Givi P (2011) J Comput Phys 230(19):7132. doi: 10.1016/ CrossRefzbMATHMathSciNetGoogle Scholar
  47. 47.
    Ansari N, Pisciuneri PH, Strakey PA, Givi P (2012) AIAA J 50(11):2476. doi: 10.2514/1.J051671 CrossRefGoogle Scholar
  48. 48.
    Otis CC, Ferrero P, Yilmaz SL, Candler GV, Givi P (2012) In: 48th AIAA/ASME/SAE/ASEE joint propulsion conference & exhibit. AIAA, Atlanta, GA, pp 1–11. AIAA-2012-4260. doi: 10.2514/6.2012-4260
  49. 49.
    Yilmaz SL, Ansari N, Pisciuneri PH, Nik MB, Otis CC, Givi P (2013) J Appl Fluid Mech 6(3):311Google Scholar
  50. 50.
    Gikhman II, Skorokhod AV (1972) Stochastic differential equations. Springer, New YorkCrossRefzbMATHGoogle Scholar
  51. 51.
    Karlin S, Taylor HM (1981) A second course in stochastic processes. Academic Press, New YorkzbMATHGoogle Scholar
  52. 52.
    Stratonovich RL (1963) Introduction to the theory of random noise. Gordon and Breach, New YorkGoogle Scholar
  53. 53.
    Risken H (1989) The Fokker-Planck equation, methods of solution and applications. Springer, New YorkCrossRefzbMATHGoogle Scholar
  54. 54.
    Pope SB (1994) Phys Fluids 6(2):973. doi: 10.1063/1.868329 CrossRefzbMATHGoogle Scholar
  55. 55.
    Haworth DC, Pope SB (1986) Phys Fluids 29(2):387. doi: 10.1063/1.865723 CrossRefzbMATHMathSciNetGoogle Scholar
  56. 56.
    Dreeben TD, Pope SB (1997) Phys Fluids 9(1):154. doi: 10.1063/1.869157 CrossRefMathSciNetGoogle Scholar
  57. 57.
    Pope SB (1994) Annu Rev Fluid Mech 26:23. doi: 10.1146/annurev.fl.26.010194.000323 CrossRefMathSciNetGoogle Scholar
  58. 58.
    Dopazo C (1994) In: Libby PA, Williams FA (eds) Turbulent reacting flows, Chap 7, Academic Press, London, pp 375–474Google Scholar
  59. 59.
    Borghi R (1988) Prog Energy Combust 14(4):245. doi: 10.1016/0360-1285(88)90015-9 CrossRefGoogle Scholar
  60. 60.
    Gicquel LYM, Givi P, Jaberi FA, Pope SB (2002) Phys Fluids 14(3):1196. doi: 10.1063/1.1436496 CrossRefzbMATHMathSciNetGoogle Scholar
  61. 61.
    Nik MB, Yilmaz SL, Givi P, Sheikhi MRH, Pope SB (2010) AIAA J 48(7):1513. doi: 10.2514/1.50239 CrossRefGoogle Scholar
  62. 62.
    Nik MB, Yilmaz SL, Sheikhi MRH, Givi P (2010) Flow Turbul Combust 85(3–4):677. doi: 10.1007/s10494-010-9272-5 CrossRefzbMATHGoogle Scholar
  63. 63.
    Gardiner CW (1990) Handbook of stochastic methods for physics, chemistry and the natural sciences, 2nd edn. Springer, New YorkzbMATHGoogle Scholar
  64. 64.
    Grigoriu M (1995) Applied non-Gaussian processes. Prentice-Hall, Englewood CliffszbMATHGoogle Scholar
  65. 65.
    Kloeden PE, Platen E, Schurz H (1997) Numerical solution of stochastic differential equations through computer experiments, 2nd edn. Springer, New YorkGoogle Scholar
  66. 66.
    Madnia CK, Jaberi FA, Givi P (2006) In: Minkowycz WJ et al (eds) Handbook of numerical heat transfer, Chap 5, 2nd edn. Wiley, New York, pp 167–189. doi: 10.1002/9780470172599.ch5
  67. 67.
    Yilmaz SL, Nik MB, Sheikhi MRH, Strakey PA, Givi P (2011) J Sci Comput 47(1):109. doi: 10.1007/s10915-010-9424-8 CrossRefzbMATHMathSciNetGoogle Scholar
  68. 68.
    Gropp W, Lusk E, Skjellum A (1999) Using MPI: portable parallel programming with the message-passing interface, 2nd edn, Scientific and engineering computation. MIT Press, CambridgeGoogle Scholar
  69. 69.
    Gropp W, Lusk E, Thakur R (1999) Using MPI-2: advanced features of the message-passing interface. Scientific and engineering computation. MIT Press, CambridgeGoogle Scholar
  70. 70.
    Pisciuneri PH (2008) Large eddy simulation of a turbulent nonpremixed jet flame using a finite-rate chemistry model. M.S. thesis, Department of Mechanical Engineering and Materials Science, University of Pittsburgh, Pittsburgh, PAGoogle Scholar
  71. 71.
    Barlow RS, Frank JH (1998) Proc Combust Inst 27(1):1087. doi: 10.1016/S0082-0784(98)80510-9 CrossRefGoogle Scholar
  72. 72.
    Nooren PA, Versluis M, van der Meer TH, Barlow RS, Frank JH (2000) Appl Phys B 71(1):95. doi: 10.1007/s003400000278 CrossRefGoogle Scholar
  73. 73.
    Sandia National Laboratories (2015) TNF workshop website, piloted jet flames.
  74. 74.
    Kee RJ, Rupley FM, Meeks E, Miller JA (1996) CHEMKIN-III: a FORTRAN chemical kinetics package for the analysis of gas-phase chemical and plasma kinetics. Technical report. SAND96-8216, Sandia National Laboratories, Livermore, CAGoogle Scholar
  75. 75.
    Brown PN, Byrne GD, Hindmarsh AC (1989) SIAM J Sci Stat Comput 10(5):1038. doi: 10.1137/0910062 CrossRefzbMATHMathSciNetGoogle Scholar
  76. 76.
    Valiant LG (1990) Commun ACM 33(8):103. doi: 10.1145/79173.79181 CrossRefGoogle Scholar
  77. 77.
    Karypis G, Kumar V (1998) METIS: a software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices, version 4.0. University of Minnesota, Minneapolis, MN.
  78. 78.
    Karypis G, Schloegel K (2013) ParMETIS: parallel graph partitioning and sparse matrix ordering library, version 4.0. University of Minnesota, MinneapolisGoogle Scholar
  79. 79.
    Devine K, Boman E, Heaphy R, Hendrickson B, Vaughan C (2002) Comput Sci Eng 4(2):90. doi: 10.1109/5992.988653 CrossRefGoogle Scholar
  80. 80.
    Boman E, Devine K, Heaphy R, Hendrickson B, Leung V, Riesen LA, Vaughan C, Catalyurek U, Bozdag D, Mitchell W, Teresco J (2007) Zoltan 3.0: parallel partitioning, load balancing, and data-management services; user’s guide. Technical report. SAND2007-4748W, Sandia National Laboratories, Albuquerque, NM.
  81. 81.
    Chen YC, Peters N, Schneemann GA, Wruck N, Renz U, Mansour MS (1996) Combust Flame 107(3):223. doi: 10.1016/S0010-2180(96)00070-3 CrossRefGoogle Scholar
  82. 82.
    Mallampalli HP, Fletcher TH, Chen JY (1998) J Eng Gas Turbines Power 120(4):703. doi: 10.1115/1.2818457 CrossRefGoogle Scholar
  83. 83.
    Amdahl GM (1967) In: Proceedings of the April 18–20 1967, Spring joint computer conference, AFIPS’67 (Spring). ACM, New York, pp 483–485. doi: 10.1145/1465482.1465560
  84. 84.
    Josuttis NM (1999) The C++ standard library: a tutorial and handbook. Addison-Wesley, ReadingGoogle Scholar
  85. 85.
    Boost C++ Libraries (2015).
  86. 86.
    Pisciuneri PH, Yilmaz SL, Strakey PA, Givi P (2013) SIAM J Sci Comput 35(4):C438. doi: 10.1137/130911512 CrossRefzbMATHMathSciNetGoogle Scholar
  87. 87.
    OpenMP Architecture Review Board (2015) The OpenMP API specification for parallel programming.
  88. 88.
    Oak Ridge National Laboratory (2015) Titan user guide.
  89. 89.
    Texas Advanced Computing Center (2015) The University of Texas at Austin. Stampede user guide.
  90. 90.
    OpenMP Architecture Review Board (2013) OpenMP application program interface version 4.0—July 2013Google Scholar
  91. 91.
    NVIDIA Corporation (2015) NVIDIA CUDA parallel programming and computing platform.
  92. 92. (2015) OpenACC directives for accelerators.
  93. 93.
    Devine KD, Boman EG, Karypis G (2006) In: Heroux M, Raghavan A, Simon H (eds) Frontiers of scientific computing, Chap 1. SIAM, Philadelphia, pp 1–29Google Scholar
  94. 94.
    Kale LV, Bhatele A (2013) In: Parallel science and engineering applications: the Charm++ approach. Series in computational physics. CRC Press, Boca Raton, Chap 1.

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • P. H. Pisciuneri
    • 1
    Email author
  • S. L. Yilmaz
    • 2
  • P. A. Strakey
    • 3
  • P. Givi
    • 4
  1. 1.Center for Simulation and ModelingUniversity of PittsburghPittsburghUSA
  2. 2.MathWorks Inc.NatickUSA
  3. 3.National Energy Technology LaboratoryMorgantownUSA
  4. 4.Department of Mechanical Engineering and Materials ScienceUniversity of PittsburghPittsburghUSA

Personalised recommendations