Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Massively Parallel FDF Simulation of Turbulent Reacting Flows

  • Chapter
  • First Online:
Stochastic Equations for Complex Systems

Part of the book series: Mathematical Engineering ((MATHENGIN))

Abstract

A review is presented of the evolution of a massively parallel solver for large eddy simulation (LES) of turbulent reacting flows via the filtered density function (FDF). Development of an efficient parallel implementation is particularly challenging due to the hybrid Eulerian/Lagrangian structure of typical FDF simulators. The performance of a novel parallel simulator is assessed at each of the major steps of its development. Subsequent efforts to improve scaling at each of these stages are discussed along with the prospects for further enhancements.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hawthorne WR, Weddell DS, Hottel HC (1948) Third Symp Combust Flame Explos Phenom 3(1):266. doi:10.1016/S1062-2896(49)80035-3

    Article  Google Scholar 

  2. Kuo KK, Acharya R (2012) Fundamentals of turbulent and multiphase combustion. Wiley, Hoboken

    Book  Google Scholar 

  3. Pope SB (2013) Proc Combust Inst 34(1):1. doi:10.1016/j.proci.2012.09.009

    Article  Google Scholar 

  4. Holden C (1991) Science 252:1110. doi:10.1126/science.252.5009.1107

    Google Scholar 

  5. Chen JH (2011) Proc Combust Inst 33(1):99. doi:10.1016/j.proci.2010.09.012

    Article  Google Scholar 

  6. Poinsot T, Veynante D (2011) Theoretical and numerical combustion, 3rd edn. R.T. Edwards Inc, Philadelphia

    Google Scholar 

  7. Givi P (2006) AIAA J 44(1):16. doi:10.2514/1.15514

    Article  Google Scholar 

  8. Pope SB (2000) Turbulent flows. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  9. Bilger RW (2000) Prog Energy Combust 26(4–6):367. doi:10.1016/S0360-1285(00)00015-0

    Article  Google Scholar 

  10. Peters N (2000) Turbulent combustion. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  11. Minkowycz WJ, Sparrow EM, Murthy JY (eds) (2006) Handbook of numerical heat transfer, 2nd edn. Wiley, New York

    Google Scholar 

  12. Fox RO (2003) Computational models for turbulent reacting flows. Cambridge University Press, Cambridge

    Book  Google Scholar 

  13. Heinz S (2003) Flow Turbul Combust 70(1–4):115. doi:10.1023/B:APPL.0000004933.17800.46

    Article  MATH  Google Scholar 

  14. Haworth DC (2010) Prog Energy Combust 36(2):168. doi:10.1016/j.pecs.2009.09.003

    Article  Google Scholar 

  15. Haworth DC (2011) In: Vervisch L, Veynante D, van Beeck JPAJ (eds) Turbulent combustion. von Karman institute for fluid dynamics lecture series. Rhode-Saint-Genèse, Belgium

    Google Scholar 

  16. Haworth DC, Pope SB (2011) In: Echekki T, Mastorakos E (eds) Turbulent combustion modeling, fluid mechanics and its applications, vol 95. Springer, Netherlands, pp 119–142. doi:10.1007/978-94-007-0412-1_6

  17. Ansari N, Jaberi FA, Sheikhi MRH, Givi P (2011) In: Maher ARS (ed) Engineering applications of computational fluid dynamics: volume 1. International energy and environment foundation, Chap 1, pp 1–22

    Google Scholar 

  18. Sagaut P (2010) Large eddy simulation for incompressible flows, 3rd edn. Springer, New York

    Google Scholar 

  19. Geurts BJ (2004) Elements of direct and large-eddy simulation. R.T. Edwards Inc, Philadelphia

    Google Scholar 

  20. Ghosal S, Moin P (1995) J Comput Phys 118(1):24. doi:10.1006/jcph.1995.1077

    Article  MATH  MathSciNet  Google Scholar 

  21. Vreman B, Geurts B, Kuerten H (1994) J Fluid Mech 278:351. doi:10.1017/S0022112094003745

    Article  MATH  Google Scholar 

  22. Williams FA (1985) Combustion theory, 2nd edn. The Benjamin/Cummings Publishing Company, Menlo Park

    Google Scholar 

  23. Sheikhi MRH, Drozda TG, Givi P, Pope SB (2003) Phys Fluids 15(8):2321. doi:10.1063/1.1584678

    Article  Google Scholar 

  24. Sheikhi MRH, Givi P, Pope SB (2007) Phys Fluids 19(9):095106. doi:10.1063/1.2768953

    Article  Google Scholar 

  25. Sheikhi MRH, Givi P, Pope SB (2009) Phys Fluids 21(7):075102. doi:10.1063/1.3153907

    Article  Google Scholar 

  26. Colucci PJ, Jaberi FA, Givi P, Pope SB (1998) Phys Fluids 10(2):499. doi:10.1063/1.869537

    Article  MATH  MathSciNet  Google Scholar 

  27. Jaberi FA, Colucci PJ, James S, Givi P, Pope SB (1999) J Fluid Mech 401:85. doi:10.1017/S0022112099006643

    Article  MATH  Google Scholar 

  28. Zhou XY, Pereira JCF (2000) Flow Turbul Combust 64(4):279. doi:10.1023/A:1026595626129

    Article  MATH  Google Scholar 

  29. Heinz S (2003) Flow Turbul Combust 70(1–4):153. doi:10.1023/B:APPL.0000004934.22265.74

    Article  MATH  Google Scholar 

  30. Raman V, Pitsch H, Fox RO (2005) Combust Flame 143(1–2):56. doi:10.1016/j.combustflame.2005.05.002

    Article  Google Scholar 

  31. Sheikhi MRH, Drozda TG, Givi P, Jaberi FA, Pope SB (2005) Proc Combust Inst 30(1):549. doi:10.1016/j.proci.2004.08.028

    Article  Google Scholar 

  32. Raman V, Pitsch H (2005) Combust Flame 142(4):329. doi:10.1016/j.combustflame.2005.03.014

    Article  Google Scholar 

  33. van Vliet E, Derksen JJ, van den Akker HEA (2005) AIChE J 51(3):725. doi:10.1002/aic.10365

    Article  Google Scholar 

  34. Carrara MD, DesJardin PE (2006) Int J Multiph Flow 32(3):365. doi:10.1016/j.ijmultiphaseflow.2005.11.003

    Article  MATH  Google Scholar 

  35. Mustata R, Valiéo L, Jiménez C, Jones W, Bondi S (2006) Combust Flame 145(1–2):88. doi:10.1016/j.combustflame.2005.12.002

    Article  Google Scholar 

  36. Jones WP, Navarro-Martinez S, Röhl O (2007) Proc Combust Inst 31(2):1765. doi:10.1016/j.proci.2006.07.041

    Article  Google Scholar 

  37. Jones WP, Navarro-Martinez S (2007) Combust Flame 150(3):170. doi:10.1016/j.combustflame.2007.04.003

    Article  Google Scholar 

  38. James S, Zhu J, Anand MS (2007) Proc Combust Inst 31(2):1737. doi:10.1016/j.proci.2006.07.160

    Article  Google Scholar 

  39. Chen JY (2007) Combust Theory Model 11(5):675. doi:10.1080/13647830601091723

    Article  MATH  Google Scholar 

  40. McDermott R, Pope SB (2007) J Comput Phys 226(1):947. doi:10.1016/j.jcp.2007.05.006

    Article  MATH  MathSciNet  Google Scholar 

  41. Raman V, Pitsch H (2007) Proc Combust Inst 31(2):1711. doi:10.1016/j.proci.2006.07.152

    Article  Google Scholar 

  42. Drozda TG, Sheikhi MRH, Madnia CK, Givi P (2007) Flow Turbul Combust 78(1):35. doi:10.1007/s10494-006-9052-4

    Article  MATH  Google Scholar 

  43. Réveillon J, Vervisch L (1998) AIAA J 36(3):336. doi:10.2514/2.401

    Article  Google Scholar 

  44. Cha CM, Trouillet P (2003) Phys Fluids 15(6):1496. doi:10.1063/1.1569920

    Article  Google Scholar 

  45. Yilmaz SL, Nik MB, Givi P, Strakey PA (2010) J Propuls Power 26(1):84. doi:10.2514/1.44600

    Article  Google Scholar 

  46. Ansari N, Goldin GM, Sheikhi MRH, Givi P (2011) J Comput Phys 230(19):7132. doi:10.1016/j.jcp.2011.05.015

    Article  MATH  MathSciNet  Google Scholar 

  47. Ansari N, Pisciuneri PH, Strakey PA, Givi P (2012) AIAA J 50(11):2476. doi:10.2514/1.J051671

    Article  Google Scholar 

  48. Otis CC, Ferrero P, Yilmaz SL, Candler GV, Givi P (2012) In: 48th AIAA/ASME/SAE/ASEE joint propulsion conference & exhibit. AIAA, Atlanta, GA, pp 1–11. AIAA-2012-4260. doi:10.2514/6.2012-4260

  49. Yilmaz SL, Ansari N, Pisciuneri PH, Nik MB, Otis CC, Givi P (2013) J Appl Fluid Mech 6(3):311

    Google Scholar 

  50. Gikhman II, Skorokhod AV (1972) Stochastic differential equations. Springer, New York

    Book  MATH  Google Scholar 

  51. Karlin S, Taylor HM (1981) A second course in stochastic processes. Academic Press, New York

    MATH  Google Scholar 

  52. Stratonovich RL (1963) Introduction to the theory of random noise. Gordon and Breach, New York

    Google Scholar 

  53. Risken H (1989) The Fokker-Planck equation, methods of solution and applications. Springer, New York

    Book  MATH  Google Scholar 

  54. Pope SB (1994) Phys Fluids 6(2):973. doi:10.1063/1.868329

    Article  MATH  Google Scholar 

  55. Haworth DC, Pope SB (1986) Phys Fluids 29(2):387. doi:10.1063/1.865723

    Article  MATH  MathSciNet  Google Scholar 

  56. Dreeben TD, Pope SB (1997) Phys Fluids 9(1):154. doi:10.1063/1.869157

    Article  MathSciNet  Google Scholar 

  57. Pope SB (1994) Annu Rev Fluid Mech 26:23. doi:10.1146/annurev.fl.26.010194.000323

    Article  MathSciNet  Google Scholar 

  58. Dopazo C (1994) In: Libby PA, Williams FA (eds) Turbulent reacting flows, Chap 7, Academic Press, London, pp 375–474

    Google Scholar 

  59. Borghi R (1988) Prog Energy Combust 14(4):245. doi:10.1016/0360-1285(88)90015-9

    Article  Google Scholar 

  60. Gicquel LYM, Givi P, Jaberi FA, Pope SB (2002) Phys Fluids 14(3):1196. doi:10.1063/1.1436496

    Article  MATH  MathSciNet  Google Scholar 

  61. Nik MB, Yilmaz SL, Givi P, Sheikhi MRH, Pope SB (2010) AIAA J 48(7):1513. doi:10.2514/1.50239

    Article  Google Scholar 

  62. Nik MB, Yilmaz SL, Sheikhi MRH, Givi P (2010) Flow Turbul Combust 85(3–4):677. doi:10.1007/s10494-010-9272-5

    Article  MATH  Google Scholar 

  63. Gardiner CW (1990) Handbook of stochastic methods for physics, chemistry and the natural sciences, 2nd edn. Springer, New York

    MATH  Google Scholar 

  64. Grigoriu M (1995) Applied non-Gaussian processes. Prentice-Hall, Englewood Cliffs

    MATH  Google Scholar 

  65. Kloeden PE, Platen E, Schurz H (1997) Numerical solution of stochastic differential equations through computer experiments, 2nd edn. Springer, New York

    Google Scholar 

  66. Madnia CK, Jaberi FA, Givi P (2006) In: Minkowycz WJ et al (eds) Handbook of numerical heat transfer, Chap 5, 2nd edn. Wiley, New York, pp 167–189. doi:10.1002/9780470172599.ch5

  67. Yilmaz SL, Nik MB, Sheikhi MRH, Strakey PA, Givi P (2011) J Sci Comput 47(1):109. doi:10.1007/s10915-010-9424-8

    Article  MATH  MathSciNet  Google Scholar 

  68. Gropp W, Lusk E, Skjellum A (1999) Using MPI: portable parallel programming with the message-passing interface, 2nd edn, Scientific and engineering computation. MIT Press, Cambridge

    Google Scholar 

  69. Gropp W, Lusk E, Thakur R (1999) Using MPI-2: advanced features of the message-passing interface. Scientific and engineering computation. MIT Press, Cambridge

    Google Scholar 

  70. Pisciuneri PH (2008) Large eddy simulation of a turbulent nonpremixed jet flame using a finite-rate chemistry model. M.S. thesis, Department of Mechanical Engineering and Materials Science, University of Pittsburgh, Pittsburgh, PA

    Google Scholar 

  71. Barlow RS, Frank JH (1998) Proc Combust Inst 27(1):1087. doi:10.1016/S0082-0784(98)80510-9

    Article  Google Scholar 

  72. Nooren PA, Versluis M, van der Meer TH, Barlow RS, Frank JH (2000) Appl Phys B 71(1):95. doi:10.1007/s003400000278

    Article  Google Scholar 

  73. Sandia National Laboratories (2015) TNF workshop website, piloted jet flames. http://www.sandia.gov/TNF/pilotedjet.html

  74. Kee RJ, Rupley FM, Meeks E, Miller JA (1996) CHEMKIN-III: a FORTRAN chemical kinetics package for the analysis of gas-phase chemical and plasma kinetics. Technical report. SAND96-8216, Sandia National Laboratories, Livermore, CA

    Google Scholar 

  75. Brown PN, Byrne GD, Hindmarsh AC (1989) SIAM J Sci Stat Comput 10(5):1038. doi:10.1137/0910062

    Article  MATH  MathSciNet  Google Scholar 

  76. Valiant LG (1990) Commun ACM 33(8):103. doi:10.1145/79173.79181

    Article  Google Scholar 

  77. Karypis G, Kumar V (1998) METIS: a software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices, version 4.0. University of Minnesota, Minneapolis, MN. http://glaros.dtc.umn.edu/gkhome/views/metis

  78. Karypis G, Schloegel K (2013) ParMETIS: parallel graph partitioning and sparse matrix ordering library, version 4.0. University of Minnesota, Minneapolis

    Google Scholar 

  79. Devine K, Boman E, Heaphy R, Hendrickson B, Vaughan C (2002) Comput Sci Eng 4(2):90. doi:10.1109/5992.988653

    Article  Google Scholar 

  80. Boman E, Devine K, Heaphy R, Hendrickson B, Leung V, Riesen LA, Vaughan C, Catalyurek U, Bozdag D, Mitchell W, Teresco J (2007) Zoltan 3.0: parallel partitioning, load balancing, and data-management services; user’s guide. Technical report. SAND2007-4748W, Sandia National Laboratories, Albuquerque, NM. http://www.cs.sandia.gov/Zoltan/ug_html/ug.html

  81. Chen YC, Peters N, Schneemann GA, Wruck N, Renz U, Mansour MS (1996) Combust Flame 107(3):223. doi:10.1016/S0010-2180(96)00070-3

    Article  Google Scholar 

  82. Mallampalli HP, Fletcher TH, Chen JY (1998) J Eng Gas Turbines Power 120(4):703. doi:10.1115/1.2818457

    Article  Google Scholar 

  83. Amdahl GM (1967) In: Proceedings of the April 18–20 1967, Spring joint computer conference, AFIPS’67 (Spring). ACM, New York, pp 483–485. doi:10.1145/1465482.1465560

  84. Josuttis NM (1999) The C++ standard library: a tutorial and handbook. Addison-Wesley, Reading

    Google Scholar 

  85. Boost C++ Libraries (2015). http://www.boost.org/

  86. Pisciuneri PH, Yilmaz SL, Strakey PA, Givi P (2013) SIAM J Sci Comput 35(4):C438. doi:10.1137/130911512

    Article  MATH  MathSciNet  Google Scholar 

  87. OpenMP Architecture Review Board (2015) The OpenMP API specification for parallel programming. http://www.openmp.org

  88. Oak Ridge National Laboratory (2015) Titan user guide. https://www.olcf.ornl.gov/support/system-user-guides/titan-user-guide/

  89. Texas Advanced Computing Center (2015) The University of Texas at Austin. Stampede user guide. https://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide

  90. OpenMP Architecture Review Board (2013) OpenMP application program interface version 4.0—July 2013

    Google Scholar 

  91. NVIDIA Corporation (2015) NVIDIA CUDA parallel programming and computing platform. http://www.nvidia.com/object/cuda_home_new.html

  92. OpenACC.org (2015) OpenACC directives for accelerators. http://www.openacc-standard.org

  93. Devine KD, Boman EG, Karypis G (2006) In: Heroux M, Raghavan A, Simon H (eds) Frontiers of scientific computing, Chap 1. SIAM, Philadelphia, pp 1–29

    Google Scholar 

  94. Kale LV, Bhatele A (2013) In: Parallel science and engineering applications: the Charm++ approach. Series in computational physics. CRC Press, Boca Raton, Chap 1. http://www.crcpress.com/product/isbn/9781466504127

Download references

Acknowledgments

The work at the University of Pittsburgh is sponsored by AFOSR under Grant FA9550-12-1-0057, by NSF under Grant CBET-1250171, and by the NSF Extreme Science and Engineering Discovery Environment (XSEDE) under Grants TG-CTS070055N& TG-CTS120015. We are thankful to members of the Center for Simulation and Modeling at the University of Pittsburgh for their help with numerous computational issues.

Author information

Authors and Affiliations

  1. Center for Simulation and Modeling, University of Pittsburgh, Pittsburgh, PA, USA

    P. H. Pisciuneri

  2. MathWorks Inc., Natick, MA, USA

    S. L. Yilmaz

  3. National Energy Technology Laboratory, Morgantown, WV, USA

    P. A. Strakey

  4. Department of Mechanical Engineering and Materials Science, University of Pittsburgh, Pittsburgh, PA, USA

    P. Givi

Authors
  1. P. H. Pisciuneri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. L. Yilmaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. A. Strakey
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. P. Givi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to P. H. Pisciuneri .

Editor information

Editors and Affiliations

  1. University of Wyoming, Laramie, Wyoming, USA

    Stefan Heinz

  2. University of Wyoming, Laramie, Wyoming, USA

    Hakima Bessaih

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Pisciuneri, P.H., Yilmaz, S.L., Strakey, P.A., Givi, P. (2015). Massively Parallel FDF Simulation of Turbulent Reacting Flows. In: Heinz, S., Bessaih, H. (eds) Stochastic Equations for Complex Systems. Mathematical Engineering. Springer, Cham. https://doi.org/10.1007/978-3-319-18206-3_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-18206-3_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-18205-6

  • Online ISBN: 978-3-319-18206-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation