Journal of Mechanical Science and Technology

, Volume 31, Issue 4, pp 1739–1751 | Cite as

A solver for massively parallel direct numerical simulation of three-dimensional multiphase flows

  • Seungwon Shin
  • Jalel Chergui
  • Damir Juric


We present a new solver for massively parallel simulations of fully three-dimensional multiphase flows. The solver runs on a variety of computer architectures from laptops to supercomputers and on 262144 threads or more (limited only by the availability to us of more threads). The code is wholly written by the authors in Fortran 2008 and uses a domain decomposition strategy for parallelization with MPI. The fluid interface solver is based on a parallel implementation of the LCRM hybrid front tracking/level set method designed to handle highly deforming interfaces with complex topology changes. We discuss the implementation of this interface method and its particular suitability to distributed processing where all operations are carried out locally on distributed subdomains. We have developed parallel GMRES and Multigrid iterative solvers suited to the linear systems arising from the implicit solution of the fluid velocities and pressure in the presence of strong density and viscosity discontinuities across fluid phases. Particular attention is drawn to the details and performance of the parallel Multigrid solver. The code includes modules for flow interaction with immersed solid objects, contact line dynamics, species and thermal transport with phase change. Here, however, we focus on the simulation of the canonical problem of drop splash onto a liquid film and report on the parallel performance of the code on varying numbers of threads. The 3D simulations were run on mesh resolutions up to 10243 with results at the higher resolutions showing the fine details and features of droplet ejection, crown formation and rim instability observed under similar experimental conditions.


Direct numerical simulation Multiphase flow Parallel or distributed processing Interface dynamics Front tracking 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    G. Tryggvason, R. Scardovelli and S. Zaleski, Direct numerical simulations of gas-liquid multiphase flows, Cambridge University Press, Cambridge, England (2011).CrossRefMATHGoogle Scholar
  2. [2]
    C. W. Hirt and B. D. Nichols, Volume of fluid (VOF) method for the dynamics of free boundaries, J. Comp. Phys., 39 (1981) 201–225.CrossRefMATHGoogle Scholar
  3. [3]
    S. Osher and R. P. Fedkiw, Level set methods: An overview and some recent results, J. Comp. Phys., 169 (2001) 463–502.MathSciNetCrossRefMATHGoogle Scholar
  4. [4]
    D. Jamet, O. Lebaigue, N. Coutris and J. M. Delhaye, The second gradient method for the direct numerical simulation of liquid-vapor flows with phase change, J. Comp. Phys., 169 (2001) 624–651.MathSciNetCrossRefMATHGoogle Scholar
  5. [5]
    G. Tryggvason, B. Bunner, A. Esmaeeli, D. Juric, N. Al-Rawahi, W. Tauber, J. Han, S. Nas and Y.-J. Jan, A fronttracking method for the computations of multiphase flow, J. Comp. Phys., 169 (2001) 708–759.MathSciNetCrossRefMATHGoogle Scholar
  6. [6]
    W. L. George and J. A. Warren, A parallel 3D dendritic growth simulator using the phase-field method, J. Comp. Phys., 177 (2002) 264–283.CrossRefMATHGoogle Scholar
  7. [7]
    K. Wang, A. Chang, L. V. Kale and J. A. Dantzig, Parallelization of a level set method for simulating dendritic growth, J. Parallel Distrib. Comput., 66 (2006) 1379–1386.CrossRefMATHGoogle Scholar
  8. [8]
    M. Sussman, A parallelized, adaptive algorithm for multiphase flows in general geometries, Comput. and Struct., 83 (2005) 435–444.CrossRefGoogle Scholar
  9. [9]
    O. Fortmeier and H. M. Bücker, A parallel strategy for a level set simulation of droplets moving in a liquid medium, Lect. Notes Comput. Sci., 6449 (2011) 200–209.CrossRefMATHGoogle Scholar
  10. [10]
    D. Zuzio and J. L. Estivalezes, An efficient block parallel AMR method for two phase interfacial flow simulations, Computers & Fluids, 44 (2011) 339–357.MathSciNetCrossRefMATHGoogle Scholar
  11. [11]
    V. Aggarwal, V. H. Gada and A. Sharma, Parallelization methodology and performance study for Level-Set-Method based simulation of a 3-D transient two-phase flow, Numerical Heat Transfer, Part B, 63 (2013) 327–356.CrossRefGoogle Scholar
  12. [12]
    A. Banari, C. Janßen, S. T. Grilli and M. Krafczyk, Efficient GPGPU implementation of a lattice Boltzmann model for multiphase flows with high density ratios, Computers & Fluids, 93 (2014) 1–17.MathSciNetCrossRefGoogle Scholar
  13. [13]
    S. Popinet, Gerris: A tree-based adaptive solver for the incompressible Euler equations in complex geometries, J. Comp. Phys., 190 (2003) 572–600.MathSciNetCrossRefMATHGoogle Scholar
  14. [14]
    S. Popinet, An accurate adaptive solver for surface-tensiondriven interfacial flows, J. Comp. Phys., 228 (2009) 5838–5866.MathSciNetCrossRefMATHGoogle Scholar
  15. [15]
    M. J. Thoraval, K. Takehara, T. Goji Etoh, S. Popinet, P. Ray, C. Josserand, S. Zaleski and S. T. Thoroddsen, von Kármán vortex street within an impacting drop, Phys. Rev. Lett., 108 (2012) 264506.Google Scholar
  16. [16]
    C. Kuan, J. Sim and W. Shyy, Adaptive thermos-fluid moving boundary computations for interfacial dynamics, Acta Mechanica Sinica., 28 (4) (2012) 99–1021.MathSciNetCrossRefMATHGoogle Scholar
  17. [17]
    E. Uzgoren, R. Singh, J. Sim and W. Shyy, Computational modelling for multiphase flows with spacecraft application, Progress in Aerospace Science, 43 (4) (2007) 138–192.CrossRefGoogle Scholar
  18. [18]
    B. Bunner and G. Tryggvason, Direct numerical simulations of three-dimensional bubbly flows, Phys. Fluids, 11 (1999) 1967–1969.CrossRefMATHGoogle Scholar
  19. [19]
    S. Shin and D. Juric, Modeling three-dimensional multiphase flow using a level contour reconstruction method for front tracking without connectivity, J. Comp. Phys., 180 (2002) 427–470.CrossRefMATHGoogle Scholar
  20. [20]
    S. Shin, S. I. Abdel-Khalik, V. Daru and D. Juric, Accurate representation of surface tension using the level contour reconstruction method, J. Comp. Phys., 203 (2005) 493–516.CrossRefMATHGoogle Scholar
  21. [21]
    S. Shin and D. Juric, High order level contour reconstruction method, Journal of Mechanical Science and Technology, 21 (2) (2007) 311–326.CrossRefGoogle Scholar
  22. [22]
    S. Shin, Computation of the curvature field in numerical simulation of multiphase flow, J. Comp. Phys., 222 (2007) 872–878.MathSciNetCrossRefMATHGoogle Scholar
  23. [23]
    S. Shin and D. Juric, A hybrid interface method for threedimensional multiphase flows based on front-tracking and level set techniques, Int. J. Num. Meth. Fluids, 60 (2009) 753–778.CrossRefMATHGoogle Scholar
  24. [24]
    S. Shin and D. Juric, Simulation of droplet impact on a solid surface using the level contour reconstruction method, Journal of Mechanical Science and Technology, 23 (2009) 2434–2443.CrossRefGoogle Scholar
  25. [25]
    S. Shin, I. Yoon and D. Juric, The Local Front Reconstruction Method for direct simulation of two-and threedimensional multiphase flows, J. Comp. Phys., 230 (2011) 6605–6646.CrossRefMATHGoogle Scholar
  26. [26]
    M. Sussman and E. G. Puckett, A coupled level set and volume-of-fluid method computing 3D and axisymmetric incompressible two-phase flows, J. Comp. Phys., 162 (2000) 301–337.MathSciNetCrossRefMATHGoogle Scholar
  27. [27]
    E. Coyajee and B. J. Boersma, Numerical simulation of drop impact on a liquid-liquid interface with a multiple marker front-capturing method, J. Comp. Phys., 228 (2009) 4444–4467.CrossRefMATHGoogle Scholar
  28. [28]
    D. Enright, R. Fedkiw, J. Ferziger and I. Mitchell, A hybrid particle level set method for improved interface capturing, J. Comp. Phys., 183 (2002) 83–116.MathSciNetCrossRefMATHGoogle Scholar
  29. [29]
    E. Aulisa, S. Manservisi and R. Scardovelli, A mixed markers and volume-of-fluid method for the reconstruction and advection of interfaces in two-phase and free-boundary flows, J. Comp. Phys., 188 (2003) 611–639.CrossRefMATHGoogle Scholar
  30. [30]
    C. S. Peskin, Numerical analysis of blood flow in the heart, J. Comp. Phys., 25 (1977) 220–252.MathSciNetCrossRefMATHGoogle Scholar
  31. [31]
    A. J. Chorin, Numerical solution of the Navier-Stokes equations, Math. Comput., 22 (1968) 745–762.MathSciNetCrossRefMATHGoogle Scholar
  32. [32]
    F. H. Harlow and J. E. Welch, Numerical calculation of time dependent viscous incompressible flow of fluid with free surface, Phys. Fluids, 8 (1965) 2182–2189.MathSciNetCrossRefMATHGoogle Scholar
  33. [33]
    W. L. Briggs, S. McCormick and V. Henson, A Multigrid Tutorial. SIAM, Second Edition (2000).CrossRefMATHGoogle Scholar
  34. [34]
    M. Francois, E. Uzgoren, J. Jackson and W. Shyy, Multigrid computations with the immersed boundary technique for multiphase flows, International Journal of Numerical Methods for Heat & Fluid Flow, 14 (1) (2004) 98–115.CrossRefMATHGoogle Scholar
  35. [35]
    D. Y. Kwak and J. S. Lee, Multigrid algorithm for cellcentred finite difference method II: Discontinuous coefficient case, Department of Mathematics, KAIST, Daejon, Korea (2003) 305–701.Google Scholar
  36. [36]
    P. Wesseling, Cell-centred multigrid for interface problems, J. Comp. Phys., 79 (1097) 85–91.MathSciNetCrossRefMATHGoogle Scholar
  37. [37]
    N. Nikolopoulos, A. Theodorakakos and G. Bergeles, Three-dimensional numerical investigation of a droplet impinging normally onto a wall film, J. Comp. Phys., 225 (2007) 322–341.CrossRefMATHGoogle Scholar
  38. [38]
    M. Rieber and A. Frohn, A numerical study on the mechanism of splashing, International Journal of Heat and Fluid Flow, 20 (1999) 455–461.CrossRefGoogle Scholar
  39. [39]
    T. Okawa, T. Shiraishi and T. Mori, Production of secondary drops during the single water drop impact onto a plane water surface, Exp. Fluids, 41 (2006) 965–974.CrossRefGoogle Scholar
  40. [40]
    H. E. Edgerton, Stopping Time: The Photographs of Harold Edgerton, Abrams, New York (1977).Google Scholar
  41. [41]
    F. H. Harlow and J. P. Shannon, The splash of a liquid drop, J. Appl. Phys., 38 (1967) 3855–3866.CrossRefGoogle Scholar
  42. [42]
    R. Krechetnikov and G. M. Homsy, Crown-forming instability phenomena in the drop splash problem, J. Colloid and Interface Science, 331 (2009) 555–559.CrossRefGoogle Scholar
  43. [43]
    L. V. Zhang, P. Brunet, J. Eggers and R. Deegan, Wavelength selection in the crown splash, Phys. Fluids, 22 (2010) 122105.CrossRefGoogle Scholar
  44. [44]
    R. D. Deegan, P. Brunet and J. Eggers, Complexities of splashing, Nonlinearity, 21 (2008) C1–C11.MathSciNetCrossRefMATHGoogle Scholar
  45. [45]
    S. Mukherjee and J. Abraham, Crown behavior in drop impact on wet walls, Phys. Fluids, 19 (2007) 052103.CrossRefMATHGoogle Scholar
  46. [46]
    J. M. Fullana and S. Zaleski, Stability of a growing end rim in a liquid sheet of uniform thickness, Phys. Fluids, 11 (1999) 952.CrossRefMATHGoogle Scholar
  47. [47]
    D. Gueyffier and S. Zaleski, Finger formation during droplet impact on a liquid film, C.R. Acad. Sci., Ser. IIc: Chim, 326 (1998) 839.Google Scholar
  48. [48]
    C. Josserand and S. Zaleski, Droplet splashing on a thin liquid film, Phys. Fluids, 15 (2003) 1650.CrossRefMATHGoogle Scholar
  49. [49]
    G. Agbaglah, C. Josserand and S. Zaleski, Longitudinal instability of a liquid rim, Phys. Fluids, 25 (2013) 022103.CrossRefGoogle Scholar

Copyright information

© The Korean Society of Mechanical Engineers and Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  1. 1.School of Mechanical and System Design EngineeringHongik UniversitySeoulKorea
  2. 2.Laboratoire d’Informatique pour la Mécanique et les Sciences de l’Ingénieur (LIMSI)CNRS-UPR 3251OrsayFrance

Personalised recommendations