Computational Geosciences

, Volume 22, Issue 3, pp 815–832 | Cite as

A distributed parallel multiple-relaxation-time lattice Boltzmann method on general-purpose graphics processing units for the rapid and scalable computation of absolute permeability from high-resolution 3D micro-CT images

  • F. O. AlpakEmail author
  • F. Gray
  • N. Saxena
  • J. Dietderich
  • R. Hofmann
  • S. Berg
Original paper


Digital rock physics (DRP) is a rapidly evolving technology targeting fast turnaround times for repeatable core analysis and multi-physics simulation of rock properties. We develop and validate a rapid and scalable distributed-parallel single-phase pore-scale flow simulator for permeability estimation on real 3D pore-scale micro-CT images using a novel variant of the lattice Boltzmann method (LBM). The LBM code implementation is designed to take maximum advantage of distributed computing on multiple general-purpose graphics processing units (GPGPUs). We describe and extensively test the distributed parallel implementation of an innovative LBM algorithm for simulating flow in pore-scale media based on the multiple-relaxation-time (MRT) model that utilizes a precise treatment of body force. While the individual components of the resulting simulator can be separately found in various references, our novel contributions are (1) the integration of all of the mathematical and high-performance computing components together with a highly optimized code implementation and (2) the delivery of quantitative results with the simulator in terms of robustness, accuracy, and computational efficiency for a variety of flow geometries including various types of real rock images. We report on extensive validations of the simulator in terms of accuracy and provide near-ideal distributed parallel scalability results on large pore-scale image volumes that were largely computationally inaccessible prior to our implementation. We validate the accuracy of the MRT-LBM simulator on model geometries with analytical solutions. Permeability estimation results are then provided on large 3D binary microstructures including a sphere pack and rocks from various sandstone and carbonate formations. We quantify the scalability behavior of the distributed parallel implementation of MRT-LBM as a function of model type/size and the number of utilized GPGPUs for a panoply of permeability estimation problems.


Lattice Boltzmann method LBM Multiple-relaxation-time MRT Digital rock physics DRP Pore-scale flow simulation Permeability Upscaling Computational fluid dynamics CFD General-purpose graphics processing unit GPGPU Parallel computing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



We thank Shell International Exploration and Production Inc. for permission to publish this paper. Farrel Gray’s doctoral studentship was provided by the Qatar Carbonates and Carbon Storage Research Centre jointly funded by Qatar Petroleum, Shell and the Qatar Science and Technology Park.


  1. 1.
    Adler, P.M., Jacquin, C.G., Quiblier, J.A.: Flow in simulated porous media. Intern. J. Multiphase Flow 16, 691–712 (1990)CrossRefGoogle Scholar
  2. 2.
    Andrä, H., Combaret, N., Dvorkin, J., Glatt, E., Han, J., Kabel, M., Keehm, Y., Krzikalla, F., Lee, M., Madonna, C., Marsh, M., Mukerji, T., Saenger, E.H., Sain, R., Saxena, N., Ricker, S., Wiegmann, A., Zhan, X.: Digital rock physics benchmarks—part I: imaging and segmentation. Comput. Geosci. 50, 25–32 (2013)CrossRefGoogle Scholar
  3. 3.
    Andrä, H., Combaret, N., Dvorkin, J., Glatt, E., Han, J., Kabel, M., Keehm, Y., Krzikalla, F., Lee, M., Madonna, C., Marsh, M., Mukerji, T., Saenger, E.H., Sain, R., Saxena, N., Ricker, S., Wiegmann, A., Zhan, X.: Digital rock physics benchmarks—part II: computing effective properties. Comput. Geosci. 50, 33–43 (2013)CrossRefGoogle Scholar
  4. 4.
    ANSYS Inc: ANSYS FLUENT Theory Guide (2010)Google Scholar
  5. 5.
    Arns, C.H., Bauget, F., Limaye, A., Sakellariou, A., Senden, T., Sheppard, A., Sok, R.M., Pinczewski, V., Bakke, S., Berge, L.I., Oren, P., Knackstedt, M.: Pore-scale characterization of carbonates using x-ray microtomography. SPE J. 10, 26–29 (2005)CrossRefGoogle Scholar
  6. 6.
    Arns, C.H., Knackstedt, M.A., Pinczewski, W.V., Garboczi, E.J.: Computation of linear elastic properties from microtomographic images: methodology and agreement between theory and experiment. Geophysics 67, 1396–1405 (2002)CrossRefGoogle Scholar
  7. 7.
    Bear, J.: Dynamics of fluids in porous media. American Elsevier Publishing Company Inc., New York (1972)Google Scholar
  8. 8.
    Benzi, R., Succi, S., Vergassola, M.: The lattice Boltzmann-equation—theory and applications. Phys. Rep. 222, 145–197 (1992)CrossRefGoogle Scholar
  9. 9.
    Berg, S., Ott, H., Klapp, S.A., Schwing, A., Neiteler, R., Brussee, N., Makurat, A., Leu, L., Enzmann, F., Schwarz, J. -O., Kersten, M., Irvine, S., Stampanoni, M.: Real-time 3D imaging of Haines jumps in porous media flow. Proc. Nat. Acad. Sci. (PNAS) 110, 3755–3759 (2013)CrossRefGoogle Scholar
  10. 10.
    Berg, S., Rücker, M., Ott, H., Georgiadis, A., van der Linde, H., Enzmann, F., Kersten, M., Armstrong, R.T., de With, S., Becker, J., Wiegmann, A.: Connected pathway relative permeability from pore-scale imaging of imbibition. Adv. Water Res. 90, 24–35 (2016)CrossRefGoogle Scholar
  11. 11.
    Bernaschi, M., Fatica, M., Melchionna, S., Succi, S., Kaxiras, E.: A flexible high-performance lattice Boltzmann GPU code for the simulations of fluid flows in complex geometries. Concurr. Comput.: Prac. Exp. 22, 1–14 (2010)CrossRefGoogle Scholar
  12. 12.
    Berryman, J.G., Blair, S.C.: Use of digital image analysis to estimate fluid permeability of porous materials: application of two-point correlation functions. J. Appl. Phys. 60, 1930–1938 (1986)CrossRefGoogle Scholar
  13. 13.
    Bhatnagar, P.L., Gross, E.P., Krook, M.: A model for collision processes in gases. I. Small amplitude processes in charged and neutral one-component systems. Phys. Rev. 94, 511–525 (1954)CrossRefGoogle Scholar
  14. 14.
    Bisson, M., Bernaschi, M., Melchionna, S., Succi, S., Kaxiras, E.: Multiscale hemodynamics using GPU clusters. Commun. Comput. Phys. 11, 48–64 (2012)CrossRefGoogle Scholar
  15. 15.
    Blunt, M.J., Bijeljic, B., Dong, H., Gharbi, O., Iglauer, S., Mostaghimi, P., Paluszny, A., Pentland, C.: Pore-scale imaging and modelling. Adv. Water Res. 51, 197–216 (2013)CrossRefGoogle Scholar
  16. 16.
    Bouzidi, M.H., Firdaouss, M., Lallemand, P.: Momentum transfer of a Boltzmann-lattice fluid with boundaries. Phys. Fluids 13, 3452–3459 (2001)CrossRefGoogle Scholar
  17. 17.
    Cancelliere, A., Chang, C., Foti, E., Rothman, D.H., Succi, S.: The permeability of a random medium: comparison of simulation with theory. Phys. Fluids A 2, 2085–2088 (1990)CrossRefGoogle Scholar
  18. 18.
    Chen, H., Teixeira, C., Molvig, K.: Digital physics approach to computational fluid dynamics: some basic theoretical features. Intern. J. Modern Phys. C 8, 675–684 (1997)CrossRefGoogle Scholar
  19. 19.
    Chen, C., Wang, Z., Majeti, D., Vrvilo, N., Warburton, T., Sarkar, V., Li, G.: Optimization of lattice Boltzmann simulation with graphics-processing-unit parallel computing and the application in reservoir characterization. SPE J. 21(3), 1425–1435 (2016)CrossRefGoogle Scholar
  20. 20.
    COMSOL Multiphysics Documentation: COMSOL Multiphysics, version 5.3, Pore-scale flow. Accessed: 4 June 2017 (2017)
  21. 21.
    d’Humières, D., Ginzburg, I., Krafczyk, M., Lallemand, P., Luo, L.: Multiple-relaxation-time lattice Boltzmann models in three dimensions. Philos. Trans. Royal Soc. A 360, 437–451 (2002)CrossRefGoogle Scholar
  22. 22.
    Dvorkin, J., Armbruster, M., Baldwin, C., Fang, Q., Derzhi, N., Gomez, C., Nur, B., Nur, A., Mu, Y.: The future of rock physics: computational methods vs. lab testing. First Break 26, 63–68 (2008)Google Scholar
  23. 23.
    Dvorkin, J., Derzhi, N., Diaz, E., Fang, Q.: Relevance of computational rock physics. Geophysics 76, E141–E153 (2011)CrossRefGoogle Scholar
  24. 24.
    Ferrèol, B., Rothman, D.H.: Lattice-Boltzmann simulations of flow through Fontainebleau sandstone. Transp. Porous Media 20, 3–20 (1995)CrossRefGoogle Scholar
  25. 25.
    Ginzburg, I., d’Humières, D.: Multireflection boundary conditions for lattice Boltzmann models. Phys. Rev. E 68, 066614 (2003)CrossRefGoogle Scholar
  26. 26.
    Gray, F., Boek, E.: Enhancing computational precision for lattice Boltzmann schemes in porous media flows. Computation 4, 11 (2016)CrossRefGoogle Scholar
  27. 27.
    Guo, Z., Zheng, C.: Analysis of lattice Boltzmann equation for microscale gas flows: relaxation times, boundary conditions and the Knudsen layer. Intern. J. Comput. Fluid Dyn. 22, 465–473 (2008)CrossRefGoogle Scholar
  28. 28.
    Guo, Z., Zheng, C., Shi, B.: Discrete lattice effects on the forcing term in the lattice Boltzmann method. Phys. Rev. E 65, 046308 (2002)CrossRefGoogle Scholar
  29. 29.
    Hassanizadeh, S.M., Gray, W.: High velocity flow in porous media. Transp. Porous Media 2, 521–531 (1987)CrossRefGoogle Scholar
  30. 30.
    He, X., Luo, L.S.: Theory of the lattice Boltzmann method: from the Boltzmann equation to the lattice Boltzmann equation. Phys. Rev. E 56, 6811–6817 (1997)CrossRefGoogle Scholar
  31. 31.
    He, X., Zou, Q., Luo, L., Dembo, M.: Analytic solutions of simple flows and analysis of nonslip boundary conditions for the lattice Boltzmann BGK model. J. Stat. Phys. 87, 115–136 (1997)CrossRefGoogle Scholar
  32. 32.
    Hinch, E.J.: An averaged-equation approach to particle interactions in a fluid suspension. J. Fluid Mech. 83, 695–720 (1977)CrossRefGoogle Scholar
  33. 33.
    Howells, I.D.: Drag due to the motion of a Newtonian fluid through a sparse random array of small fixed rigid objects. J. Fluid Mech. 64, 449–476 (1974)CrossRefGoogle Scholar
  34. 34.
    Keehm, Y.: Computational rock physics: Transport properties in porous media and applications. Ph.D. Dissertation, Stanford University, Stanford (2003)Google Scholar
  35. 35.
    Keehm, Y., Mukerji, T., Nur, A.: Permeability prediction from thin sections: 3D reconstruction and Lattice-Boltzmann flow simulation. Geophys. Res. Lett. 31, L04606 (2004)CrossRefGoogle Scholar
  36. 36.
    Kendon, V.M., Cates, M.E., Desplat, J. -C., Pagonabarraga, I., Bladon, P.: Inertial effects in three-dimensional spinodal decomposition of a symmetric binary fluid mixture: a lattice Boltzmann study. J. Fluid Mech. 440, 147–203 (2000)Google Scholar
  37. 37.
    Knackstedt, M.A., Sheppard, A.P., Sahimi, M.: Pore network modelling of two-phase flow in porous rock: the effect of correlated heterogeneity. Adv. Water Res. 24, 257–277 (2001)CrossRefGoogle Scholar
  38. 38.
    Knackstedt, M.A., Arns, C., Madadi, M., Sheppard, A.P., Latham, S., Sok, R., Bächle, G., Eberli, G.: Elastic and flow properties of carbonate core derived from 3D X ray-CT images. SEG Tech. Prog. Expanded Abstr. 27, 1804–1809 (2008)Google Scholar
  39. 39.
    Knackstedt, M.A., Latham, S., Madadi, M., Sheppard, A., Varslot, T., Arns, C.: Digital rock physics: 3D imaging of core material and correlations to acoustic and flow properties. Lead. Edge 28, 28–33 (2009)CrossRefGoogle Scholar
  40. 40.
    Kuzmin, A., Guo, Z., Mohamad, A.: Simultaneous incorporation of mass and force terms in the multi-relaxation-time framework for lattice Boltzmann schemes. Philos. Transac. Royal Soc. A Math. Phys. Eng. Sci. 369, 2219–2227 (2011)CrossRefGoogle Scholar
  41. 41.
    Ladd, A.J.C.: Numerical simulations of particulate suspensions via a discretized Boltzmann equation: part 1: theoretical foundation. J. Fluid Mech. 271, 285–309 (1994)CrossRefGoogle Scholar
  42. 42.
    Ladd, A.J.C.: Numerical simulations of particulate suspensions via a discretized Boltzmann equation: part 2. Numerical results. J. Fluid Mech. 271, 311–339 (1994)CrossRefGoogle Scholar
  43. 43.
    Lallemand, P., Luo, L.S.: Theory of the lattice Boltzmann method: dispersion, dissipation, isotropy, Galilean invariance, and stability. Phys. Rev. E 61, 6546–6562 (2000)CrossRefGoogle Scholar
  44. 44.
    Leu, L., Berg, S., Enzmann, F., Armstrong, R.T., Kersten, M.: Fast x-ray micro-tomography of multiphase flow in Berea Sandstone: a sensitivity study on image processing. Transp. Porous Media 105, 451–469 (2014)CrossRefGoogle Scholar
  45. 45.
    Llwellin, E.W.: LBflow: An extensible lattice Boltzmann framework for the simulation of geophysical flows. Part I: theory and implementation. Comput. Geosci. 36, 115–122 (2010)CrossRefGoogle Scholar
  46. 46.
    Llwellin, E.W.: LBflow: An extensible lattice Boltzmann framework for the simulation of geophysical flows. Part II: usage and validation. Comput. Geosci. 36, 123–132 (2010)CrossRefGoogle Scholar
  47. 47.
    Martys, N.S., Chen, H.: Simulation of multicomponent fluids in complex three dimensional geometries by the lattice Boltzmann method. Phys. Rev. E 53, 743–750 (1996)CrossRefGoogle Scholar
  48. 48.
    Mavko, G., Mukerji, T., Dvorkin, J.: The rock physics handbook: Tools for seismic analysis of porous media, 2nd edn. Cambridge University Press, Cambridge (2009)CrossRefGoogle Scholar
  49. 49.
    Meakin, P., Tartakovsky, A.M.: Modeling and simulation of pore-scale multiphase fluid flow and reactive transport in fractured and porous media. Rev. Geophys. RG3002, 1–47 (2009)Google Scholar
  50. 50.
    Otomo, H., Fan, H., Li, Y., Dressler, M., Staroselsky, I., Zhang, R., Chen, H.: Studies of accurate multi-component lattice Boltzmann models on benchmark cases required for engineering applications. J. Comput. Sci. 17, 334–339 (2016)CrossRefGoogle Scholar
  51. 51.
    Øren, P., Bakke, S., Rueslåtten, H.: Digital core laboratory: rock and flow properties derived from computer generated rocks. In: International Symposium of the Society of Core Analysts held in Trondheim, pp. 1–12, Norway (2006)Google Scholar
  52. 52.
    Palabos Documentation. Accessed: 4 June 2017 (2017)
  53. 53.
    Pan, C., Luo, L.-S., Miller, C.T.: An evaluation of lattice Boltzmann schemes for porous medium flow simulation. Comput. Fluids 35, 898–909 (2006)CrossRefGoogle Scholar
  54. 54.
    Piller, M., Casagrande, D., Schena, G., Santini, M.: Pore-scale simulation of laminar flow through porous media. J. Phys.: Conf. Ser. 501, 012010 (2014)Google Scholar
  55. 55.
    Premnath, K.N., Abraham, J.: Three-dimensional multi-relaxation-time (MRT) lattice-Boltzmann models for multiphase flow. J. Comput. Phys. 224, 539–559 (2007)CrossRefGoogle Scholar
  56. 56.
    Premnath, K.N., Pattison, M.J., Banerjee, S.: An investigation of the Lattice Boltzmann method for large eddy simulation of complex turbulent separated flow. J. Fluids Eng. 135, 051401–1-12 (2013)CrossRefGoogle Scholar
  57. 57.
    Rumpf, H.C.H., Gupte, A.R.: Einflüsse der Porosität und Korngröarticleßenverteilung im Widerstandsgesetz der Porenströmung. Chem. Ing. Tech. 43, 367–375 (1971)CrossRefGoogle Scholar
  58. 58.
    Saenger, E.H., Enzmann, F., Keehm, Y., Steeb, H.: Digital rock physics: effect of fluid viscosity on effective elastic properties. J. Appl. Geophys. 74, 236–241 (2011)CrossRefGoogle Scholar
  59. 59.
    Sain, R., Mukerji, T., Mavko, G.: How computational rock-physics tools can be used to simulate geologic processes, understand pore-scale heterogeneity, and refine theoretical models. Lead. Edge 33, 324–334 (2014)CrossRefGoogle Scholar
  60. 60.
    Saxena, N., Mavko, G.: Estimating elastic moduli of rocks from thin sections: digital rock study of 3D properties from 2D images. Comput. Geosci. 88, 9–21 (2016)CrossRefGoogle Scholar
  61. 61.
    Saxena, N., Hofmann, R., Alpak, F.O., Berg, S., Dietderich, J., Agarwal, U., Tandon, K., Hunter, S., Freeman, J., Wilson, O.B.: References and benchmarks for pore-scale flow simulated using micro-CT images of porous media and digital rocks. Adv. Water Res. 109, 211–235 (2017)CrossRefGoogle Scholar
  62. 62.
    Saxena, N., Hofmann, R., Alpak, F.O., Dietderich, J., Hunter, S., Day-Stirrat, R.: Effect of image segmentation andamp; voxel size on micro-CT computed effective transport andamp; elastic properties. Mar. Petrol. Geol. 86, 972–990 (2017)CrossRefGoogle Scholar
  63. 63.
    Saxena, N., Hofmann, R., Hows, A., Alpak, F.O., Freeman, J., Hunter, S., Appel, M.: Imaging and computational considerations for digital rock physics. Advances in Water Resources in review (2018)Google Scholar
  64. 64.
    Saxena, N., Mavko, G., Hofmann, R., Srisutthiyakorn, N.: Estimating permeability from thin sections without reconstruction: graphical study of 3D properties from 2D images. Comput. Geosci. 102, 79–99 (2017)CrossRefGoogle Scholar
  65. 65.
    Skordos, P.A.: Initial and boundary conditions for the lattice Boltzmann method. Phys. Rev. 48, 4823–4842 (1993)CrossRefGoogle Scholar
  66. 66.
    Shah, S.M., Gray, F., Crawshaw, J.P., Boek, E.S.: Micro-computed tomography pore-scale study of flow in porous media: effect of voxel resolution. Adv. Water Res. 95, 276–287 (2016)CrossRefGoogle Scholar
  67. 67.
    Sisavath, S., Jing, X., Pain, C.C., Zimmerman, R.W.: Creeping flow through an axisymmetric sudden contraction or expansion. J. Fluids Eng. 124, 273–278 (2002)CrossRefGoogle Scholar
  68. 68.
    Sisavath, S., Jing, X., Zimmerman, R.W.: Creeping flow through a pipe of varying radius. Phys. Fluids 13, 2762–2772 (2001)CrossRefGoogle Scholar
  69. 69.
    Sochi, T.: Newtonian flow in converging-diverging capillaries. Intern. J. Model. Simul. Scientific Comput. 4, 1350011 (2013)CrossRefGoogle Scholar
  70. 70.
    Succi, S.: The lattice boltzmann equation for fluid dynamics and beyond. numerical mathematics and scientific computation series. Clarendon Press, Oxford (2001)Google Scholar
  71. 71.
    Sukop, M.C., Thorne, D.T.: Lattice Boltzmann Modeling: An Introduction for Geoscientists and Engineers. Springer-Verlag, Berlin (2006)Google Scholar
  72. 72.
    Tölke, J.: Implementation of a lattice Boltzmann kernel using the compute unified device architecture developed by nVIDIA. Comput. Visual. Sci. 13(1), 29–39 (2010)CrossRefGoogle Scholar
  73. 73.
    Tölke, J., Baldwin, C., Mu, Y., Derzhi, N., Fang, Q., Grader, A., Dvorkin, J.: Computer simulations of fluid flow in sediment: from images to permeability. Lead. Edge 29, 68–74 (2010)CrossRefGoogle Scholar
  74. 74.
    Torquato, S: Random Heterogeneous Materials: Microstructure and Macroscopic Properties. Springer Science & Business Media, New York (2002)CrossRefGoogle Scholar
  75. 75.
    Wiegmann, A.: Computation of the permeability of porous materials from their microstructure by FFF-Stokes. Berichte des Fraunhofer ITWM, Nr. 129. Kaiserslautern, Germany (2007)Google Scholar
  76. 76.
    Zhang, D., Zhang, R., Chen, S., Soll, W.E.: Pore scale study of flow in porous media: scale dependency, REV, and statistical REV. Geophys. Res. Lett. 27, 1195–1198 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • F. O. Alpak
    • 1
    • 2
    Email author
  • F. Gray
    • 3
  • N. Saxena
    • 1
  • J. Dietderich
    • 1
  • R. Hofmann
    • 1
  • S. Berg
    • 4
    • 5
  1. 1.Shell International Exploration and Production Inc.HoustonUSA
  2. 2.Department of Computational & Applied MathematicsRice UniversityHoustonUSA
  3. 3.Qatar Carbonates and Carbon Storage Research Centre, Department of Chemical EngineeringImperial College LondonLondonEngland
  4. 4.Shell Global Solutions International B.V.AmsterdamThe Netherlands
  5. 5.Department of Earth Science & Engineering, Department of Chemical EngineeringImperial College LondonLondonEngland

Personalised recommendations