Skip to main content
SpringerLink
Log in

Large-Scale Optimization-Based Non-negative Computational Framework for Diffusion Equations: Parallel Implementation and Performance Studies

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

It is well-known that the standard Galerkin formulation, which is often the formulation of choice under the finite element method for solving self-adjoint diffusion equations, does not meet maximum principles and the non-negative constraint for anisotropic diffusion equations. Recently, optimization-based methodologies that satisfy maximum principles and the non-negative constraint for steady-state and transient diffusion-type equations have been proposed. To date, these methodologies have been tested only on small-scale academic problems. The purpose of this paper is to systematically study the performance of the non-negative methodology in the context of high performance computing (HPC). PETSc and TAO libraries are, respectively, used for the parallel environment and optimization solvers. For large-scale problems, it is important for computational scientists to understand the computational performance of current algorithms available in these scientific libraries. The numerical experiments are conducted on the state-of-the-art HPC systems, and a single-core performance model is used to better characterize the efficiency of the solvers. Our studies indicate that the proposed non-negative computational framework for diffusion-type equations exhibits excellent strong scaling for real-world large-scale problems.

Graphical Abstract

This figure shows the fate of chromium after 180 days using the single-field Galerkin formulation. The white regions indicate the violation of the non-negative constraint.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Adams, R.J., Fournier, J.J.F.: Sobolev Spaces, 2nd edn. Academic press, Amsterdam (2003)

    MATH  Google Scholar 

  2. Ayachit, U.: The ParaView Guide: A Parallel Visualization Application. Kitware, Clifton Park (2015)

    Google Scholar 

  3. Balay, S., Abhyankar, S., Adams, M.F., Brown, J., Brune, P., Buschelman, K., Eijkhout, V., Gropp, W.D., Kaushik, D., Knepley, M.G., McInnes, L.C., Rupp, K., Smith, B.F., Zampini, S., Zhang, H.: PETSc users manual. Technical Report ANL-95/11 - Revision 3.6, Argonne National Laboratory, 2015

  4. Benson, S.J., Munson, T.S.: Flexible complementary solvers for large-scale applications. Optim. Methods Softw. 21, 155–168 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  5. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge, UK (2004)

    Book  MATH  Google Scholar 

  6. Childs, H., Brugger, E., Whitlock, B., Meredith, J., Ahern, S., Pugmire, D., Biagas, K., Miller, M., Harrison, C., Weber, G.H., Krishnan, H., Fogal, T., Sanderson, A., Garth, C., Bethel, E.W., Camp, D., Rübel, O., Durant, M., Favre, J.M., Navrátil, P.: VisIt: An end-user tool for visualizing and analyzing very large data. In: High Performance Visualization-Enabling Extreme-Scale Scientific Insight, (pp. 357–372) (2012). https://wci.llnl.gov/simulation/computer-codes/visit

  7. Ciarlet, P.G., Raviart, P.-A.: Maximum principle and uniform convergence for the finite element method. Comput. Methods Appl. Methods Eng. 2, 17–31 (1973)

    Article  MathSciNet  MATH  Google Scholar 

  8. Evans, L.C.: Partial Differential Equations. American Mathematical Society, Providence (1998)

    MATH  Google Scholar 

  9. Facchinei, F., Pang, J.-S.: Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer, New York (2003)

    MATH  Google Scholar 

  10. Graser, C., Kornhuber, R.: Multigrid methods for obstacle problems. J. Comput. Math. 27, 1–44 (2009)

    MathSciNet  MATH  Google Scholar 

  11. Gropp, W.D., Kaushik, D.K., Keyes, D.E., Smith, B.F.: Toward realistic performance bounds for implicit CFD codes. In Proceedings of Parallel CFD ‘99, pp. 233–240. Elsevier, (1999)

  12. Hammond, G.E., Lichtner, P.C.: Field-scale model for the natural attenuation of uranium at the Hanford 300 Area using high-performance computing. Water Resour. Res. 46, W09602 (2010)

    Article  Google Scholar 

  13. Harp, D.R., Vesselinov, V.V.: Contaminant remediation decision analysis using information gap theory. Stoch. Environ. Res. Risk Assess. 27(1), 159–168 (2013)

    Article  Google Scholar 

  14. Heikoop, J.M., Johnson, T.M., Birdsell, K.H., Longmire, P., Hickmott, D.D., Jacobs, E.P., Broxton, D.E., Katzman, D., Vesselinov, V.V., Ding, M., Vanimana, D.T., Reneaua, S.L., Goering, T.J., Glessnerb, J., Basu, A.: Isotopic evidence for reduction of anthropogenic hexavalent chromium in Los Alamos National Laboratory groundwater. Chem. Geol. 373, 1–9 (2014)

    Article  Google Scholar 

  15. Hjelmstad, K.D.: Fundamentals of Structural Mechanics, 2nd edn. Springer, New York (2005)

    Google Scholar 

  16. Karra, S., Painter, S.L., Lichtner, P.C.: Three-phase numerical model for subsurface hydrology in permafrost-affected regions (PFLOTRAN-ICE v1.0). Cryosphere 8(5), 1935–1950 (2014)

    Article  Google Scholar 

  17. Karypis, G., Kumar, V.: A fast and highly quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput. 20, 359–392 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  18. Kelkar, S., Lewis, K., Karra, S., Zyvoloski, G., Rapaka, S., Viswanathan, H., Mishra, P.K., Chu, S., Coblentz, D., Pawar, R.: A simulator for modeling coupled thermo-hydro-mechanical processes in subsurface geological media. Int. J. Rock Mech. Min. Sci. 70, 569–580 (2014)

    Google Scholar 

  19. Kinderlehrer, D., Stampacchia, G.: An Introduction to Variational Inequalities and Their Applications. SIAM Classics in Applied Mathematics, New York (2000)

    Book  MATH  Google Scholar 

  20. Kirby, R.C.: FIAT: Numerical Construction of Finite Element Basis Functions, Chapter 13. Springer, New York (2012)

    Google Scholar 

  21. Knepley, M.G., Brown, J., Rupp, K., Smith, B.F.: Achieving High Performance with Unified Residual Evaluation. arXiv:1309.1204, September (2013)

  22. Knepley, M.G., Karpeev, D.A.: Mesh algorithms for PDE with vieve I: Mesh distribution. Sci. Program. 17, 215–230 (2009)

    Google Scholar 

  23. Lange, M., Knepley, M.G., Gorman., G.J.: Flexible, scalable mesh and data management using PETSc DMPlex. In Proceedings of the 3rd International Conference on Exascale Applications and Software, EASC ‘15, pp. 71–76. University of Edinburgh, (2015)

  24. Lichtner, P.C., Hammond, G.E., Lu, C., Karra, S., Bisht, G., Andre, B., Mills, R.T., Kumar, J.: PFLOTRAN user manual: A massively parallel reactive flow and transport model for describing surface and subsurface processes. Technical Report Report No.: LA-UR-15-20403, Los Alamos National Laboratory, (2015)

  25. Lichtner, P.C., Karra, S.: Modeling multiscale-multiphase-multicomponent reactive flows in porous media: Application to CO\(_2\) sequestration and enhanced geothermal energy using pflotran. In R. Al-Khoury and J. Bundschuh, editors, Computational Models for CO\(_2\) Geo-sequestration & Compressed Air Energy Storage, pp. 81–136. CRC Press, http://www.crcnetbase.com/doi/pdfplus/10.1201/b16790-6, (2014)

  26. Liska, R., Shashkov, M.: Enforcing the discrete maximum principle for linear finite element solutions of second-order elliptic problems. Commun. Comput. Phys. 3(4), 852–877 (2008)

    MathSciNet  MATH  Google Scholar 

  27. Lo, Y.J., Williams, S., Straalen, B.V., Ligocki, T.J., Cordery, M.J., Wright, N.J., Hall, M.W., Oliker, L.: Roofline: an insightful visual performance model for multicore architectures. High Perform. Comput. Syst. Perfom. Model. Benchmarking Simul. 8966, 129–148 (2015)

    Google Scholar 

  28. Logg, A.: Efficient representation of computational Meshes. Int. J. Comput. Sci. Eng. 4, 283–295 (2009)

    Article  Google Scholar 

  29. Logg, A., Mardal, K.A., Wells, G.N.: Automated Solution of Differential Equations by the Finite Element Method. Springer, New York (2012)

    Book  MATH  Google Scholar 

  30. May, D.A., Brown, J., Laetitia, L.L.: pTatin3D: High-performance methods for long-term lithospheric dynamics. In:Proceedings of the International Conference for High Performance Computing, Network, Storage and Analysis, SC ‘14, pp. 274–284. IEEE Press, (2014)

  31. McCalpin, J.: STREAM: sustainable memory bandwidth in high performance computers, (1995). https://www.cs.virginia.edu/stream/

  32. Mudunuru, M.K., Nakshatrala, K.B.: On enforcing maximum principles and achieving element-wise species balance for advection-diffusion-reaction equations under the finite element method. J. Comput. Phys. 305, 448–493 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  33. Mudunuru, M.K., Nakshatrala, K.B.: On mesh restrictions to satisfy comparison principles, maximum principles, and the non-negative constraint: Recent developments and new results. Mechanics of Advanced Materials and Structures (2016). doi:10.1080/15502287.2016.1166160

    MATH  Google Scholar 

  34. Munson, T., Sarich, J., Wild, S., Benson, S., McInnes, L.C.: TAO 2.0 users manual. Technical Report ANL/MCS-TM-322, Mathematics and Computer Science Division, Argonne National Laboratory. http://www.mcs.anl.gov/tao (2012)

  35. Nagarajan, H., Nakshatrala, K.B.: Enforcing the non-negativity constraint and maximum principles for diffusion with decay on general computational grids. Int. J. Numer. Methods Fluids 67, 820–847 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  36. Nakshatrala, K.B., Mudunuru, M.K., Valocchi, A.J.: A numerical framework for diffusion-controlled bimolecular-reactive systems to enforce maximum principles and the non-negative constraint. J. Comput. Phys. 253, 278–307 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  37. Nakshatrala, K.B., Nagarajan, H., Shabouei, M.: A numerical methodology for enforcing maximum principles and the non-negative constraint for transient diffusion equations. Commun. Comput. Phys. 19, 53–93 (2016)

    Article  MathSciNet  Google Scholar 

  38. Nakshatrala, K.B., Valocchi, A.J.: Non-negative mixed finite element formulations for a tensorial diffusion equation. J. Comput. Phys. 228, 6726–6752 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  39. Payette, G.S., Nakshatrala, K.B., Reddy, J.N.: On the performance of high-order finite elements with respect to maximum principles and the nonnegative constraint for diffusion-type equations. Int. J. Numer. Methods Eng. 91, 742–771 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  40. Pruess, K.: The TOUGH codes–a family of simulation tools for multiphase flow and transport processes in permeable media. Vadose Zone J. 3(3), 738–746 (2004)

    Google Scholar 

  41. The HDF Group. Hierarchical Data Format, version 5. http://www.hdfgroup.org/HDF5/ (1997–2015)

  42. Ulbrich, M.: Semismooth Newton Methods for Variational Inequalities and Constrained Optimization Problems in Function Spaces. SIAM, Philadelphia (2011)

  43. US EPA. Cleaning up the nation’s waste sites: Markets and technology trends. Technical Report EPA 542-R-04-015, (2004)

  44. Williams, S., Waterman, A., Patterson, D.: Roofline: an insightful visual performance model for multicore architectures. Commun. ACM 52, 65–76 (2009)

    Article  Google Scholar 

  45. Zyvoloski, G.: FEHM: A control volume finite element code for simulating subsurface multi-phase multi-fluid heat and mass transfer. Los Alamos Unclassified Report LA-UR-07-3359, (2007)

Download references

Acknowledgments

The authors thank Matthew G. Knepley (Rice University) for his invaluable advice. The authors also thank the Los Alamos National Laboratory (LANL) Institutional Computing program. JC and KBN acknowledge the financial support from the Houston Endowment Fund and from the Department of Energy through Subsurface Biogeochemical Research Program. SK thanks the LANL LDRD program and the LANL Environmental Programs Directorate for their support. The opinions expressed in this paper are those of the authors and do not necessarily reflect that of the sponsors.

Author information

Authors and Affiliations

  1. Department of Civil and Environmental Engineering, University of Houston, Texas, TX, 77204-4003, USA

    J. Chang & K. B. Nakshatrala

  2. Los Alamos National Laboratory, Los Alamos, NM, USA

    S. Karra

Authors
  1. J. Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Karra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. K. B. Nakshatrala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K. B. Nakshatrala.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chang, J., Karra, S. & Nakshatrala, K.B. Large-Scale Optimization-Based Non-negative Computational Framework for Diffusion Equations: Parallel Implementation and Performance Studies. J Sci Comput 70, 243–271 (2017). https://doi.org/10.1007/s10915-016-0250-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-016-0250-5

Keywords

Search

Navigation