Skip to main content
SpringerLink
Log in

The Foundations of Verification in Modeling and Simulation

  • Chapter
  • First Online:
Book cover Computer Simulation Validation

Part of the book series: Simulation Foundations, Methods and Applications ((SFMA))

Abstract

The practice of verification is grounded in mathematics highlighting the fundamental nature of its practice. Models of reality are fundamentally mathematical and verification assures the connection between the modeling intended and achieved in code. Code verification is a process where the correctness of a computer code for simulation and modeling is proven. This “proof” is defined by the collection of evidence that the numerical approximations are congruent with the model for the physical phenomena. The key metric in code verification is the order of accuracy of the approximation that should match theoretical expectations. In contrast, solution verification is an aspect of uncertainty estimation associated with numerical error in simulations. Solution verification uses many of the same approaches as code verification, but its principal outcome is an estimate of the numerical error. The order of convergence is a secondary outcome. Together these two practices form an important part of the foundation of quality and credibility in modeling and simulation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    There are some simple cases where numerical solutions are exact. These cases are often simple or highly contrived such as the finite element “patch” test.

  2. 2.

    Documentary, dir. Russel Barnes; writer Richard Dawkins; 2007, here quoted from https://www.goodreads.com.

  3. 3.

    Quoted from http://www-history.mcs.st-and.ac.uk/Quotations/Bott.html.

  4. 4.

    Here quoted after https://www.goodreads.com/author/quotes/14129212.Karl_Pippart_III.

References

  • Ascher, U. M., & Petzold, L. R. (1998). Computer methods for ordinary differential equations and differential-algebraic equations, vol. 61. Siam.

    Google Scholar 

  • Banks, J. W., Aslam, T., & Rider, W. (2008). On sub-linear convergence for linearly degenerate waves in capturing schemes. Journal of Computational Physics, 227(14), 6985–7002.

    Article  MathSciNet  Google Scholar 

  • Bjork, A. (1996). Numerical Methods for Least Squares Problems. SIAM.

    Google Scholar 

  • Brown, L. M., & Feynman, R. P. (2000). Selected papers of Richard Feynman: With commentary (Vol. 27), World scientific series in 20th century physics Singapore: World Scientific.

    Google Scholar 

  • Drikakis, D., & Rider, W. (2006). High-resolution methods for incompressible and low-speed flows. Springer Science & Business Media.

    Google Scholar 

  • Eça, L., Hoekstra, M., Hay, A., & Pelletier, D. (2007). On the construction of manufactured solutions for one and two-equation eddy-viscosity models. International Journal for Numerical Methods in Fluids, 54(2), 119–154.

    Article  Google Scholar 

  • Galli, G., & Pasquarello, A. (1993). First-principles molecular dynamics. In Computer simulation in chemical physics (Springer, pp. 261–313).

    Google Scholar 

  • Gartling, D. K. (1990). A test problem for outflow boundary conditions-flow over a backward-facing step. International Journal for Numerical Methods in Fluids, 11(7), 953–967.

    Article  Google Scholar 

  • Gottlieb, J., & Groth, C. P. (1988). Assessment of Riemann solvers for unsteady one-dimensional inviscid flows of perfect gases. Journal of Computational Physics, 78(2), 437–458.

    Article  Google Scholar 

  • Hairer, E., Nørsett, S. P., & Wanner, G. (1993). Solving ordinary differential equations. I, volume 8 of Springer Series in Computational Mathematics. Springer-Verlag, Berlin.

    Google Scholar 

  • Huber, P. (1996). Robust Statistical Procedures. SIAM.

    Google Scholar 

  • Kamm, J. R., Brock, J. S., Brandon, S. T., Cotrell, D. L., Johnson, B., Knupp, P., et al. (2008). Enhanced verification test suite for physics simulation codes. Los Alamos National Laboratory (LANL), Los Alamos, NM: Tech. rep.

    Google Scholar 

  • Kim, J., & Moin, P. (1985). Application of a fractional-step method to incompressible navier-stokes equations. Journal of Computational Physics, 59(2), 308–323.

    Article  MathSciNet  Google Scholar 

  • Kotschenreuther, M., Dorland, W., Beer, M., & Hammett, G. (1995). Quantitative predictions of tokamak energy confinement from first-principles simulations with kinetic effects. Physics of Plasmas, 2(6), 2381–2389.

    Article  Google Scholar 

  • Lamb, H. (1932). Hydrodynamics. Cambridge: Cambridge University Press.

    MATH  Google Scholar 

  • Lavery, D., & Burkhead, C. (Eds.). (2011). Joss Whedon: Conversations. Jackson: University Press of Mississippi.

    Google Scholar 

  • Lax, P. D., & Richtmyer, R. D. (1956). Survey of the stability of linear finite difference equations. Communications on Pure and Applied Mathematics, 9(2), 267–293.

    Article  MathSciNet  Google Scholar 

  • Le, H., Moin, P., & Kim, J. (1997). Direct numerical simulation of turbulent flow over a backward-facing step. Journal of Fluid Mechanics, 330, 349–374.

    Article  Google Scholar 

  • Lin, P. T., Sala, M., Shadid, J. N., & Tuminaro, R. S. (2006). Performance of fully-coupled algebraic multilevel domain decomposition preconditioners for incompressible flow and transport. International Journal for Numerical Methods in Engineering, 67, 208–225.

    Article  Google Scholar 

  • Moin, P., & Mahesh, K. (1998). Direct numerical simulation: a tool in turbulence research. Annual Review of Fluid Mechanics, 30(1), 539–578.

    Article  MathSciNet  Google Scholar 

  • Moser, R. D., Kim, J., & Mansour, N. N. (1999). Direct numerical simulation of turbulent channel flow up to re \(\tau \)= 590. Physics of Fluids, 11(4), 943–945.

    Article  Google Scholar 

  • Oberkampf, W. L., & Roy, C. J. (2010). Verification and validation in scientific computing. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Pawlowski, R. P., Shadid, J. N., Simonis, J. P., & Walker, H. F. (2006). Globalization techniques for Newton-Krylov methods and applications to the fully coupled solution of the Navier-Stokes equations. SIAM Review, 48(4), 700–721.

    Article  MathSciNet  Google Scholar 

  • Pippart, K. (2015). Operation Mexico!: Carl Kiekhaefer vs the 1951–1953 Pan American Road Race. Minneapolis: Mill City Press.

    Google Scholar 

  • Prabhakar, V., & Reddy, J. (2006). Spectral/hp penalty least-squares finite element formulation for the steady incompressible Navier-Stokes equations. Journal of Computational Physics, 215(1), 274–297.

    Article  MathSciNet  Google Scholar 

  • Rider, W., Witkowski, W., Kamm, J. R., & Wildey, T. (2016). Robust verification analysis. Journal of Computational Physics, 307, 146–163.

    Article  MathSciNet  Google Scholar 

  • Rider, W. J., Kamm, J. R., & Weirs, V. G. (2016). Procedures for calculation verification. Simulation Credibility, 31.

    Google Scholar 

  • Rider, W. J., Kamm, J. R., Weirs, V. G., & Cacui, D. G. (2010). Verification, validation and uncertainty quantification workflow in CASL. Albuquerque, NM: Sandia National Laboratories.

    Google Scholar 

  • Roache, P. (1998). Verification and validation in computational science and engineering. Hermosa Publishers.

    Google Scholar 

  • Roache, P. (2009). Fundamentals of verification and validation. Hermosa Publishers.

    Google Scholar 

  • Sagaut, P. Large eddy simulation for incompressible flows: An introduction. Springer Science & Business Media, 2006.

    Google Scholar 

  • Stern, F., Wilson, R. V., Coleman, H. W., & Paterson, E. G. (2001). Comprehensive approach to verification and validation of CFD simulations-part 1: Methodology and procedures. Journal Fluids Engrng., 123, 793802.

    Google Scholar 

  • Taylor, G. I., & Green, A. E. (1937). Mechanism of the production of small eddies from large ones. Proceedings of the Royal Society of London, 158(895), 499–521.

    MATH  Google Scholar 

  • Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society. Series B (Methodological), 267–288.

    MathSciNet  MATH  Google Scholar 

  • Twain, M. (2015). Autobiography of Mark Twain. Vol. 3. The Mark Twain papers. University of California Press, Berkeley. A publication of the Mark Twain Project of the Bancroft Library.

    Google Scholar 

  • White, F. (1991). Viscous fluid flow. New York: McGraw-Hill Inc.

    Google Scholar 

Download references

Acknowledgements

The author would like to thank Tim Trucano, Vince Mousseau, Patrick Knupp, Bill Oberkampf, Chris Roy, Greg Weirs, and Patrick Roache for numerous discussions and informative suggestions. The DOE ASC V&V program has provided support for the development of the verification methodology. Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, National Technology and Engineering Solutions of Sandia, LLC. Operated for the United States Department of Energy by National Technology and Engineering Solutions of Sandia, LLCDE-AC04-94AL85000.

Author information

Authors and Affiliations

  1. Sandia National Laboratories, Center for Computing Research, Albuquerque, NM, 87185, USA

    William J. Rider

Authors
  1. William J. Rider
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William J. Rider .

Editor information

Editors and Affiliations

  1. University of Bern, Bern, Switzerland

    Claus Beisbart

  2. Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Nicole J. Saam

Appendices

Appendix

Details of Code Verification Documentation

Below, I briefly describe the characteristics of each type of benchmark documentation (could be called artifacts or meta-data) associated with a code verification benchmarks. These artifacts take a number of concrete forms such as a written document, computer code, mathematical solution in document or software form, input files for executable codes, input to automatic computer analysis, output from software quality systems, among others. While all of these will not typically exist, a subset of these documents should be available to support any questions or concerns about the verification problem or its implementation.

  • Detailed technical description of the benchmark (report or paper): This can include a technical paper in a journal or conference proceeding describing the benchmark and its solution. Another form would be a report informal or formal from an institution providing the same information.

  • Analysis of the mathematics (report or paper): For any solution that is closed form, or requiring a semi-analytical solution, the mathematics must be described in detail. This can be included in the paper (report) discussed previously or in a separate document.

  • Computer analysis of solution (input file): If the mathematics or solution is accomplished using a computerized analysis, the program used and the input to the program should be included. Some sort of written documentation such as a manual for the software ideally accompanies this artifact.

  • Computer solution of the mathematical solution: The actual computerized solution of the mathematical problem should be included in whatever form the computerized solution takes. This should include any error analysis completed with this solution.

  • Computer implementation of the numerical solution: The analytical solution should be implemented in a computational form to allow the comparison with the numerical solution. This should include some sort of error analysis in the form of a report.

  • Derivation of the source term and software implementation or input: In the case of the method of manufactured solutions, the source term used to drive the numerical method must be derived through a well-defined numerical procedure. This should be documented through a document, and numerical tools used for the derivation and implementation.

  • Computer implementation of the source term (manufactured solution): The source term should be included in a form amenable to direct use in a computer code. The language for the computer code should be clearly defined as well as the compiler and computer system used.

  • Grids for numerical solution: If a solution is computed using another simulation code all relevant details on the numerical grid(s) used must be included. This could be direct grid files, or input files to well-defined grid generation software.

  • Convergence and error estimation of numerical solution: The numerical solution must include a convergence study and error estimate. These should be detailed in an appropriately peer-reviewed document.

  • Uncertainty and sensitivity study of numerical solution: The various modeling options in the code used to provide the numerical solution must be examined vis-a-vis the uncertainty and sensitivity of the solution to these choices. This study should be used to justify the methodology used for the baseline solution.

  • Description and analysis of computational methods: The methods used by the code used for the baseline solution must be completely described and analyzed. This can take the form of a complete bibliography of readily available literature

  • Numerical analysis theory associated with convergence: The nature of the convergence and the magnitude of error in the numerical solution must be described and demonstrated. This can take the form of a complete bibliography of readily available literature.

  • Code description/manuals: The code manual and complete description must be included with the analysis and description.

  • Input files for benchmarks and auxiliary software: The input file used to produce the solution must be included. Any auxiliary software used to produce or analyze the solution must be full described or included.

  • Unusual boundary conditions (inflow, piston, outflow, Robin, symmetry,): Should the benchmark require unusual or involved boundary or initial conditions, these must be described in additional detail including the nature of implementation.

  • Physics restrictions (boundary layer theory, inviscid, parabolized Navier–Stokes,): If the solution requires the solution of a reduced or restricted set of equations, this must be fully described. Examples are boundary layer theory, truly inviscid flow, or various asymptotic limits.

  • Software quality documents: Of non-commercial software used to produce solutions, the software quality pedigree should be clearly established by documenting the software quality and steps taken to assure the maintenance of the quality.

  • Scripts and auxiliary software for verification: Auxiliary software or scripts used to determine the verification or compute error estimates for a software used to produce solution should be included.

  • Source code: If possible the actual source code for the software along with instructions for producing an executable (makefile, scripts) should be included with all other documentation.

  • A full mathematical or computational description of metrics used in error analysis and evaluation of solution implementation or numerical solution.

  • Verification results including code version, date, and other identifying characteristics: The verification basis for the code used to produce the baseline solution must be included. This includes any documentation of verification, peer-review, code version, date completed and error estimates.

  • Feature coverage in verification: The code features covered by verification benchmarks must be documented. Any gaps where the feature used for the baseline solution are not verified must be explicitly documented.

Here are the necessary data requirements for each category of benchmark, again arranged in order of increasing level of documentation required. For completeness each data type would expected to be available to describe a benchmark of a given type.

  • Common elements for all types of benchmarks (it is notable that the use of proper verification using an analytical solution results in the most compact set of requirements for data, manufactured solutions also).

    1. 1.

      Paper or report

    2. 2.

      Mathematical analysis

    3. 3.

      Computerized solution and input

    4. 4.

      Error and uncertainty analysis

    5. 5.

      Computer implementation of the evaluation of the solution

    6. 6.

      Restrictions

    7. 7.

      Boundary or initial conditions

  • Closed form analytical solution

    1. 1.

      Paper or report

    2. 2.

      Mathematical analysis

    3. 3.

      Computerized solution and input

    4. 4.

      Error and uncertainty analysis

    5. 5.

      Computer implementation of the evaluation of the solution

    6. 6.

      Restrictions

    7. 7.

      Boundary or initial conditions

  • Manufactured Solution

    1. 1.

      Paper or report

    2. 2.

      Mathematical analysis

    3. 3.

      Computational solution and input

    4. 4.

      Error and uncertainty analysis

    5. 5.

      Computer implementation of the evaluation of the solution

    6. 6.

      Derivation and implementation of the source term

    7. 7.

      Restrictions

    8. 8.

      Boundary or initial conditions

  • Numerical solution with analytical solution

  • Series solution, Nonlinear algebraic solution, Nonlinear ODE solution

    1. 1.

      Paper or report

    2. 2.

      Mathematical analysis

    3. 3.

      Computerized solution and input

    4. 4.

      Error and uncertainty analysis

    5. 5.

      Computer implementation of the evaluation of the solution

    6. 6.

      Input files

    7. 7.

      Source code

    8. 8.

      Source code SQA

    9. 9.

      Method description and manual

    10. 10.

      Restrictions

    11. 11.

      Boundary or initial conditions

  • Highly accurate numerical solution (not analytical), numerical benchmarks or code-to-code comparisons.

    1. 1.

      Paper or report

    2. 2.

      Mathematical analysis

    3. 3.

      Computational solution and input

    4. 4.

      Error and uncertainty analysis for the solution

    5. 5.

      Computer implementation of the evaluation of the solution

    6. 6.

      Input files

    7. 7.

      Grids

    8. 8.

      Source code

    9. 9.

      Source code SQA

    10. 10.

      Method description and manual

    11. 11.

      Method analysis

    12. 12.

      Method verification analysis and coverage

    13. 13.

      Restrictions

    14. 14.

      Boundary or initial conditions.

Rights and permissions

Reprints and permissions

Copyright information

© 2019 This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Rider, W.J. (2019). The Foundations of Verification in Modeling and Simulation. In: Beisbart, C., Saam, N. (eds) Computer Simulation Validation. Simulation Foundations, Methods and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-70766-2_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-70766-2_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-70765-5

  • Online ISBN: 978-3-319-70766-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation