Skip to main content
SpringerLink
Log in

Testing Simulation Models Using Frequentist Statistics

  • Chapter
  • First Online:
Computer Simulation Validation

Part of the book series: Simulation Foundations, Methods and Applications ((SFMA))

Abstract

One approach to validating simulation models is to formally compare model outputs with independent data. We consider such model validation from the point of view of Frequentist statistics. A range of estimates and tests of goodness of fit have been advanced. We review these approaches, and demonstrate that some of the tests suffer from difficulties in interpretation because they rely on the null hypothesis that the model is similar to the observations. This reliance creates two unpleasant possibilities, namely, a model could be spuriously validated when data are too few, or inappropriately rejected when data are too many. Finally, these tests do not allow a principled declaration of what a reasonable level of difference would be considering the purposes to which the model will be put. We consider equivalence tests, and demonstrate that they do not suffer from the previously identified shortcomings. We provide two case studies to illustrate the claims of the chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Aigner, D. J. (1972). A note on verification of computer simulation models. Management Science, 18(11), 615–619.

    Article  Google Scholar 

  • Alewell, C., & Manderscheid, B. (1998). Use of objective criteria for the assessment of biogeochemical ecosystem models. Ecological Modelling, 107, 213–224.

    Article  Google Scholar 

  • Bartelink, H. H. (1998). Radiation interception by forest trees: A simulation study on effects of stand density and foliage clustering on absorption and transmission. Ecological Modelling, 105, 213–225.

    Article  Google Scholar 

  • Berger, R. L., & Hsu, J. C. (1996). Bioequivalence trials, intersection-union tests and equivalence confidence sets. Statistical Science, 11(4), 283–319.

    Article  MathSciNet  Google Scholar 

  • Capes, H., et al. (2017). The allometric quarter-power scaling model and its applicability to Grand fir and Eucalyptus trees. Journal of Agricultural, Biological, and Environmental Statistics, $7$, 1–23.

    Google Scholar 

  • Casella, G., & Berger, R. L. (1990). Statistical inference. Belmont, CA.: Duxbury Press.

    MATH  Google Scholar 

  • Caswell, H. (1976). The validation problem. In B. Patten (Ed.), Systems analysis and simulation in ecology (Vol. 4, pp. 313–325). Cambridge: Academic Press.

    Chapter  Google Scholar 

  • Cohen, K. J., & Cyert, R. M. (1961). Computer models in dynamic economics. The Quarterly Journal of Economics, 75(1), 112–127.

    Article  Google Scholar 

  • Duursma, R., Marshall, J., Robinson, A., & Pangle, R. (2007). Description and test of a simple process-based model of forest growth for mixed-species stands. Ecological Modelling, 203(3–4), 297–311.

    Article  Google Scholar 

  • Freese, F. (1960). Testing accuracy. Forest Science, 6(2), 139–145.

    Google Scholar 

  • Gentil, S., & Blake, G. (1981). Validation of complex ecosystem models. Ecological Modelling, 14, 21–38.

    Article  Google Scholar 

  • Gregoire, T. G., & Reynolds, M. R, Jr. (1988). Accuracy testing and estimation alternatives. Forest Science, 34(2), 302–320.

    Google Scholar 

  • Jans-Hammermeister, D. C., & McGill, W. B. (1997). Evaluation of three simulation models used to describe plant residue decomposition in soil. Ecological Modelling, 104, 1–13.

    Article  Google Scholar 

  • Kleijnen, J. P. C. (1995). Verification and validation of simulation models. European Journal of Operational Research, 82, 145–162.

    Article  MathSciNet  Google Scholar 

  • Kleijnen, J. P. C., Bettonvil, B., & Van Groenendaal, W. (1998). Validation of trace-driven simulation models: A novel regression test. Management Science, 44(6), 812–819.

    Google Scholar 

  • Kleijnen, J. P. C. (1974). Statistical techniques in simulation (part 1). New York.: Marcel Dekker.

    MATH  Google Scholar 

  • Landsberg, J. J., Waring, R. H., & Coops, N. C. (2003). Performance of the forest productivity model 3-PG applied to a wide range of forest types. Forest Ecology and Management, 172, 199–214.

    Article  Google Scholar 

  • Loehle, C. (1997). A hypothesis testing framework for evaluating ecosystem model performance. Ecological Modelling, 97, 153–165.

    Article  Google Scholar 

  • Mayer, D. G., & Butler, D. G. (1993). Statistical validation. Ecological Modelling, 68, 21–32.

    Article  Google Scholar 

  • McBride, G. B. (1999). Equivalence tests can enhance environmental science and management. Australian and New Zealand Journal of Statistics, 41(1), 19–29.

    Article  MathSciNet  Google Scholar 

  • Meyners, M. (2012). Equivalence tests—A review. Food Quality and Preference, 26(2), 231–245.

    Article  Google Scholar 

  • Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994). Verification, validation, and confirmation of numerical models in the earth sciences. Science, 263, 641–646.

    Article  Google Scholar 

  • Ottosson, F., & HÃ¥kanson, L. (1997). Presentation and analysis of a model simulating the pH response of lake liming. Ecological Modelling, 105, 89–111.

    Article  Google Scholar 

  • Parkhurst, D. F. (2001). Statistical significance tests: equivalence and reverse tests should reduce misinterpretation. Bioscience, 51(12), 1051–1057.

    Article  Google Scholar 

  • Pocewicz, A. L., Gessler, P., & Robinson, A. P. (2004). The relationship between effective plant area index and landsat spectral response across elevation, solar insolation, and spatial scales in a northern Idaho forest. Canadian Journal of Forest Research, 34(2), 465–480.

    Article  Google Scholar 

  • R Core Team (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

    Google Scholar 

  • Reynolds, M. R, Jr. (1984). Estimating the error in model predictions. Forest Science, 30(2), 454–469.

    Google Scholar 

  • Reynolds, M. R, Jr., Burkhart, H. E., & Daniels, R. F. (1981). Procedures for statistical validation of stochastic simulation models. Forest Science, 27(2), 349–364.

    Google Scholar 

  • Robinson, A. (2016). Equivalence: Provides tests and graphics for assessing tests of equivalence. R package version 0.7.2.

    Google Scholar 

  • Robinson, A., Duursma, R., & Marshall, J. (2005). A regression-based equivalence test for model validation: Shifting the burden of proof. Tree Physiology, 25(7), 903.

    Article  Google Scholar 

  • Robinson, A. P., & Ek, A. R. (2000). The consequences of hierarchy for modelling in forest ecosystems. Canadian Journal of Forest Research, 30(12), 1837–1846.

    Article  Google Scholar 

  • Robinson, A. P., & Froese, R. E. (2004). Model validation using equivalence tests. Ecological Modelling, 176(3–4), 349–358.

    Article  Google Scholar 

  • Rykiel, E. J. (1996). Testing ecological models—The meaning of validation. Ecological Modelling, 90(3), 229–244.

    Article  Google Scholar 

  • Sargent, R. G. (2012). Verification and validation of simulation models. Journal of Simulation, 7(1), 12–24.

    Article  MathSciNet  Google Scholar 

  • Vanclay, J. K., & Skovsgaard, J. P. (1997). Evaluating forest growth models. Ecological Modelling, 98(1), 1–12.

    Article  Google Scholar 

  • Wellek, S. (2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). Chapman and Hall/CRC.

    Google Scholar 

  • Wykoff, W., Crookston, N., & Stage, A. (1982). User’s guide to the stand prognosis model. USDA Forest Service Intermountain Research Station, Ogden, UT. GTR-INT 133, 113 p.

    Google Scholar 

Download references

Acknowledgements

This study is supported in part by the Centre of Excellence for Biosecurity Risk Analysis, School of BioSciences, University of Melbourne, Australia. Thoughtful review comments by Lori Dalton, Steve Lane, Anca Hanea, James Camac, and the two editors have greatly improved this chapter.

Author information

Authors and Affiliations

  1. CEBRA, School of BioSciences, The University of Melbourne, Parkville, Victoria, 3010, Australia

    Andrew P. Robinson

Authors
  1. Andrew P. Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew P. Robinson .

Editor information

Editors and Affiliations

  1. University of Bern, Bern, Switzerland

    Claus Beisbart

  2. Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Nicole J. Saam

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Robinson, A.P. (2019). Testing Simulation Models Using Frequentist Statistics. In: Beisbart, C., Saam, N. (eds) Computer Simulation Validation. Simulation Foundations, Methods and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-70766-2_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-70766-2_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-70765-5

  • Online ISBN: 978-3-319-70766-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation