Advertisement

Error propagation in computer models: analytic approaches, advantages, disadvantages and constraints

  • K. K. Benke
  • S. Norng
  • N. J. Robinson
  • L. R. Benke
  • T. J. Peterson
Original Paper

Abstract

Uncertainty and its propagation in computer models has relevance in many disciplines, including hydrology, environmental engineering, ecology and climate change. Error propagation in a model results in uncertainty in prediction due to uncertainties in model inputs and parameters. Common methods for quantifying error propagation are reviewed, namely Differential Error Analysis and Monte Carlo Simulation, including underlying principles, together with a discussion on their differences, advantages and disadvantages. The separate case of uncertainty in the model calibration process is different to error propagation in a fixed model in that it is associated with a dynamic process of iterative parameter adjustment, and is compared in the context of non-linear regression and Bayesian approaches, such as Markov Chain Monte Carlo Simulation. Error propagation is investigated for a soil model representing the organic carbon depth profile and also a streamflow model using probabilistic simulation. Different sources of error are compared, including uncertainty in inputs, parameters and geometry. The results provided insights into error propagation and its computation in systems and models in general.

Keywords

Uncertainty Differential error analysis Monte Carlo simulation Soil Streamflow 

References

  1. Abusam A, Keesman KJ, Van Straten G (2003) Forward and backward uncertainty propagation: an oxidation ditch modelling example. Water Res 37(2):429–435CrossRefGoogle Scholar
  2. Bayes T (1763) An essay towards solving a problem in the doctrine of chances. Philos Trans R Soc Lond 53:1393–1442Google Scholar
  3. Benke KK, Benke KE (2013) Uncertainty in health risks from artificial lighting due to disruption of circadian rhythm and melatonin secretion: a review. Hum Ecol Risk Assess 19:916–929CrossRefGoogle Scholar
  4. Benke KK, Robinson NJ (2017) Quantification of uncertainty in mathematical models: the statistical relationship between field and laboratory pH measurements. Appl Environ Soil Sci 20:12.  https://doi.org/10.1155/2017/5857139 CrossRefGoogle Scholar
  5. Benke KK, Lowell KE, Hamilton AJ (2008) Parameter uncertainty, sensitivity analysis and prediction error in a water-balance hydrological model. Math Comput Model 47:1134–1149CrossRefGoogle Scholar
  6. Beven K (2008) Comment on ‘‘Equifinality of formal (DREAM) and informal (GLUE) Bayesian approaches in hydrologic modeling? Vrugt J.A., ter Braak, C.J.F., Gupta, H.V., Robinson, B.A., 2008. Stoch Environ Res Risk Assess.  https://doi.org/10.1007/s00477-008-0283-x CrossRefGoogle Scholar
  7. Beven KJ, Binley AM (1992) The future of distributed models: model calibration and predictive uncertainty. Hydrol Process 6:279–298CrossRefGoogle Scholar
  8. Beverly C, Christy B, Weeks A (2006) Application of the 2CSalt model to the Bet Bet, Wild Duck, Gardner and Sugarloaf Catchments in Victoria. Department of Primary Industries, VictoriaGoogle Scholar
  9. Bolstad WM (2007) Introduction to Bayesian statistics, 2nd edn. Wiley, HobokenCrossRefGoogle Scholar
  10. Box GEP, Cox DR (1982) An analysis of transformations revisited, rebutted. J Am. Stat Assoc 77:209–210CrossRefGoogle Scholar
  11. Brown JD, Heuvelink GBM (2005) 79: assessing uncertainty propagation through physically based models of soil water flow and solute transport. In: Anderson MG (ed) Encyclopaedia of hydrological sciences. Wiley, New YorkGoogle Scholar
  12. Buckland ST (1984) Monte Carlo confidence intervals. Biometrics 40:811–817CrossRefGoogle Scholar
  13. Carlin BP, Louis TA (2008) Bayesian methods for data analysis, 3rd edn. Chapman and Hall, Boca RatonGoogle Scholar
  14. Chen C-S, Chen C-S (2018) A composite spatial predictor via local criteria under a misspecified model. Stoch Environ Res Risk Assess 32:341–355CrossRefGoogle Scholar
  15. Doherty J (2003) MICA: model independent Markov Chain Monte Carlo analysis. Watermark Numerical Computing, BrisbaneGoogle Scholar
  16. Donnelly SM, Kramer A (1999) Testing for multiple species in forest samples: an evaluation and comparison of tests for equal relative variation. Am J Phys Anthopol 108:507–529CrossRefGoogle Scholar
  17. Dotto CBS, Mannina G, Kleidorfer M, Vezzaro L, Henrichs M, McCarthy DT, Freni G, Rauch W, Deletic A (2012) Comparison of different uncertainty techniques in urban stormwater quantity and quality modelling. Water Res 46(8):2545–2558CrossRefGoogle Scholar
  18. Freeze RA (2004) The role of stochastic hydrogeological modeling in real-world engineering applications. Stoch Env Res Risk Assess 18(4):286–289CrossRefGoogle Scholar
  19. Freni G, Mannina G (2009) Urban runoff modelling uncertainty: comparison among Bayesian and pseudo-Bayesian methods. Environ Model Softw 24:1100–1111CrossRefGoogle Scholar
  20. Freni G, Mannina G (2010) Bayesian approach for uncertainty quantification in water quality modelling: the influence of prior distribution. J Hydrol 392:31–39CrossRefGoogle Scholar
  21. Freund JE (1998) Mathematical statistics. Prentice-Hall, New YorkGoogle Scholar
  22. Garg A, Vijayaraghavan V, Zhang J, Li S, Liang X (2017a) Design of robust battery capacity model for electric vehicle by incorporation of uncertainties. Int J Energy Res 41(10):1436–1451CrossRefGoogle Scholar
  23. Garg A, Vijayaraghavan V, Zhang J, Lam JSL (2017b) Robust model design for evaluation of power characteristics of the cleaner energy system. Renew Energy 112:302–313CrossRefGoogle Scholar
  24. Gelman A, Carlin JB, Stern HS, Rubin DB (2004) Bayesian data analysis. Chapman and Hall/CRC, Boca RatonGoogle Scholar
  25. Geman S, Geman D (1984) Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Trans Pattern Anal Mach Intell 6:721–741CrossRefGoogle Scholar
  26. Goidts E, van Wesmael B, Crucifix M (2009) Magnitude and sources of uncertainties in soil organic carbon (SOC) stock assessments at various scales. Eur J Soil Sci 60:723–729CrossRefGoogle Scholar
  27. Gupta HV, Clark MP, Vrugt JA, Abramowitz G, Ye M (2012) Towards a comprehensive assessment of model structural adequacy. Water Resour Res 48:W08301.  https://doi.org/10.1029/2011WR011044 CrossRefGoogle Scholar
  28. Haas CN, Eisenberg JNS (2001) Risk assessment. In: Fewtrell L, Bartram J (eds) Water quality: guidelines, standards and health. Assessment of risk and risk management for water-related infectious disease. IWA Publishing, London, pp 161–183Google Scholar
  29. Hamilton AJ, Basset Y, Benke KK, Grimbacher PS, Miller SE, Novotný V, Samuelson GA, Stork NE, Weiblen GD, Yen JD (2010) Quantifying uncertainty in estimation of tropical arthropod species richness. Am Nat 176:90–95CrossRefGoogle Scholar
  30. Hamilton AJ, Novotny V, Waters EK, Basset Y, Benke KK, Grimbacher PS, Miller SE, Samuelson GA, Weiblen GD, Yen JD, Stork NE (2013) Estimating global arthropod species richness: refining probabilistic models using probability bounds analysis. Oecologia 171:357–365.  https://doi.org/10.1007/s00442-012-2434-5 CrossRefGoogle Scholar
  31. Helton JC (1993) Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal. Reliab Eng Syst Saf 42:327–367CrossRefGoogle Scholar
  32. Heuvelink GBM, Burrough PA (2002) Developments in statistical approaches to spatial uncertainty and its propagation. Int J Geogr Inf Sci 16:111–113CrossRefGoogle Scholar
  33. Heuvelink GB, Burrough PA, Stein A (1989) Propagation of errors in spatial modelling with GIS. Int J Geogr Inf Syst 3(4):303–322CrossRefGoogle Scholar
  34. Kavetski D, Kuczera G, Franks SW (2006a) Bayesian analysis of input uncertainty in hydrological modelling: 1. Theory. Water Resour J 42(W03407):1–9Google Scholar
  35. Kavetski D, Kuczera G, Franks SW (2006b) Calibration of conceptual hydrological models revisited: 1. Overcoming numerical artefacts. J Hydrol 32(1–2):173–186CrossRefGoogle Scholar
  36. Kavetski D, Kuczera G, Franks SW (2006c) Calibration of conceptual hydrological models revisited: 2. Improving optimisation and analysis. J Hydrol 32(1–2):187–201CrossRefGoogle Scholar
  37. Kline SJ, McClintock FA (1953) Describing uncertainties in single sample experiments. Mech Eng 75:3–8Google Scholar
  38. Kroese DP, Taimre T, Botev ZI (2011) Handbook of Monte Carlo methods. Wiley, New YorkCrossRefGoogle Scholar
  39. Kuczera G, Parent E (1998) Monte Carlo assessment of parameter uncertainty in conceptual models: the metropolis algorithm. J Hydrol 211:69–85CrossRefGoogle Scholar
  40. Lagos-Álvarez BM, Toribio RF, Figueroa-Zunuga J, Mateu J (2017) Geostastistical mixed beta regression: a Bayesian approach. Stoch Environ Res Risk Assess 31:571–584CrossRefGoogle Scholar
  41. Lark RM, Webster R (2006) Geostastistical mapping of geomorphic surfaces in the presence of trend. Earth Surf Process Landf 31:862–874CrossRefGoogle Scholar
  42. Littleboy M (2005) UserGuide_2CSalt, Rev 1.0, CRC for Catchment Hydrology, Australia. www.toolkit.net.au
  43. Malone BP, McBratney AB, Minasny B (2011) Empirical estimates of uncertainty for mapping continuous depth functions of soil attributes. Geoderma 160:614–625CrossRefGoogle Scholar
  44. Mandel J (1964) The statistical analysis of experimental data. Dover Publications Inc., New YorkGoogle Scholar
  45. McKay MD (1995) Evaluating prediction uncertainty. Report No. LA-12915-MS, Statistics Group, Los Alamos National Laboratory, NM, USAGoogle Scholar
  46. Minasny B, Vrugt JA, McBratney AB (2011) Confronting uncertainty in model-based geostatistics using Markov Chain Monte Carlo simulation. Geoderma 163:150–162CrossRefGoogle Scholar
  47. Nelson MA, Bishop TFA, Odeh IOA, Triantafilis J (2011) An error budget for different sources of error in digital soil mapping. Eur J Soil Sci 62:417–430CrossRefGoogle Scholar
  48. Oberkampf WL, Helton JC, Joslyn CA, Wojtkiewicz SF, Ferson S (2004) Challenge problems: uncertainty in system response given uncertain parameters. Reliab Eng Syst Saf 85:11–19CrossRefGoogle Scholar
  49. Oliver MA, Webster R (2014) A tutorial guide to geostatistics: computing and modelling variograms and kriging. Catena 113:56–69CrossRefGoogle Scholar
  50. Oritz JO, Felgueiras CA, Camargo ECG, Rennó CD, Oritz MJ (2017) Spatial modelling of soil lime requirements with uncertainty assessment using geostatistical sequential indicator simulation. Open J Soil Sci 7:133–148CrossRefGoogle Scholar
  51. Oya A, Navarro-Moreno J, Ruiz-Molina JC (2007) Spatial random field simulation by a numerical series representation. Stoch Env Res Risk Assess 21:317–326CrossRefGoogle Scholar
  52. Parratt LG (1971) Probability and experimental errors in science. Dover Publications Inc., New YorkGoogle Scholar
  53. Patil A, Deng ZQ, Malone RF (2011) Input data resolution-induced uncertainty in watershed modelling. Hydrol Process 25:2302–2312CrossRefGoogle Scholar
  54. Qian SS, Stow CA, Borsuk ME (2003) On Monte Carlo methods for Bayesian inference. Ecol Model 159:269–277CrossRefGoogle Scholar
  55. Raftery AE, Gneiting T, Balabdaoui F, Polakowski M (2005) Using Bayesian model averaging to calibrate forecast ensembles. Mon Weather Rev 133:1155–1174CrossRefGoogle Scholar
  56. Rayment GE, Lyons DJ (2011) Soil chemical methods: Australasia, vol 3. CSIRO publishing, CollingwoodGoogle Scholar
  57. Refsgaard JC, van der Sluijs JP, Højberg AL, Vanrolleghem PA (2007) Uncertainty in the environmental modelling process—a framework and guidance. Environ Model Softw 22:1543–1556CrossRefGoogle Scholar
  58. Robinson NJ, Benke KK, Norng S (2015) Identification and interpretation of sources of uncertainty in soils change in a global systems-based modelling process. Soil Res 53(6):592–604CrossRefGoogle Scholar
  59. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323:533–536CrossRefGoogle Scholar
  60. Savelyeva E, Utkin S, Kazakpv S, Demyanov V (2010) Modeling spatial uncertainty for locally uncertain data. Geoenv VII Geostat Environ Appl 16:295–306Google Scholar
  61. Sklar A (1959) Fonctions de répartition à n dimensions et leurs marges”. Publ Inst Stat Univ Paris 8:229–231Google Scholar
  62. Smith MS, Khaled MA (2012) Estimation of copula models with discrete margins via Bayesian data augmentation. J Am Stat Assoc 107:290–303CrossRefGoogle Scholar
  63. Stenson MP, Littleboy M, Gilfedder M (2011) Estimation of water and salt generation from unregulated upland catchments. Environ Model Softw.  https://doi.org/10.1016/j.envsoft.2011.05.013 CrossRefGoogle Scholar
  64. Taylor JR (1997) An introduction to error analysis. University Science Books, SausalitoGoogle Scholar
  65. Trucano T, Swiler L, Igusa T, Oberkampf W, Pilch W (2006) Calibration, validation, and sensitivity analysis: what’s what. Reliability engineering and system safety. In: 4th international conference on sensitivity. analysis of model output-SAMO 2004, vol 91, No. 10–11, pp 1331–1357Google Scholar
  66. Vandenberghe V, Bauwens W, Vanrolleghem PA (2007) Evaluation of uncertainty propagation into river water quality predictions to guide future monitoring campaigns. Environ Model Softw 22:725–732CrossRefGoogle Scholar
  67. Vose D (2008) Risk analysis. Wiley, ChichesterGoogle Scholar
  68. Vrugt JA, Gupta HV, Bouten W, Sorooshian S (2003) The shuffled complex evolutionary metropolis algorithm for optimisation and uncertainty assessment of hydrological parameters. Water Resour Res 39(8):1-1–1-9Google Scholar
  69. Vrugt JA, ter Braak CJF, Gupta HV, Robinson BA (2009a) Equifinality of formal (DREAM) and informal (GLUE) Bayesian approaches in hydrologic modeling? Stoch Environ Res Risk Assess 23:1011–1026.  https://doi.org/10.1007/s00477-008-0274-y CrossRefGoogle Scholar
  70. Vrugt JA, ter Braak CJF, Gupta HV, Robinson BA (2009b) Response to Keith Beven comment on “equifinality of formal (DREAM) and informal (GLUE) Bayesian approaches in hydrologic modeling?”. Stoch Environ Res Risk Assess 23:1061–1062.  https://doi.org/10.1007/s00477-008-0284-9 CrossRefGoogle Scholar
  71. Wang D, Lu WZ (2006) Forecasting Ozone Levels and analyzing their dynamics by a Bayesian multilayer perceptron model for two air-monitoring sites in Hong Kong. Hum Ecol Risk Assess 12:313–327CrossRefGoogle Scholar
  72. Wöhling T, Vrugt JA (2008) Combining multi-objective optimization and Bayesian model averaging to calibrate forecast ensembles of soil hydraulic models. Water Resour Res 44:W12432.  https://doi.org/10.1029/2008WR007154 CrossRefGoogle Scholar
  73. Wu F, Chen C (2009) Bayesian updating of parameters for a sediment entrainment model via Markov Chain Monte Carlo. J Hydraul Eng 135:22–37CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Economic Development, Jobs, Transport and Resources (DEDJTR)AgriBio CentreBundooraAustralia
  2. 2.School of EngineeringUniversity of MelbourneParkvilleAustralia
  3. 3.Department of Economic Development, Jobs, Transport and Resources (DEDJTR)Epsom CentreEpsomAustralia
  4. 4.School of Science, Information Technology and EngineeringUniversity of BallaratMt HelenAustralia

Personalised recommendations