Advertisement

Stochastic Hydrology and Hydraulics

, Volume 11, Issue 3, pp 229–254 | Cite as

On the problem of model validation for predictive exposure assessments

  • M. B. Beck
  • J. R. Ravetz
  • L. A. Mulkey
  • T. O. Barnwell
Originals

Abstract

The development and use of models for predicting exposures are increasingly common and are essential for many risk assessments of the United States Environmental Protection Agency (EPA). Exposure assessments conducted by the EPA to assist regulatory or policy decisions are often challenged to demonstrate their “scientific validity”. Model validation has thus inevitably become a major concern of both EPA officials and the regulated community, sufficiently so that the EPA's Risk Assessment Forum is considering guidance for model validation. The present paper seeks to codify the issues and extensive foregoing discussion of validation with special reference to the development and use of models for predicting the impact of novel chemicals on the environment. Its preparation has been part of the process in formulating a White Paper for the EPA's Risk Assessment Forum. Its subject matter has been drawn from a variety of fields, including ecosystem analysis, surface water quality management, the contamination of groundwaters from high-level nuclear waste, and the control of air quality. The philosophical and conceptual bases of model validation are reviewed, from which it is apparent that validation should be understood as a task of product (or tool) design, for which some form of protocol for quality assurance will ultimately be needed. The commonly used procedures and methods of model validation are also reviewed, including the analysis of uncertainty. Following a survey of past attempts at resolving the issue of model validation, we close by introducing the notion of a model having maximum relevance to the performance of a specific task, such as, for example, a predictive exposure assessment.

Key words

Model validation analysis of uncertainty model verification quality assurance system identification model calibration 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ababou, R.; Sagar, B.; Wittmeyer, G. 1992: Testing procedures for spatially distributed flow models, Adv. Water Resour. 15(3), 181–198CrossRefGoogle Scholar
  2. Akaike, H. 1974: A new look at statistical model identification, IEEE Trans on Automatic Control AC-19, 716–722CrossRefGoogle Scholar
  3. American Society for Testing and Materials. 1984: Standard practice for evaluating environmental fate models of chemicals, Standard E 978-84, American Society for Testing and Materials, Philadelphia, PennsylvaniaGoogle Scholar
  4. Beck, M.B. 1987: Water quality modeling: a review of the analysis of uncertainty, Water Resour. Res. 23(8), 1393–1442CrossRefGoogle Scholar
  5. Beck, M.B. 1991: Forecasting environmental change, J. Forecasting 10(1/2), 3–19Google Scholar
  6. Beck, M.B. 1994: Understanding uncertain environmental systems, in Predictability and Nonlinear Modelling in Natural Sciences and Economics (J. Grasman and G. van Straten, eds), Kluwer, Dordrecht, pp 294–311Google Scholar
  7. Beck, M.B.; Halfon E. 1991: Uncertainty, identifiability and the propagation of prediction errors: a case study of Lake Ontario, J. Forecasting 10(1/2), 135–161Google Scholar
  8. Beck, M.B.; Chen, J. 1977: Process models: validation and structural change, in Assessment of Environmental Impact (Proceedings of a Workshop), International Centre for Mathematical Sciences, Edinburgh, UK (in preparation)Google Scholar
  9. Burns, L.A. 1983: Validation of exposure models: the role of conceptual verification, sensitivity analysis, and alternative hypotheses, in Proceedings 6th Symposium, Aquatic Toxicology and Hazard Assessment (W E Bishop, R D Cardwell, and B B Heidolph, eds), American Society for Testing and Materials, Philadelphia, ASTM. Special Technical Publication 802, pp 255–281Google Scholar
  10. Burns, L.A.; Barber, M.C.; Bird, S.L.; Mayer, F.L., Suárez, A. 1990: PIRANHA: Pesticide and Industrial Chemical Risk Analysis and Hazard Assessment, Internal Report, Environmental Research Laboratory, United States Environmental Protection Agency, Athens, GeorgiaGoogle Scholar
  11. Cale, W.G. Jr.; O'Neill, R.V.; Shugart, H.H. 1983: Development and application of desirable ecological models, Ecological Modelling 18, 171–186CrossRefGoogle Scholar
  12. Caswell, H. 1976: The validation problem, in Systems Analysis and Simulation in Ecology (B C Patten, ed), Academic, New York, Vol IV, pp 313–325Google Scholar
  13. Chen, J.; Beck, M.B. 1977: Discriminating power of the EPA's Multi-Media Model in performing a screening task, Technical Report, United States Environmental Protection Agency, Athens, Georgia (in review)Google Scholar
  14. Costanza, R.; Sklar, F.H. 1975: Mathematical models of freshwater wetlands and shallow water ecosystems: an articulated review, in Proceedings SCOPE International Conference on Freshwater Wetlands and Shallow Water BodiesGoogle Scholar
  15. Cox, D.C.; Baybutt, P. 1981: Methods of uncertainty analysis: a comparative survey, Risk Analysis 1(4), 251–258CrossRefGoogle Scholar
  16. Davis, P.A.; Price, L.L.; Wahi, K.K.; Goodrich, M.T.; Gallegos, D.P.; Bonano E.J.; Guzowksi, R.V. 1990: Components of an overall performance assessment methodology, Report NUREG/CR5256, SAND88-3020, Sandia National Laboratories, Albuquerque, New MexicoGoogle Scholar
  17. Donigian A.S.; Rao, P.S.C. 1990: Selection, application, and validation of environmental models, in Proceedings International Symposium on Water Quality Modeling of Agricultural Non-Point Sources, (D G DeCoursey, ed), Report ARS-81, Agricultural Research Service, United States Department of Agriculture, pp 577–600Google Scholar
  18. Environmental Protection Agency. 1991: Guidelines for Exposure Assessment, Science Advisory Board Draft Final, AugustGoogle Scholar
  19. Funtowicz, S.O.; Ravetz, J. 1990: Uncertainty and Quality in Science for Policy, Kluwer, DordrechtGoogle Scholar
  20. Hassanizadeh, S.M.; Carrera, J. 1992: Editorial: special issue on validation of geo-hydrological models, Adv. Water Resour. 15(1), 1–3CrossRefGoogle Scholar
  21. Hermann, C.F. 1967: Validation problems in games and simulations with special reference to models of international politics, Behavioral Science 12, 216–231Google Scholar
  22. Hornberger, G.M.; Spear, R.C. 1980: Eutrophication in Peel Inlet, I. Problem-defining behaviour and a mathematical model for the phosphorus scenario, Water Res. 14, 29–42CrossRefGoogle Scholar
  23. Iman, R.L.; Helton, J.C. 1988: An investigation of uncertainty and sensitivity analysis techniques for computer models, Risk Analysis 8(1), 71–90CrossRefGoogle Scholar
  24. Konikow, L.F.; Bredehoeft, J.D. 1992: Ground-water models cannot be validated, Adv. Water Resour. 15(1), 75–83CrossRefGoogle Scholar
  25. Luis, S.J.; McLaughlin, D.B. 1992: A stochastic approach to model validation, Adv. Water Resour. 15(1), 15–32CrossRefGoogle Scholar
  26. MacFarlane, A.G.J. 1990: Interactive computing: a revolutionary medium for teaching and design, Computing and Control Engineering Journal 1(4), 149–158CrossRefGoogle Scholar
  27. Mankin, J.B.; O'Neill, R.V.; Shugart, H.H.; Rust, B.W. 1977: The importance of validation in ecosystem analysis, in New Directions in the Analysis of Ecological Systems, (G S Innis, ed), Simulation Council, La Jolla, California, Proceedings Series 5(1), pp 63–72Google Scholar
  28. Mihram, G.A. 1973: Some practical aspects of the verification and validation of simulation models, Operations Research Quarterly 23, 17–29CrossRefGoogle Scholar
  29. Miller, D.R.; Butler, G.; Bramall, C. 1976: Validation of ecological system models, J. Environmental Management 4, 383–401Google Scholar
  30. Morton, A. 1993: Mathematical models: questions of trustworthiness, British J. Phil of Science 44, 659–674Google Scholar
  31. National Acid Precipitation Assessment Program (NAPAP). 1990: Evaluation of regional acidic deposition models and selected applications of RADM, Acidic Deposition: State of Science and Technology, Vol I, Report 5, The National Acid Precipitation Assessment Program, Washington, DCGoogle Scholar
  32. National Research Council. 1990: Ground Water Models: Scientific and Regulatory Applications, Water Science and Technology Board, United States National Research Council, National Academy Press, Washington DCGoogle Scholar
  33. Oreskes, N.; Shrader-Frechette, K.; Belitz, K. 1994: Verification, validation, and confirmation of numerical models in the earth sciences, Science 263, 641–646Google Scholar
  34. Parrish, R.S.; Smith, C.N. 1990: A method for testing whether model predictions fall within a prescribed factor of true values, with an application to pesticide leaching, Ecological Modelling 51, 59–72CrossRefGoogle Scholar
  35. Popper, K.R. 1959: The Logic of Scientific Discovery, Harper, New YorkGoogle Scholar
  36. Popper, K.R. 1963: Conjectures and Refutations: The Growth of Scientific Knowledge, Harer, New YorkGoogle Scholar
  37. Reckhow, K.H.; Chapra, S.C. 1983: Confirmation of water quality models, Ecological Modelling 20, 113–133CrossRefGoogle Scholar
  38. Reckhow, K.H.; Clements, J.T.; Dodd, R.C. 1990: Statistical evaluation of mechanistic water-quality models, J. Environmental Engineering, Proc American Society of Civil Engineers, 116(2), 250–268.Google Scholar
  39. Scavia D. 1980: An ecological model of Lake Ontario, Ecological Modelling 8, 49–78CrossRefGoogle Scholar
  40. Spear, R.C.; Hornberger, G.M. 1980: Eutrophication in Peel Inlet, II. Identification of critical uncertainties via generalised sensitivity analysis, Water Res. 14, 43–49CrossRefGoogle Scholar
  41. United States Environmental Protection Agency. 1989: Resolution on Use of Mathematical Models by EPA for Regulatory Assessment and Decision-Making, Report, Environmental Engineering Committee, Science Advisory Board, United States Environmental Protection Agency, Washington DCGoogle Scholar
  42. Usunoff, E.; Carrera, J.; Mousavi, S.F. 1992: An approach to the design of experiments for discriminating among alternative conceptual models, Adv. Water Resour. 15(3), 199–214CrossRefGoogle Scholar
  43. Van Straten, G.; Keesman, K.J. 1991: Uncertainty propagation and speculation in projective forecasts of environmental change: a lake-eutrophication example, J. Forecasting 10(1/2), 163–190Google Scholar
  44. Varis, O. 1995: Belief networks for modelling and assessment of environmental change, Environmetrics 6, 439–444Google Scholar
  45. Versar Inc. 1988: Current and suggested practices in the validation of exposure assessment models, Report, Office of Health and Environmental Assessment, United States Environmental Protection Agency, Washington DCGoogle Scholar
  46. Young, P.C. 1978: General theory of modelling badly defined systems, in Modelling, Identification and Control in Environmental Systems, (G.C. Vansteenkiste, ed), North-Holland, Amsterdam, pp 103–135Google Scholar
  47. Young, P.C.; Hornberger, G.M.; Spear, R.C. 1978: Modelling badly defined systems — Some further thoughts, in Proceedings SIMSIG Simulation Conference, Australian National University, Canberra, 24–32Google Scholar
  48. Zimmerman, D.A.; Wahi, K.K.; Gutjahr, A.L.; Davis, P.A. 1990: A review of techniques for propagating data and parameter uncertainties in high-level radioactive waste repository performance assessment models, Report NUREG/CR-5393, SAND89-1432, Sandia National Laboratories, Albuquerque, New MexicoGoogle Scholar

Copyright information

© Springer-Verlag 1997

Authors and Affiliations

  • M. B. Beck
    • 1
  • J. R. Ravetz
    • 2
  • L. A. Mulkey
    • 3
  • T. O. Barnwell
    • 3
  1. 1.Warnell School of Forest ResourcesUniversity of GeorgiaAthensUSA
  2. 2.The Research Methods Consultancy Ltd.LondonUK
  3. 3.Environmental Research LaboratoryUnited States Environmental Protection AgencyAthensUSA

Personalised recommendations