Abstract
In the domain of dynamic modeling and simulation, the assurance of model validity is a prominent challenge. An extensive number of contributions concerning model tests, terminology, and the epistemological foundations of validation have been elaborated. These contributions, however, do not fully answer the questions for novice modelers, which validation tests to choose, when and how to apply them, and at what point to cease their validation efforts. The intention here is to help close this gap by introducing a complexity hierarchy of validation tests, an integrative validation process, and a decision heuristic about when to stop validation efforts. The chapter concludes by providing directions for future research.
You can understand and enjoy the adventure of science, because the thinking used in science is the same thinking you use in daily life. You use reality checks to decide whether the way you think the world is matches the way the world is. Craig Rusbult (2011)
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We intentionally excluded commonly known procedures, e.g., evaluation of the model’s face validity, walkthroughs, or group model building, from Table 7.1. These are procedures which provide the environment in which validation tests are executed. For instance, a structured walkthrough meeting assembles a group of experts who together inspect the structure and behavior of a simulation model and discuss its validity, and detect and document faults (Balci, 1994). To achieve this purpose, the tests as outlined in Table 7.1 are applied.
References
Anastasakis, L., Olphert, C. W., & Wilson, J. M. (2008). Experiences in using a contingency factor-based validation methodology for spreadsheet DSS. Journal of the Operational Research Society, 59(6), 756–761.
Back, G., Love, G., & Falk, J. (2000). The doing of model verification and validation: Balancing cost and theory. Paper presented at the 18th International Conference of the System Dynamics Society, Bergen, Norway.
Balci, O. (1994). Validation, verification and testing techniques throughout the life cycle of a simulation study. Annals of Operations Research, 53(1), 121–173.
Balci, O., & Sargent, R. G. (1981). A methodology for cost-risk analysis in the statistical validation of simulation-models. Communications of the ACM, 24(4), 190–197.
Barlas, Y. (1989). Multiple tests for validation of system dynamics type of simulation models. European Journal of Operational Research, 42(1), 59–87.
Barlas, Y. (1990). An autocorrelation function test for output validation. Simulation, 55(1), 7–16.
Barlas, Y. (1996). Formal aspects of model validity and validation in system dynamics. System Dynamics Review, 12(3), 183–210.
Barlas, Y., & Carpenter, S. (1990). Philosophical roots of model validation: Two paradigms. System Dynamics Review, 6(2), 148–166.
Bossel, H. (2004). Systeme, dynamik, simulation: Modellbildung, analyse und simulation komplexer systeme. Norderstedt, Germany: Books on Demand.
Bratley, P., Fox, B., & Schrage, L. (1987). A guide to simulation. New York: Springer.
Coyle, R. G., & Exelby, D. R. (2000). The validation of commercial system dynamics models. System Dynamics Review, 16(1), 27–41.
Defense Modeling and Simulation Office. (1996). Verification, validation and accreditation: Recommended practices guide. US Department of Defense, Washington.
Déry, R., Landry, M., & Banville, C. (1993). Revisiting the issue of model validation in OR: An epistemological view. European Journal of Operational Research, 66(2), 168–183.
Eberlein, R. L., & Wang, Q. (1983). Validation of oscillatory behavior modes using spectral analysis. Paper presented at the International Conference of the System Dynamics Society, Chestnut Hill, PA.
Finlay, P. N., Forsey, G. J., & Wilson, J. M. (1988). The validation of expert systems—Contrasts with traditional methods. Journal of the Operational Research Society, 39(10), 933–938.
Finlay, P. N., & Wilson, J. M. (1997). Validity of decision support systems: Towards a validation methodology. Systems Research and Behavioral Science, 14(3), 169–182.
Finlay, P. N., & Wilson, J. M. (2000). A survey of contingency factors affecting the validation of end-user spreadsheet-based decision support systems. Journal of the Operational Research Society, 51(8), 949–958.
Ford, D. N. (1999). A behavioral approach to feedback loop dominance analysis. System Dynamics Review, 15(1), 3–36.
Forrester, J. W. (1961). Industrial dynamics. Cambridge, MA: Productivity Press.
Forrester, J. W. (1994). Policies, decisions, and information sources for modeling. In J. D. W. Morecroft & J. D. Sterman (Eds.), Modeling for learning organizations (pp. 51–84). Portland, OR: Productivity Press.
Forrester, J. W. (2007). System dynamics: The next fifty years. System Dynamics Review, 23(2–3), 359–370.
Forrester, J. W., & Senge, P. M. (1980). Tests for building confidence in system dynamics models. In A. A. Legasto, J. W. Forrester, & J. M. Lyneis (Eds.), System dynamics: TIMS studies in the management sciences (Vol. 14). Amsterdam: North-Holland.
Forrester, J. W. (1985). The ‘model’ versus a modeling ‘process’. System Dynamics Review, 1(1), 133–143.
Gass, S. I. (1983). Decision-aiding models—validation, assessment, and related issues for policy analysis. Operations Research, 31(4), 603–631.
Glaser, B. G. (1978). Theoretical sensitivity: Advances in the methodology of grounded theory. Mill Valley, CA: Sociology Press.
Glaser, B. G. (1992). Basics of grounded theory analysis. Mill Valley, CA: Sociology Press.
Glaser, B. G., & Strauss, A. L. (Eds.). (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine De Gruyter.
Graham, A. K. (1980). Parameter estimation in system dynamics modeling. In J. Randers (Ed.), Elements of the system dynamics method (pp. 143–161). Cambridge, MA: Productivity Press.
Groesser, S. N., & Bruppacher, S. (2007). Decisions in the planning process of a building: development of a dynamic model about individual’s energy efficiency intention over time. Paper presented at the 25th International Conference of the System Dynamics Society, Boston.
Groesser, S. N., & Schaffernicht, M. (2012). Mental models of dynamic systems: Taking stock and looking ahead. System Dynamics Review.
Groesser, S. N., & Ulli-Beer, S. (2006). Diffusion dynamics of energy-efficient innovations in the residential building environment: A simulation approach. Paper presented at the 24th International Conference of the System Dynamics Society, Nijmegen, The Netherlands.
Groesser, S. N., & Ulli-Beer, S. (2008). Long-Term innovation diffusion in the building construction industry: Empirically-based theory building. Paper presented at the 26th International Conference of the System Dynamics Society, Athens, Greece.
GĂ¼neralp, B. (2006). Towards coherent loop dominance analysis: Progress in Eigenvalue elasticity analysis. System Dynamics Review, 22(3), 263–289.
Hall, R. I. (1976). A system pathology of an organization: The rise and fall of the old Saturday evening post. Administrative Science Quarterly, 21(2), 185–211.
Homer, J. B. (1983). Partial-model testing as a validation tool for system dynamics. Paper presented at the International Conference of the System Dynamics Society, Chestnut Hill, PA.
Homer, J. B. (1996). Why we iterate: Scientific modeling in theory and practice. System Dynamics Review, 12(1), 1–19.
Kahneman, D., & Tversky, A. (1979). Prospect theory – analysis of decision under risk. Econometrica, 47(2), 263–291.
Kampmann, C. E., & Oliva, R. (2009). System dynamics: Analytical methods for structural dominance analysis. In Encyclopedia of complexity and systems science. New York/London/Berlin: Springer.
Kennedy, P. (2003). A guide to econometrics (5th ed.). Oxford: Basil Blackwell.
Kleijnen, J. P. C. (1995). Statistical validation of simulation-models. European Journal of Operational Research, 87(1), 21–34.
Kleindorfer, G. B., & Geneshan, R. (1993). The philosophy of science and validation in simulation, Proceedings of the 25th conference on winter simulation. Los Angeles: Winter simulation conference.
Kirchner, J. W. (1984). A methodological framework for system dynamics model evaluation. Dynamica, 10(1), 9–15.
Lane, D. C. (1995). The folding star: A comparative reframing and extension of validity concepts in system dynamics. Paper presented at the International Conference of the System Dynamics Society, Tokyo.
Law, A. M., & Kelton, D. W. (1982). Simulation modeling and analysis. New York: McGraw-Hill.
Oliva, R. (2003). Model calibration as a testing strategy for system dynamics models. European Journal of Operational Research, 151(3), 552–568.
Olphert, C. W., & Wilson, J. M. (2004). Validation of decision-aiding spreadsheets: The influence of contingency factors. Journal of the Operational Research Society, 55(1), 12–22.
Oral, M., & Kettani, O. (1993). The facets of the modeling and validation process in operations research. European Journal of Operational Research, 66(2), 216–234.
Peterson, D. W., & Eberlein, R. L. (1994). Reality checks: A bridge between systems thinking and system dynamics. System Dynamics Review, 10(2/3), 159–174.
Richardson, G. P., & Pugh, A. L., III. (1981). Introduction to system dynamics modeling with DYNAMO. Cambridge, MA: Productivity Press.
Rusbult, C. (2011). Introduction to scientific method: Using logical reality checks in science and other areas of life. Madison, WI. Accessed May 2, 2011, from http://www.asa3.org/ASA/education/think/scientific-method.htm.
Sargent, R. G. (1992). Validation and verification of simulation models, Proceedings of the 24th conference on winter simulation. Arlington, VA: Winter simulation conference.
Sargent, R. G. (2008). Verification and validation of simulation models, Proceedings of the 40th conference on winter simulation. Miami, FL: Winter simulation conference.
Saysel, A. K., & Barlas, Y. (2006). Model simplification and validation with indirect structure validity tests. System Dynamics Review, 22(3), 241–262.
Schaffernicht, M., & Groesser, S. N. (2011). Comparison of mental models: Development of a comprehensive method. European Journal of Operational Research, 210(2), 57–67.
Schwaninger, M. (2009). Intelligent organizations: Powerful models for systemic management (2nd ed.). Berlin/Heidelberg: Springer.
Schwaninger, M., & Groesser, S. N. (2008). Model-based theory-building with system dynamics. Systems Research and Behavioral Science, 25(4), 447–465.
Schwaninger, M., & Groesser, S. N. (2009). System dynamics modeling: Validation for quality assurance. In Encyclopedia of complexity and system science. Berlin/London/Paris: Springer.
Simon, H. A. (1964). Models of man: Social and rational mathematical essays on rational human behavior in a social setting. New York: Wiley.
Sterman, J. D. (1984). Appropriate summary statistics for evaluating the historical fit of system dynamics models. Dynamica, 10(2), 51–66.
Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. Boston: McGraw-Hill.
Sterman, J. D. (2002). All models are wrong: Reflections on becoming a systems scientist. System Dynamics Review, 18(4), 501–531.
Strauss, A. L., & Corbin, J. M. (Eds.). (1997). Grounded theory in practice. Thousand Oaks, CA: Sage Publications.
Strauss, A. L., & Corbin, J. M. (1998). Basics of qualitative research techniques and procedures for developing grounded theory (2nd ed.). Thousand Oaks, CA: Sage Publications.
Taylor, A. J. (1980). Loop analysis and validation. Dynamica, 6(1), 2–8.
Weick, K. E. (1989). Theory construction as disciplined imagination. Academy of Management Review, 14(4), 516–531.
Weil, H. B. (1983). What is an adequate model? Paper presented at the International Conference of the System Dynamics Society, Chestnut Hill, PA.
Wolstenholme, E. F. (1999). Qualitative vs quantitative modelling: The evolving balance. Journal of the Operational Research Society, 50(4), 422–428.
Zeigler, B. P., Praehofer, H., & Kim, T. G. (2000). Theory of modeling and simulation (2nd ed.). San Diego, CA: Academic.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Grösser, S.N. (2013). Hierarchy, Process, and Cessation: Contributions to When and How to Validate. In: Co-Evolution of Standards in Innovation Systems. Contributions to Management Science. Physica, Heidelberg. https://doi.org/10.1007/978-3-7908-2858-0_7
Download citation
DOI: https://doi.org/10.1007/978-3-7908-2858-0_7
Published:
Publisher Name: Physica, Heidelberg
Print ISBN: 978-3-7908-2857-3
Online ISBN: 978-3-7908-2858-0
eBook Packages: Business and EconomicsBusiness and Management (R0)