Abstract
This discussion challenges classic notions of validation, suggesting that ‘validity’ is not just an attribute of a model. It is a function of the relationship of a particular characteristic of the model (the probability that the model will produce a data set that will match a set it has not yet seen) and a user. If we shift our focus from the model to the user, we can identify other ways in which models can be of use as well as start clarifying how we can identify their goodness in multiple use scenarios. In this discussion, we distinguish validation from calibration, and research or generalizable models from site-specific models. We use literature from fields as varied as physics, systems ecology and computational linguistics to characterize the process of validation. Finally, by extending the use space for models beyond prediction, we introduce the possibility of assessments of goodness other than validation.
Keywords
- Social modeling
- validation
- computational models
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
U.S. Department of Defense: Instruction No. 5000.61 (December 9, 2009)
American Institute of Aeronautics and Astronautics, AIAA guide for the verification and validation of computational fluid dynamics simulations. G-077-1998e (1998)
U.S. Department of Commerce, National Institute of Standards and Technology: State Weights and Measures Laboratories Program Handbook. NIST Handbook 143 (2006), http://www.ncbi.nlm.nih.gov (accessed)
Beven, K.: Towards a coherent philosophy for modeling the environment. Proceedings of the Royal Society of London 458, 2465–2484 (2002)
McDonnell, J.J., Sivapalan, M., Vaché, K., Dunn, S., Grant, G., Haggerty, R., Hinz, C., Hooper, R., Kirchner, J., Roderick, M.L., Selker, J., Weiler, M.: Moving beyond heterogeneity and process complexity: A new vision for watershed hydrology. Water Resources Research 43, W07301 (2007), doi:10.1029/2006WR005467
Vogel, R.M., Sankarasubramanian, A.: Validation of a watershed model without calibration. Water Resources Research 39(10), 1292–1300 (2003)
Refsgaard, J.C., Henriksen, H.J.: Modeling guidelines – terminology and guiding principles. Advances in Water Resources 27, 71–82 (2004)
Oberkampf, W.L., Trucano, T.G.: Verification and validation in computational fluid dynamics. Progress in Aerospace Sciences 38(3), 209–272 (2002)
Oberkampf, W. L., Trucano, T. G.: Verification and validation benchmarks. Sandia report, SAND 2007-0853, U.S. Department of Energy, Sandia National Laboratories, Albuquerque, NM (2007)
Oberkampf, W. L., Barone, M. F.: Measures of agreement between computation and experiment: validation measures. Sandia report, SAND 2005-4302, U.S. Department of Energy, Sandia National Laboratories, Albuquerque, NM (2005)
Oberkampf, W. L., Trucano, T. G., Hirsch, C.: Verification, validation, and predictive capability in computational engineering and physics. Sandia report, SAND 2003-3769, U.S. Department of Energy, Sandia National Laboratories, Albuquerque, NM (2003)
McNamara, L.: Why models don’t forecast. Draft Paper Prepared for the Workshop Unifying Social Frameworks, August 16-17. National Research Council, Washington, DC (2010), http://www7.nationalacademies.org/bbcss/Why%20Models%20Dont%20Forecast-McNamara.pdf
Atkinson, R., Shiffrin, R.M.: Human memory: A proposed system and its control processes. In: Spence, K.W. (ed.) The Psychology of Learning and Motivation: Advances in Research and Theory, vol. 2, pp. 89–195. Academic Press, New York (1968)
Dewey, J.: How we think. FQ Books, New York (1910, 2010)
Ormrod, J.E.: Educational psychology: developing learners, 4th edn. Merrill Prentice Hall, Upper Saddle River (2003)
Bruner, J., Goodnow, J., Austin, A.: A study of thinking. Wiley, New York (1956)
Piaget, J.: The Psychology of Intelligence. Routledge, New York (1950)
Richardson, V.: Constructivist pedagogy. Teachers College Record 105(9), 1623–1640 (2003)
Horst, W., Rittel, J., Webber, M.M.: Dilemmas in a general theory of planning. Policy Sciences 4, 155–169 (1973)
Holland, J.: Complex adaptive systems. Daedalus 121(1), 17–30 (1992)
Beemer, B., Gregg, D.G.: Advisory systems to support decision making. In: Handbook on Decision Support Systems. Springer, London (2008)
Sniezek, J.A.: Judge Advisor Systems Theory and Research and Applications to Collaborative Systems and Technology. In: Proceedings of the 32nd Hawaii International Conference on System Sciences, pp. 1057–1066. IEEE (1999)
Shieber, S.: Criteria for designing computer facilities for linguistic analysis. Linguistics 23, 189–211 (1985)
Janssen, M.A., Ostrom, E.: Empirically based, agent-based models. Ecology and Society 11(2) (2006), http://www.ecologyandsociety.org/vol11/iss2/art37/ (accessed)
Karas, T.: Modelers and policymakers: improving the relationships. Sandia report, SAND 2004-2888, U.S. Department of Energy, Sandia National Laboratories, Albuquerque, NM (2004)
Turnley, J.G., Perls, A.S.: What is a computational social model anyway? A discussion of definitions, a consideration of challenges, and an explication of process. Report No. ASCO 2008-009, Defense Threat Reduction Agency, Advanced Systems and Concepts Office, US Department of Defense, Washington DC (2008)
Konikow, L.F., Bredehoeft, J.D.: Ground-water models cannot be validated. Advances in Water Resources: Validation of Geo-hydrological Models Part 1 15(1), 75–83 (1992)
Zelikow, P., Jenkins, B.D., May, E.R.: National Commission on Terrorist Attacks upon the United States. The 9/11 Commission Report. W.W. Norton & Company, New York (2004)
Morgan, M.S., Morrison, M. (eds.): Models as Mediators: Perspectives on Natural and Social Science. Cambridge University Press, Cambridge (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Turnley, J.G., Chew, P.A., Perls, A.S. (2012). Beyond Validation: Alternative Uses and Associated Assessments of Goodness for Computational Social Models. In: Yang, S.J., Greenberg, A.M., Endsley, M. (eds) Social Computing, Behavioral - Cultural Modeling and Prediction. SBP 2012. Lecture Notes in Computer Science, vol 7227. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29047-3_18
Download citation
DOI: https://doi.org/10.1007/978-3-642-29047-3_18
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-29046-6
Online ISBN: 978-3-642-29047-3
eBook Packages: Computer ScienceComputer Science (R0)
