Skip to main content

Beyond Validation: Alternative Uses and Associated Assessments of Goodness for Computational Social Models

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNISA,volume 7227)

Abstract

This discussion challenges classic notions of validation, suggesting that ‘validity’ is not just an attribute of a model. It is a function of the relationship of a particular characteristic of the model (the probability that the model will produce a data set that will match a set it has not yet seen) and a user. If we shift our focus from the model to the user, we can identify other ways in which models can be of use as well as start clarifying how we can identify their goodness in multiple use scenarios. In this discussion, we distinguish validation from calibration, and research or generalizable models from site-specific models. We use literature from fields as varied as physics, systems ecology and computational linguistics to characterize the process of validation. Finally, by extending the use space for models beyond prediction, we introduce the possibility of assessments of goodness other than validation.

Keywords

  • Social modeling
  • validation
  • computational models

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. U.S. Department of Defense: Instruction No. 5000.61 (December 9, 2009)

    Google Scholar 

  2. American Institute of Aeronautics and Astronautics, AIAA guide for the verification and validation of computational fluid dynamics simulations. G-077-1998e (1998)

    Google Scholar 

  3. U.S. Department of Commerce, National Institute of Standards and Technology: State Weights and Measures Laboratories Program Handbook. NIST Handbook 143 (2006), http://www.ncbi.nlm.nih.gov (accessed)

  4. Beven, K.: Towards a coherent philosophy for modeling the environment. Proceedings of the Royal Society of London 458, 2465–2484 (2002)

    CrossRef  MathSciNet  MATH  Google Scholar 

  5. McDonnell, J.J., Sivapalan, M., Vaché, K., Dunn, S., Grant, G., Haggerty, R., Hinz, C., Hooper, R., Kirchner, J., Roderick, M.L., Selker, J., Weiler, M.: Moving beyond heterogeneity and process complexity: A new vision for watershed hydrology. Water Resources Research 43, W07301 (2007), doi:10.1029/2006WR005467

    CrossRef  Google Scholar 

  6. Vogel, R.M., Sankarasubramanian, A.: Validation of a watershed model without calibration. Water Resources Research 39(10), 1292–1300 (2003)

    CrossRef  Google Scholar 

  7. Refsgaard, J.C., Henriksen, H.J.: Modeling guidelines – terminology and guiding principles. Advances in Water Resources 27, 71–82 (2004)

    CrossRef  Google Scholar 

  8. Oberkampf, W.L., Trucano, T.G.: Verification and validation in computational fluid dynamics. Progress in Aerospace Sciences 38(3), 209–272 (2002)

    CrossRef  Google Scholar 

  9. Oberkampf, W. L., Trucano, T. G.: Verification and validation benchmarks. Sandia report, SAND 2007-0853, U.S. Department of Energy, Sandia National Laboratories, Albuquerque, NM (2007)

    Google Scholar 

  10. Oberkampf, W. L., Barone, M. F.: Measures of agreement between computation and experiment: validation measures. Sandia report, SAND 2005-4302, U.S. Department of Energy, Sandia National Laboratories, Albuquerque, NM (2005)

    Google Scholar 

  11. Oberkampf, W. L., Trucano, T. G., Hirsch, C.: Verification, validation, and predictive capability in computational engineering and physics. Sandia report, SAND 2003-3769, U.S. Department of Energy, Sandia National Laboratories, Albuquerque, NM (2003)

    Google Scholar 

  12. McNamara, L.: Why models don’t forecast. Draft Paper Prepared for the Workshop Unifying Social Frameworks, August 16-17. National Research Council, Washington, DC (2010), http://www7.nationalacademies.org/bbcss/Why%20Models%20Dont%20Forecast-McNamara.pdf

    Google Scholar 

  13. Atkinson, R., Shiffrin, R.M.: Human memory: A proposed system and its control processes. In: Spence, K.W. (ed.) The Psychology of Learning and Motivation: Advances in Research and Theory, vol. 2, pp. 89–195. Academic Press, New York (1968)

    Google Scholar 

  14. Dewey, J.: How we think. FQ Books, New York (1910, 2010)

    Google Scholar 

  15. Ormrod, J.E.: Educational psychology: developing learners, 4th edn. Merrill Prentice Hall, Upper Saddle River (2003)

    Google Scholar 

  16. Bruner, J., Goodnow, J., Austin, A.: A study of thinking. Wiley, New York (1956)

    Google Scholar 

  17. Piaget, J.: The Psychology of Intelligence. Routledge, New York (1950)

    Google Scholar 

  18. Richardson, V.: Constructivist pedagogy. Teachers College Record 105(9), 1623–1640 (2003)

    CrossRef  Google Scholar 

  19. Horst, W., Rittel, J., Webber, M.M.: Dilemmas in a general theory of planning. Policy Sciences 4, 155–169 (1973)

    CrossRef  Google Scholar 

  20. Holland, J.: Complex adaptive systems. Daedalus 121(1), 17–30 (1992)

    Google Scholar 

  21. Beemer, B., Gregg, D.G.: Advisory systems to support decision making. In: Handbook on Decision Support Systems. Springer, London (2008)

    Google Scholar 

  22. Sniezek, J.A.: Judge Advisor Systems Theory and Research and Applications to Collaborative Systems and Technology. In: Proceedings of the 32nd Hawaii International Conference on System Sciences, pp. 1057–1066. IEEE (1999)

    Google Scholar 

  23. Shieber, S.: Criteria for designing computer facilities for linguistic analysis. Linguistics 23, 189–211 (1985)

    CrossRef  Google Scholar 

  24. Janssen, M.A., Ostrom, E.: Empirically based, agent-based models. Ecology and Society 11(2) (2006), http://www.ecologyandsociety.org/vol11/iss2/art37/ (accessed)

  25. Karas, T.: Modelers and policymakers: improving the relationships. Sandia report, SAND 2004-2888, U.S. Department of Energy, Sandia National Laboratories, Albuquerque, NM (2004)

    Google Scholar 

  26. Turnley, J.G., Perls, A.S.: What is a computational social model anyway? A discussion of definitions, a consideration of challenges, and an explication of process. Report No. ASCO 2008-009, Defense Threat Reduction Agency, Advanced Systems and Concepts Office, US Department of Defense, Washington DC (2008)

    Google Scholar 

  27. Konikow, L.F., Bredehoeft, J.D.: Ground-water models cannot be validated. Advances in Water Resources: Validation of Geo-hydrological Models Part 1 15(1), 75–83 (1992)

    Google Scholar 

  28. Zelikow, P., Jenkins, B.D., May, E.R.: National Commission on Terrorist Attacks upon the United States. The 9/11 Commission Report. W.W. Norton & Company, New York (2004)

    Google Scholar 

  29. Morgan, M.S., Morrison, M. (eds.): Models as Mediators: Perspectives on Natural and Social Science. Cambridge University Press, Cambridge (1999)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Turnley, J.G., Chew, P.A., Perls, A.S. (2012). Beyond Validation: Alternative Uses and Associated Assessments of Goodness for Computational Social Models. In: Yang, S.J., Greenberg, A.M., Endsley, M. (eds) Social Computing, Behavioral - Cultural Modeling and Prediction. SBP 2012. Lecture Notes in Computer Science, vol 7227. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29047-3_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-29047-3_18

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-29046-6

  • Online ISBN: 978-3-642-29047-3

  • eBook Packages: Computer ScienceComputer Science (R0)