The Importance of Ontological Structure: Why Validation by ‘Fit-to-Data’ Is Insufficient
This chapter will briefly describe some common methods by which people make quantitative estimates of how well they expect empirical models to make predictions. However, the chapter’s main argument is that fit-to-data, the traditional yardstick for establishing confidence in models, is not quite the solid ground on which to build such belief some people think it is, especially for the kind of system agent-based modelling is usually applied to. Further, the chapter will show that the amount of data required to establish confidence in an arbitrary model by fit-to-data is often infeasible, unless there is some appropriate ‘big data’ available. This arbitrariness can be reduced by constraining the choice of model. In agent-based models, these constraints are introduced by their descriptiveness rather than by removing variables from consideration or making assumptions for the sake of simplicity. By comparing with neural networks, we show that agent-based models have a richer ontological structure. For agent-based models, in particular, this richness means that the ontological structure has a greater significance and yet is all too commonly taken for granted or assumed to be ‘common sense’. The chapter therefore also discusses some approaches to validating ontologies.
KeywordsValidation Fit-to-data Ontology Ontological structure Neural net Machine learning Calibration Generalization Model bias Variance Ockham’s razor Vapnik-Chervonenkis dimension Knowledge elicitation Interoperability Validation measures Description logic
We acknowledge funding from the Engineering and Physical Sciences Research Council (award no. 91310127), the European Commission Framework Programme 7 ‘GLAMURS’ project (grant agreement no. 613420) and the Scottish Government Rural Affairs, Food and the Environment Strategic Research Programme, Theme 2: Productive and Sustainable Land Management and Rural Economies. We are also grateful to Bruce Edmonds and Mark Brewer for useful comments on earlier drafts of this chapter; any mistakes are of course our own.
- Baader, F., & Nutt, W. (2003). Basic description logics. In F. Baader, D. Calvanese, D. L. McGuinness, D. Nardi, & P. F. Patel-Schneider (Eds.), The description logic handbook (pp. 43–95). New York, NY: Cambridge University Press.Google Scholar
- Baader, F., Küsters, R., & Wolter, F. (2003). Extensions to description logics. In F. Baader, D. Calvanese, D. L. McGuinness, D. Nardi, & P. F. Patel-Schneider (Eds.), The description logic handbook (pp. 219–261). New York, NY: Cambridge University Press.Google Scholar
- Bagosi, T., Calvanese, D., Hardi, J., Komla-Ebri, S., Lanti, D., Rezk, M., et al. (2014, August 8–12). The ontop framework for ontology based data access. In D. Zhao, J. Du, H. Wang, P. Wang, J. Donghong, & J. Z. Pan (Eds.), The semantic web and web science. 8th Chinese conference, CSWS, revised selected papers (pp. 67–77). Berlin: Springer-Verlag, Wuhan, China.Google Scholar
- Becu, N., Bousquet, F., Barreteau, O., Perez, P., & Walker, A. (2003). A methodology for eliciting and modelling stakeholders’ representations with agent based modelling. In D. Hales, B. Edmonds, E. Norling, & J. Rouchier (Eds.), Multi-Agent-Based Simulation III. MABS 2003. Lecture Notes in Computer Science 2927 (pp. 131–148). Berlin, Heidelberg: Springer.Google Scholar
- Bergman, M. (2014). 50 ontology mapping and alignment tools. http://www.mkbergman.com/1769/50-ontology-mapping-and-alignment-tools/. Accessed May 2017.
- Bharwani, S., Besa, M. C., Taylor, R., Fischer, M., Devisscher, T., & Kenfack, C. (2015). Identifying salient drivers of livelihood decision-making in the forest communities of Cameroon: Adding value to social simulation models. Journal of Artificial Societies and Social Simulation, 18(1), 3. http://jasss.soc.surrey.ac.uk/18/1/3.html. Accessed May 2017.
- Calvanese, D., & De Giacomo, G. (2003). Expressive description logics. In F. Baader, D. Calvanese, D. L. McGuinness, D. Nardi, & P. F. Patel-Schneider (Eds.), The description logic handbook (pp. 178–218). New York, NY: Cambridge University Press.Google Scholar
- Chenoweth, S. V. (1991). On the NP-hardness of blocks world. In AAAI-91 proceedings (pp. 623–628).Google Scholar
- Chester, D. L. (1990, January 15–19). Why two hidden layers are better than one. In Proceedings of the international joint conference on neural networks, (Vol. 1, pp. 265–268), Washington DC.Google Scholar
- Devlin, K. (1991). Logic and information. Cambridge, Cambridge University Press.Google Scholar
- Do, H.-H., & Rahm, E. (2002, August 20–23) COMA: A system for flexible combination of schema matching approaches. In VLDB 2002: 28th International Conference on Very Large Data Bases, Kowloon Shangri-La Hotel, Hong Kong, China. http://www.vldb.org/conf/2002/S17P03.pdf. Accessed May 2017.
- Drchal, J., Čertický, M., & Jakob, M. (2016). VALFRAM: Validation framework for activity-based models. Journal of Artificial Societies and Social Simulation, 19(3), 15. http://jasss.soc.surrey.ac.uk/19/3/15.html. Accessed May 2017.
- Edmonds, B. (2002, June 3). Simplicity is not truth-indicative. In Centre for policy modelling discussion papers CPM-02-99. http://cfpm.org/discussionpapers/111/simplicity-is-not-truth-indicative. Accessed May 2017.
- Edmonds, B., & Moss, S. (2005, July 19). From KISS to KIDS: An ‘anti-simplistic’ modelling approach. In P. Davidsson, B. Logan, & K. Takadama (Eds.), Multi-agent and multi-agent-based simulation, joint workshop MABS 2004, Revised selected papers. Lecture notes in artificial intelligence 3415 (pp. 130–114), New York, NY, USA.Google Scholar
- Elsenbroich, C. (2012). Explanation in agent-based modelling: Functions, causality or mechanisms? Journal of Artificial Societies and Social Simulation, 15(3), 1. http://jasss.soc.surrey.ac.uk/15/3/1.html. Accessed May 2017.
- Epstein, J. M. (2008). Why model? Journal of Artificial Societies and Social Simulation, 11(4), 12. http://jasss.soc.surrey.ac.uk/11/4/12.html. Accessed May 2017.
- Faria, D., Pesquita, C., Santos, E., Palmonari, M., Cruz, I. F., & Couto, F. M. (2013, September 9–13). The agreementmakerlight ontology matching system. In R. Meersman, H. Panetto, T. Dillon, J. Eder, Z. Bellahsene, N. Ritter, P. De Leenheer, & D. Dou (Eds.), On the move to meaningful internet systems: OTM 2013 conferences. Confederated international conferences CoopIS, DOA-trusted cloud, and ODBASE 2013, Proceedings. lecture notes in computer science 8185 (pp. 527–541), , Graz, Austria.Google Scholar
- Ge, J., & Polhill, J. G. (2016). Exploring the combined effect of factors influencing commuting patterns and CO2 emissions in Aberdeen using an agent-based model. Journal of Artificial Societies and Social Simulation, 19(3), 11. http://jasss.soc.surrey.ac.uk/19/3/11.html. Accessed May 2017.
- Giunchiglia, F., Autayeu, A., & Pane, J. (2012). S-match: An open source framework for matching lightweight ontologies. Semantic Web, 3(3), 307–317.Google Scholar
- Gotts, N. M., & Polhill, J. G. (2009, October 5–6). Narrative scenarios, mediating formalisms, and the agent-based simulation of land use change. In F. Squazzoni (Ed.), Epistemological aspects of computer simulation in the social sciences. Second international workshop EPOS, Revised selected and invited papers. Lecture notes in artificial intelligence 5466 (pp. 99–116), Brescia, Italy.Google Scholar
- Hertz, J., Krogh, A., & Palmer, R. G. (1991). Introduction to the theory of neural computation. Boston, MA: Addison-Wesley.Google Scholar
- Holland, J. H. (1986). Escaping brittleness: The possibilities of general-purpose learning algorithms applied to parallel rule-based systems. In R. S. Michalski, J. G. Carbonell, & T. M. Mitchell (Eds.), Machine learning: An artificial intelligence approach (Vol. II). Burlington, MA: Morgan Kaufmann.Google Scholar
- Jones, D. M., Bench-Capon, T. J. M., & Visser, P. R. S. (1998, 31 August–4 September). Methodologies for ontology development. In J. Cuena (Ed.), IT & knows: Information technologies and knowledge systems. Proceedings of a conference held as part of the XV IFIP world computer congress (pp. 62–75.), Vienna, Austria and Budapest, Hungary. http://cgi.csc.liv.ac.uk/~tbc/publications/itknows.pdf. Accessed May 2017.
- Livet, P., Muller, J.-P., Phan, D., & Sanders, L. (2010). Ontology, a mediator for agent-based modeling in social science. Journal of Artificial Societies and Social Simulation, 13(1), 3. http://jasss.soc.surrey.ac.uk/13/1/3.html. Accessed May 2017.
- Moss, S. (2008). Alternative approaches to the empirical validation of agent-based models. Journal of Artificial Societies and Social Simulation, 11(1), 5. http://jasss.soc.surrey.ac.uk/11/1/5.html. Accessed May 2017.
- Müller, J. P. (2010). A framework for integrated modeling using a knowledge-driven approach. In D. A. Swayne, W. Yang, A. A. Voinov, A. Rizzoli, & T. Filatova (Eds.), Fifth Biennial international congress on environmental modelling and software, Ottawa, Canada.http:// www.iemss.org/iemss2010/papers/S21/S.21.08.A%20framework%20foceling%20using%20a%20knowledgedriven%20approach%20-%20JEAN-PIERRE %20MULLER.pdf. Accessed May 2017.
- Ngo, D., & Bellahsene, Z. (2012, October 8–12). YAM++: A multi-strategy based approach for ontology matching task. In A. ten Teije, J. Völker, S. Handschuh, H. Stuckenschmidt, M. d’Acquin, A. Nikolov, N. Aussenac-Gilles, & N. Hernandez (Eds.), Knowledge engineering and knowledge management. 18th international conference, EKAW. Proceedings. Lecture notes in computer science 7603 (pp. 421–425), Galway City, Ireland.Google Scholar
- Object Modelling Group. (2014). Ontology definition metamodel version 1.1. In OMG Document Number: Formal/2014–09-02. http://www.omg.org/spec/ODM/1.1/PDF/. Accessed May 2017.
- Perez, P., Dray, A., Dietze, P., Moore, D., Jenkinson, R., Siokou, C., et al. (2009). An ontology-based simulation model exploring the social contexts of psychostimulant use among young Australians. International Society for the Study of Drug Policy. http://ro.uow.edu.au/smartpapers/36. Accessed May 2017.
- Polhill, J. G. (2015). Extracting OWL ontologies from agent-based models: A Netlogo extension. Journal of Artificial Societies and Social Simulation, 18(2), 15. http://jasss.soc.surrey.ac.uk/18/2/15.html. Accessed May 2017.
- Polhill, J. G., Sutherland, L.-A., & Gotts, N. M. (2010). Using qualitative evidence to enhance an agent-based modelling system for studying land use change. Journal of Artificial Societies and Social Simulation, 13(2), 10. http://jasss.soc.surrey.ac.uk/13/2/10.html. Accessed May 2017.
- Radax, W., & Rengs, B. (2010). Prospects and pitfalls of statistical testing: Insights from replicating the demographic prisoner’s dilemma. Journal of Artificial Societies and Social Simulation, 13(4), 1. http://jasss.soc.surrey.ac.uk/13/4/1.html. Accessed May 2017.
- Rossiter, S., Noble, J., & Bell, K. R. W. (2010). Social simulations: Improving interdisciplinary understanding of scientific positioning and validity. Journal of Artificial Societies and Social Simulation, 13(1), 10. http://jasss.soc.surrey.ac.uk/13/1/10.html. Accessed May 2017.
- Rumbaugh, J. (2003). Object-oriented analysis and design (OOAD). In A. Ralston, E. D. Reilly, & D. Hemmendinger (Eds.), Encyclopedia of computer science (4th ed., pp. 1275–1279). Chichester: John Wiley and Sons Ltd..Google Scholar
- Schulze, J., Müller, B., Groeneveld, J., & Grimm, V. (2017). Agent-based modelling of social-ecological systems: Achievements, challenges, and a way forward. Journal of Artificial Societies and Social Simulation, 20(2), 8. http://jasss.soc.surrey.ac.uk/20/2/8.html. Accessed May 2017.
- Shearer, R., Motik, B. and Horrocks, I. (2008, 26–27 October). HermiT: A highly-efficient OWL reasoner. In OWLED 2008. OWL: Experiences and Directions. Fifth International Workshop, Karlsruhe, Germany. http://webont.org/owled/2008/papers/owled2008eu_submission_12.pdf. Accessed May 2017.
- Sowa, J. (1999). Knowledge representation: Logical, philosophical, and computational foundations. Pacific Grove, CA: Brooks/Cole.Google Scholar
- ten Broeke, G., van Voorn, G., & Ligtenberg, A. (2016). Which sensitivity analysis method should I use for my agent-based model? Journal of Artificial Societies and Social Simulation, 19(1), 5. http://jasss.soc.surrey.ac.uk/19/1/5.html. Accessed May 2017.
- Thiele, J. C., Kurth, W., & Grimm, V. (2012). Agent-based modelling: Tools for linking NetLogo and R. Journal of Artificial Societies and Social Simulation, 15(3), 8. http://jasss.soc.surrey.ac.uk/15/3/8.html. Accessed May 2017.
- Thompson, N. S., & Derr, P. (2009). Contra Epstein, good explanations predict. Journal of Artificial Societies and Social Simulation, 12(1), 9. http://jasss.soc.surrey.ac.uk/12/1/9.html. Accessed May 2017.
- Troitzsch, K. G. (2009). Not all explanations predict satisfactorily, and not all good predictions explain. Journal of Artificial Societies and Social Simulation, 12(1), 10. http://jasss.soc.surrey.ac.uk/12/1/10.html. Accessed May 2017.
- Troitzsch, K. G. (2015). What one can learn from extracting OWL ontologies from a NetLogo model that was not designed for such an exercise. Journal of Artificial Societies and Social Simulation, 18(2), 14. http://jasss.soc.surrey.ac.uk/18/2/14.html. Accessed May 2017.
- Tsarkov, D., & Horrocks, I. (2006, August 17–20). FaCT++ description logic reasoner: System description. In U. Furbach & N. Shankar (Eds.), Automated reasoning. Third international joint conference, IJCAR 2006. Proceedings. Lecture notes in computer science 4130 (pp. 292–297), Seattle, WA, USA.Google Scholar
- Windrum, P., Fagiolo, G., & Moneta, A. (2007) Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulation 10(2), 8. http://jasss.soc.surrey.ac.uk/10/2/8.html. Accessed May 2017.
- Winograd, T. (1972). Understanding natural language. Edinburgh: Edinburgh University Press.Google Scholar
- Wilensky, U. (1999). NetLogo. Center for connected learning and computer-based modeling. Evanston, IL: Northwestern University. http://ccl.northwestern.edu/netlogo. Accessed May 2017