Skip to main content
SpringerLink
Log in

A framework for the comparison of agent-based models

  • Published:
Autonomous Agents and Multi-Agent Systems Aims and scope Submit manuscript

Abstract

We develop a methodology for comparing agent-based models that are developed for the same domain, but may differ in the data sets (e.g., geographical regions) to which they are applied, and in the structure of the model. Our approach is to learn a response surface in the common parameter space of the models and compare the regions corresponding to qualitatively different behaviors in the models. As an example, we develop an active learning algorithm to learn phase shift boundaries in contagion processes in order to compare two agent-based models of rooftop solar panel adoption developed for different regions. We present results for 2D and 3D subspaces of the parameter space, though the approach scales to higher dimensions as well.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://www.gosolarcalifornia.ca.gov/about/csi.php

  2. https://github.com/haifeng-zhang/ddabm-solar

References

  1. Adiga, A., Barrett, C., Eubank, S., Kuhlman, C.J., Marathe, M. V., Mortveit, H., Ravi, S. S., Rosenkrantz, D. J., Stearns, R. E., Swarup, S. & Vullikanti, A. (2019). Validating agent-based models of large networked systems. In: Winter simulation conference (WSC)

  2. Axtell, R., Axelrod, R., Epstein, J. M., & Cohen, M. D. (1996). Aligning simulation models: A case study and results. Computational and Mathematical Organization Theory, 1(2), 123â141.

    Article  Google Scholar 

  3. Barde, S. & van der Hoog, S. (2017). An empirical validation protocol for large-scale agent-based models. In: Bielefeld working papers in economics and management No. 04-2017. https://doi.org/10.2139/ssrn.2992473

  4. Barton, R. R. & Meckesheimer, M. (22006). Metamodel-based simulation optimization. In: S. G. Henderson, B. L. Nelson (eds.) Simulation, Handbooks in operations research and management science, Elsevier, vol. 13, pp. 535 â 574

  5. Bharathy, G. K. & Silverman, B. (2010). Validating agent based social systems models. In: Proceedings of the winter simulation conference, WSC ’10, pp. 441â453. Winter simulation conference. http://dl.acm.org/citation.cfm?id=2433508.2433559

  6. Box, G. E. P., & Wilson, K. B. (1951). On the experimental attainment of optimum conditions. Journal of the Royal Statistical Society Series B (Methodological), 13(1), 1â4.

    Article  MathSciNet  MATH  Google Scholar 

  7. Brueckner, S. A. & Parunak, H. V. D. (2003). Resource-aware exploration of the emergent dynamics of simulated systems. In: Proceedings of the international conference on autonomous agents and multi-agents systems (AAMAS). Melbourne, Australia

  8. Burton, R. M. & Obel, B. (1999). The challenge of validation and docking. In: Proceedings of the workshop on agent simulation: Applications, models, and tools. Argonne National Laboratory, pp. 216â221.

  9. Carley, K. M., Kamneva, N. Y., & Reminga, J. (2004). Response surface methodology. CASOS Technical report CMU-ISRI-04-136, Carnegie Mellon University

  10. Chang, K. H., Hong, L. J., & Wan, H. (2013). Stochastic trust-region response-surface method (strong)-a new response-surface framework for simulation optimization. INFORMS Journal on Computing, 25(2), 230â243. https://doi.org/10.1287/ijoc.1120.0498

    Article  MathSciNet  Google Scholar 

  11. Collier, N. (2003). Repast: An extensible framework for agent simulation. The University of Chicagoâs Social Science Research, 36, 2003.

    Google Scholar 

  12. Fadikar, A., Higdon, D., Chen, J., Lewis, B. L., Venkatramanan, S., & Marathe, M. V. (2018). Calibrating a stochastic agent-based model using quantile-based emulation. SIAM/ASA J. Uncertainty Quantification, 6(4), 1685â1706.

    Article  MathSciNet  MATH  Google Scholar 

  13. Forrest, S. & Uri, W. (2011) Finding forms of flocking: Evolutionary search in ABM parameter-spaces. Multi-agent-based simulation XI. MABS 2010. Lecture Notes in Computer Science 6532 (2011). https://doi.org/10.1007/978-3-642-18345-4_5

  14. Forrester, A. I. J., Sobester, A., & Keane, A. J. (2008). Engineering design via surrogate modellingâA practical guide. Wiley

  15. Grimm, V., Berger, U., DeAngelis, D., Polhill, J., Giske, J., & Railsback, S. (2010). The odd protocol: A review and first update. Ecological Modelling, 221(23), 2760â2768. https://doi.org/10.1016/j.ecolmodel.2010.08.019

    Article  Google Scholar 

  16. Gupta, A., Hu, Z., Marathe, A., Swarup, S., & Vullikanti, A.: Predictors of rooftop solar adoption in rural Virginia. In: Proceedings of the computational social science conference (2018)

  17. Gupta, A., Swarup, S., Marathe, A., Vullikanti, A., Lakkaraju, K., & Letchford, J. (2018). Designing incentives to maximize the adoption of rooftop solar technology (extended abstract). In: Proceedings of the 17th international conference on autonomous agents and multi-agents systems (AAMAS). Stockholm, Sweden (2018)

  18. Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of statistical l. Springer series in statistics. Springer, New York, NY, USA

  19. Hu, Z., Deng, X., Marathe, A., Swarup, S., & Vullikanti, A. (2019). Decision-adjusted modeling for imbalanced classification: Predicting rooftop solar panel adoption in rural virginia. In: Proceedings of the computational social science conference

  20. Kempe, D., Kleinberg, J., & Ãva, T. (2003). Maximizing the spread of influence through a social network. In: Proceedings of KDD. Washington, DC, USA

  21. Lamperti, F., Roventini, A., & Sani, A. (2018). Agent-based model calibration using machine learning surrogates. Journal of Economic Dynamic and Control, 90, 366â389.

    Article  MathSciNet  MATH  Google Scholar 

  22. Lewis, D. D., & Gale, W. A. (1994). A sequential algorithm for training text classifiers. In B. W. Croft & C. J. van Rijsbergen (Eds.), SIGIR â94 (pp. 3â12). London: Springer.

    Chapter  Google Scholar 

  23. Lookman, T., Balachandran, P. V., Xue, D., & Yuan, R. (2009) Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design. In: npj computational materials. https://doi.org/10.1038/s41524-019-0153-8

  24. Louie, M. A., Carley, K. M., Haghshenass, L., Kunz, J. C., Levitt, R. E., et al. (2003). Model comparisons: docking orgahead and simvision. In: Proceedings of NAACSOS conference, Pittsburgh, PA. Citeseer

  25. Mao, H., Guo, F., Deng, X., & Doerzaph, Z. (2019). Decision-adjusted driver risk predictive model using kinematics information. submitted to IEEE transactions on intelligent transportation systems

  26. Marathe, M., & Vullikanti, A. (2013). Computational epidemiology. Communications of the ACM, 56(7), 88â96.

    Article  Google Scholar 

  27. Minar, N., Burkhart, R., Langton, C., Askenazi, M., et al.: The swarm simulation system: A toolkit for building multi-agent simulations. Technical report (1996)

  28. Mortveit, H., & Reidys, C. (2007). An introduction to sequential dynamical systems. Springer: Universitext.

    MATH  Google Scholar 

  29. Murphy, R. F. (2011). An active role for machine learning in drug development. In: Nature chemical biology, p. 327â330. https://doi.org/10.1038/nchembio.576

  30. Neddermeijer, H. G., van Oortmarssen, G. J., Piersma, N., & Dekker, R. (2000). A framework for response surface methodology for simulation optimization. In: J. A. Joines, R. R. Barton, K. Kang, P. A. Fishwick (eds.) Proceedings of the 32nd conference on winter simulation, WSC ’00, pp. 129â136. Society for Computer Simulation International, San Diego, CA, USA. http://dl.acm.org/citation.cfm?id=510378.510401

  31. North, M., & Macal, C. M. (2002). The beer dock: Three and a half implementations of the beer distribution game. Swarm Development Group: SwarmFest.

    Google Scholar 

  32. Pérez, V. M., Renaud, J. E., & Watson, L. T. (2002). Adaptive experimental design for construction of response surface approximations. AIAA Journal, 40(12), 2495â2503.

    Article  Google Scholar 

  33. Scheffer, T., Decomain, C., & Wrobel, S. (2001). Active hidden markov models for information extraction. In: Proceedings of the 4th international conference on advances in intelligent data analysis, IDA ’01, p. 309â318. Springer, Berlin, Heidelberg

  34. Thorve, S., Hu, Z., Lakkaraju, K., Letchford, J., Vullikanti, A., Marathe, A., & Swarup, S. (2020). An active learning method for the comparison of agent-based models. In: Proceedings of the 19th international conference on autonomous agents and multi-agent systems (AAMAS)

  35. Volk-Makarewicz, W. & Cleophas, C. (2017) A meta-algorithm for validating agent-based simulation models to support decision making. In: Proceedings of the 2017 winter simulation conference, WSC ’17, pp. 102:1â102:12. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=3242181.3242289

  36. Xu, J., Gao, Y., & Madey, G. (2003) A docking experiment: Swarm and repast for social network modeling. In: Seventh annual swarm researchers meeting (Swarm2003), pp. 1â9

  37. Zhang, H., Vorobeychik, Y., Letchford, J., & Lakkaraju, K. (2016). Data-driven agent-based modeling, with application to rooftop solar adoption. Auton Agent Multi-Agent Syst, 30(6), 1023â1049. https://doi.org/10.1007/s10458-016-9326-8

    Article  Google Scholar 

  38. Zhang, Y., Li, Z., & Zhang, Y. (2020). Validation and calibration of an agent-based model: A surrogate approach. In: Discrete dynamics in nature and society. https://doi.org/10.1155/2020/6946370

  39. Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301â320. https://doi.org/10.1111/j.1467-9868.2005.00503.x.

Download references

Acknowledgements

This work was supported in part by DOE grant DE-EE0007660, NIH grant R01GM109718, NSF CRISP 2.0 grant 1832587. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energyâs National Nuclear Security Administration under contract DE-NA0003525.

Author information

Authors and Affiliations

  1. University of Virginia, Virginia, USA

    Swapna Thorve, Anil Vullikanti, Achla Marathe & Samarth Swarup

  2. Virginia Tech, Virginia, USA

    Zhihao Hu

  3. Sandia National Lab, Neq Mexico, USA

    Kiran Lakkaraju & Joshua Letchford

Authors
  1. Swapna Thorve
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhihao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kiran Lakkaraju
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joshua Letchford
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anil Vullikanti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Achla Marathe
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Samarth Swarup
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Swapna Thorve.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This paper is an extended version of our paper published in AAMAS 2020 [34]

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Thorve, S., Hu, Z., Lakkaraju, K. et al. A framework for the comparison of agent-based models. Auton Agent Multi-Agent Syst 36, 32 (2022). https://doi.org/10.1007/s10458-022-09559-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10458-022-09559-5

Keywords

Search

Navigation