Identifying Unexpected Behaviors of Agent-Based Models Through Spatial Plots and Heat Maps

Part of the Understanding Complex Systems book series (UCS)


Verification and validation (V&V) techniques commonly require modelers to collect and statistically analyze large amounts of data which require specific methods for ordering, filtering, or converting data points. Modelers need simple, intuitive, and efficient techniques for gaining insight into unexpected behaviors to help in determining if these behaviors are errors or if they are artifacts resulting from the model’s specifications. We present an approach to begin addressing this need by applying heat maps and spatial plots to visually observe unexpected behaviors within agent-based models. Our approach requires the modeler to specify hypotheses about expected model behavior. Agent level outputs of interest are then used to create graphical displays to visually test the hypotheses. Visual identification of unexpected behaviors can direct focus for additional V&V efforts and inform the selection process of follow-on V&V techniques. We apply our approach to a model of obesity.


Verification and validation Heat map Spatial plot Visualization Agent-based models 


  1. 1.
    Axelrod, R.: Advancing the art of simulation in the social sciences. Complexity 3(2), 16–22 (1997)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Balci, O.: Verification, Calibration, and Testing, book section 10, pp. 335–393. John Wiley and Sons, Inc., New York, NY (1998)Google Scholar
  3. 3.
    Bonabeau, E.: Agent-based modeling: methods and techniques for simulating human systems. Proc. Natl. Acad. Sci. U.S.A. 99(Suppl 3), 7280–7287 (2002)CrossRefGoogle Scholar
  4. 4.
    Chi, E.H.H.: A taxonomy of visualization techniques using the data state reference model. In: IEEE Symposium on Information Visualization, pp. 69–75. IEEE, Piscataway, NJ (2000)Google Scholar
  5. 5.
    Collins, A.J., Knowles Ball, D., Romberger, J.: A discussion on simulations’ visualization usage. In: Yilmaz, L., Chan, W., Moon, I., Roeder, T., Macal, C., Rossetti, M. (eds.) Proceedings of the 2015 Winter Simulation Conference, pp. 2827–2835. IEEE, Piscataway, NJ (2015)Google Scholar
  6. 6.
    Courdier, R., Guerrin, F., Andriamasinoro, F.H., Paillat, J.M.: Agent-based simulation of complex systems: application to collective management of animal wastes. J. Artif. Soc. and Soc. Simul. 5(3), 1–17 (2002)Google Scholar
  7. 7.
    Deuro, J., Lynch, C.J., Kavak, H., Padilla, J.J.: Incorporating sound in simulations. In: Chan, W., D’Ambrogio, A., Zacharewicz, G., Mustafee, N., Wainer, G. (eds.) Proceedings of the 2017 Winter Simulation Conference (2017)Google Scholar
  8. 8.
    Diallo, S.Y., Gore, R., Lynch, C.J., Padilla, J.J.: Formal methods, statistical debugging and exploratory analysis in support of system development: towards a verification and validation calculator tool. Int. J. Model. Simul. Sci. Comput. 7(1), 1–22 (2016)CrossRefGoogle Scholar
  9. 9.
    Epstein, J.M., Axtell, R.: Growing Artificial Societies: Social Science from the Bottom Up. The MIT Press, Cambridge (1996)CrossRefGoogle Scholar
  10. 10.
    Forrester, J.W.: Industrial Dynamics. The MIT Press, Cambridge (1961)Google Scholar
  11. 11.
    Gore, R., Lynch, C.J., Kavak, H.: Applying statistical debugging for enhanced trace validation of agent-based models. Simul.: Trans. Soc. Model. Simul. Int. 93(4), 273–284 (2016)CrossRefGoogle Scholar
  12. 12.
    Harris, J.A., Benedict, F.G.: A biometric study of human basal metabolism. Proc. Natl. Acad. Sci. U.S.A. 4(12), 370–373 (1918)CrossRefGoogle Scholar
  13. 13.
    Keim, D.A.: Information visualization and visual data mining. IEEE Trans. Vis. Comput. Graph. 8(1), 1–8 (2002)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Kemper, P., Tepper, C.: Trace based analysis of process interaction models. In: Kuhl, M.E., Steiger, N.M., Armstrong, F.B., Joines, J.A. (eds.) Proceedings of the 2005 Winter Simulation Conference, pp. 427–436. IEEE, Piscataway, NJ (2005)Google Scholar
  15. 15.
    Kornhauser, D., Wilensky, U., Rand, W.: Design guidelines for agent based model visualization. J. Artif. Soc. Soc. Simul. 12(2), 1–27 (2009)Google Scholar
  16. 16.
    Liu, S., Cui, W., Wu, Y., Liu, M.: A survey on information visualization: recent advances and challenges. Vis. Comput. 30(12), 1373–1393 (2014)CrossRefGoogle Scholar
  17. 17.
    Lynch, C.J.: A Multi-Paradigm Modeling Framework for Modeling and Simulating Problem Situations. Master’s thesis, Old Dominion University (2012)Google Scholar
  18. 18.
    Lynch, C.J., Diallo, S.Y.: A taxonomy for classifying terminologies that describe simulations with multiple models. In: Yilmaz, L., Chan, W.K.V., Moon, I., Roeder, T.M.K., Macal, C., Rossetti, M.D. (eds.) Proceedings of the 2015 Winter Simulation Conference, pp. 1621–1632. IEEE, Piscataway, NJ (2015)Google Scholar
  19. 19.
    Lynch, C.J., Padilla, J.J., Diallo, S.Y., Sokolowski, J.A., Banks, C.M.: A multi-paradigm modeling framework for modeling and simulating problem situations. In: Tolk, A., Diallo, S., Ryzhov, I., Yilmaz, L., Buckley, S., Miller, J. (eds.) Proceedings of the 2014 Winter Simulation Conference, pp. 1688–1699. IEEE, Piscataway, NJ (2014)Google Scholar
  20. 20.
    Morland, K., Wing, S., Roux, A.D., Poole, C.: Neighborhood characteristics associated with the location of food stores and food service places. Am. J. Prev. Med. 22(1), 23–29 (2002)CrossRefGoogle Scholar
  21. 21.
    Padilla, J.J., Diallo, S.Y., Lynch, C.J., Gore, R.: Observations on the practice and profession of modelling and simulation: a survey approach. Simul.: Trans. Soc. Model. Simul. Int. 1–28 (2017)Google Scholar
  22. 22.
    Powell, L.M., Chaloupka, F.J., Bao, Y.: The availability of fast-food and full-service restaurants in the united states: associations with neighborhood characteristics. Am. J. Prev. Med. 33(4), s240–s245 (2007)CrossRefGoogle Scholar
  23. 23.
    Rohrer, M.W.: Seeing is believing: the importance of visualization in manufacturing simulation. In: Joines, J.A., Barton, R.R., Kang, K., Fishwick, P.A. (eds.) Proceedings of the 2000 Winter Simulation Conference, pp. 1211–1216. IEEE, Piscataway, NJ (2000)Google Scholar
  24. 24.
    Roza, A.M., Shizgal, H.M.: The harris benedict equation reevaluated: resting energy requirements and the body cell mass. Am. J. Clin. Nutr. 40(1), 168–182 (1984)CrossRefGoogle Scholar
  25. 25.
    Sargent, R.G.: Some subjective validation methods using graphical displays of data. In: Charnes, J.M., Morrice, D.J., Brunner, D.T., Swain, J.J. (eds.) Proceedings of the 1996 Winter Simulation Conference, pp. 345–351. IEEE, Piscataway, NJ (1996)Google Scholar
  26. 26.
    Smith, M.I., Murray-Smith, D.J., Hickman, D.: Verification and validation issues in a generic model of electro-optic sensor systems. J. Def. Model. Simul. 4(1), 17–27 (2007). Scholar
  27. 27.
    Soban, D.S., Salmon, J., Fahringer, P.: A visual analytics framework for strategic airlift decision making. J. Def. Model. Simul. 10(2), 131–144 (2013). Scholar
  28. 28.
    Sokolowski, J.A., Banks, C.M., Diallo, S., Padilla, J.J., Lynch, C.J.: A methodology for engaging modeling and simulation to assess a corollary problem to the obesity epidemic (2012)Google Scholar
  29. 29.
    Sokolowski, J.A., Garcia, H.M.: The significance of modeling and visualization, pp. 33–48. Wiley, New York (2016)Google Scholar
  30. 30.
    Takadama, K., Kawai, T., Koyama, Y.: Micro-and macro-level validation in agent-based simulation: reproduction of human-like behaviors and thinking in a sequential bargaining game. J. Artif. Soc. Soc. Simul. 11(2), 1–17 (2008)Google Scholar
  31. 31.
    Tolk, A., Diallo, S.Y., Padilla, J.J., Herencia-Zapana, H.: Reference modelling in support of m&s foundations and applications. J. Simul. 7(2), 69–82 (2013)CrossRefGoogle Scholar
  32. 32.
    Tufte, E.R., Graves-Morris, P.R.: The Visual Display of Quantitative Information, vol. 2. Graphics Press, Cheshire (1983)Google Scholar
  33. 33.
    Vernon-Bido, D., Collins, A., Sokolowski, J.: Effective visualization in modeling and simulation. In: Proceedings of the 2015 Spring Simulation Multi-Conference - Annual Simulation Symposium, pp. 33–40. Society for Computer Simulation International, San Diego, CA (2015)Google Scholar
  34. 34.
    Wenzel, S., Bernhard, J., Jessen, U.: Visualization for modeling and simulation: A taxonomy of visualization techniques for simulation in production and logistics. In: Chick, S., Snchez, P.J., Ferrin, D., Morrice, D.J. (eds.) Proceedings of the 2003 Winter Simulation Conference, pp. 729–736. IEEE, Piscataway, NJ (2003)Google Scholar
  35. 35.
    Wilkinson, L., Friendly, M.: The history of the cluster heat map. Am. Stat. 63(2), 179–184 (2009)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Xiang, X., Kennedy, R., Madey, G., Cabaniss, S.: Verification and validation of agent-based scientific simulation models. In: Proceedings of the 2005 Spring Simulation Multi-Conference - Agent-Directed Simulation Symposium, pp. 47–55. Society for Computer Simulation International, San Diego, CA (2005)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Virginia Modeling, Analysis and Simulation CenterOld Dominion UniversitySuffolkUSA

Personalised recommendations