Explanation in Human-Agent Teamwork

  • Maaike Harbers
  • Jeffrey M. Bradshaw
  • Matthew Johnson
  • Paul Feltovich
  • Karel van den Bosch
  • John-Jules Meyer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7254)


There are several applications in which humans and agents jointly perform a task. If the task involves interdependence among the team members, coordination is required to achieve good team performance. This paper discusses the role of explanation in coordination in human-agent teams. Explanations about agent behavior for humans can improve coordination in human-agent teams for two reasons. First, with more knowledge about an agent’s actions and plans, humans can more easily adapt their own behavior to that of the agent. Second, with more insight in the reasons behind an agent’s behavior, humans will have more trust in the agents, and therefore more easily coordinate their actions. The paper also presents a study in the BW4T testbed that examines the effects of agents explaining their behavior on human-agent team performance. The results of this study show that explanations about agent behavior do not always lead to better team performance, but they do impact the user experience in a positive way.


Team Member Mental Model Recommender System Team Performance Agent Behavior 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Austin, J.: Transactive memory in organizational groups: the effects of content, consensus, specialization and accuracy on group performance. Journal of Applied Psychology 88(5), 866–878 (2003)CrossRefGoogle Scholar
  2. 2.
    Bordini, R.H., Dastani, M., El Fallah Seghrouchni, A. (eds.): Multi-Agent Programming: Languages, Tools and Applications. Springer (2009)Google Scholar
  3. 3.
    Bradshaw, J.M., Feltovich, P., Johnson, M.: Human-Agent Interaction. In: Handbook of Human-Machine Interaction, pp. 283–302. Ashgate (2011)Google Scholar
  4. 4.
    Bradshaw, J.M., et al.: Representation and reasoning about DAML-based policy and domain services in KAoS. In: Proceedings of AAMAS 2003. ACM Press (2003)Google Scholar
  5. 5.
    Bratman, M.: Intention, Plans and Practical Reason. Harvard University Press, Cambridge (1987)Google Scholar
  6. 6.
    Cannon-Bowers, J.A., Salas, E., Converse, S.: Shared mental models in expert team decision making. Individual and Group Decision Making Current Issues 39(3-4), 221–246 (1993)Google Scholar
  7. 7.
    Cooke, N.J., Salas, E., Cannon-Bowers, J.A., Stout, R.J.: Measuring team knowledge. Human Factors 42(1), 151–173 (2000)CrossRefGoogle Scholar
  8. 8.
    Core, M., Traum, T., Lane, H., Swartout, W., Gratch, J., Van Lent, M.: Teaching negotiation skills through practice and reflection with virtual humans. Simulation 82(11), 685–701 (2006)CrossRefGoogle Scholar
  9. 9.
    Dennett, D.: The Intentional Stance. MIT Press (1987)Google Scholar
  10. 10.
    Dhaliwal, J., Benbasat, I.: The use and effects of knowledge-based system explanations: theoretical foundations and a framework for empirical evaluation. Information Systems Research 7(6), 243–361 (1996)Google Scholar
  11. 11.
    Gregor, S., Benbasat, I.: Explanation from intelligent systems: theoretical foundations and implications for practice. MIS Quarterly 23(4), 497–530 (1999)CrossRefGoogle Scholar
  12. 12.
    Harbers, M., Van den Bosch, K., Meyer, J.-J.: Design and evaluation of explainable agents. In: Proceedings of IAT 2010 (2010)Google Scholar
  13. 13.
    Hindriks, K.: Programming Rational Agents in GOAL. In: Multi-Agent Programming: Languages, Tools and Applications, pp. 119–157. Springer (2009)Google Scholar
  14. 14.
    Janis, I.L.: Victims of Groupthink. Houghton Mifflin, Boston (1972)Google Scholar
  15. 15.
    Johnson, L.: Agents that learn to explain themselves. In: Proc. of the 12th Nat. Conf. on Artificial Intelligence, pp. 1257–1263 (1994)Google Scholar
  16. 16.
    Johnson, M., Bradshaw, J.M., Feltovich, P.J., Jonker, C.M., van Riemsdijk, B., Sierhuis, M.: The Fundamental Principle of Coactive Design: Interdependence Must Shape Autonomy. In: De Vos, M., Fornara, N., Pitt, J.V., Vouros, G. (eds.) COIN 2010 International Workshops. LNCS, vol. 6541, pp. 172–191. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  17. 17.
    Johnson, M., Bradshaw, J.M., Feltovich, P.J., Hoffman, R.R., Jonker, C., van Riemsdijk, B., Sierhuis, M.: Beyond cooperative robotics: The central role of interdependence in coactive design. IEEE Intelligent Systems 26, 81–88 (2011)CrossRefGoogle Scholar
  18. 18.
    Johnson, M., Jonker, C., van Riemsdijk, B., Feltovich, P.J., Bradshaw, J.M.: Joint Activity Testbed: Blocks World for Teams (BW4T). In: Aldewereld, H., Dignum, V., Picard, G. (eds.) ESAW 2009. LNCS, vol. 5881, pp. 254–256. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  19. 19.
    Jonker, C.M., van Riemsdijk, M.B., Vermeulen, B.: Shared Mental Models - A Conceptual Analysis. In: De Vos, M., Fornara, N., Pitt, J.V., Vouros, G. (eds.) COIN 2010 International Workshops. LNCS, vol. 6541, pp. 132–151. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  20. 20.
    Keil, F.: Explanation and understanding. Annual Reviews Psychology 57, 227–254 (2006)CrossRefGoogle Scholar
  21. 21.
    Lewis, K.: Measuring transactive memory systems in the field: Scale development and validation. Journal of Applied Psychology 88(4), 587–604 (2003)CrossRefGoogle Scholar
  22. 22.
    Malle, B.: How people explain behavior: A new theoretical framework. Personality and Social Psychology Review 3(1), 23–48 (1999)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Mohammed, S., Klimoski, R., Rentsch, J.R.: The measurement of team mental models: We have no shared schema. Organizational Research Methods 3(2), 123–165 (2000)CrossRefGoogle Scholar
  24. 24.
    Moreland, R.L., Myaskovsky, L.: Exploring the performance benefits of group training: Transactive memory or improved communication? Organizational Behavior and Human Decision Processes 82(1), 117–133 (2000)CrossRefGoogle Scholar
  25. 25.
    Nandkeolyar, A.K.: How do teams learn? shared mental models and transactive memory systems as determinants of team learning and effectiveness. PhD thesis, University of Iowa (2008)Google Scholar
  26. 26.
    Rao, A., Georgeff, M.: BDI-agents: From theory to practice. In: Proceedings of ICMAS 1995 (1995)Google Scholar
  27. 27.
    Rau, D.: Top management team transactive memory, information gathering, and perceptual accuracy. Journal of Business Research 59(4), 416–424 (2006)CrossRefGoogle Scholar
  28. 28.
    Rouse, W., Morris, N.M.: On looking into the black box: Prospects and limits in the search for mental models. Psychological Bulletin 100(3), 349–363 (1984)CrossRefGoogle Scholar
  29. 29.
    Sheridan, T.B., Verplank, W.L.: Human and computer control of undersea teleoperators. Technical report, Man-Machine Systems Laboratory, Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts (1978)Google Scholar
  30. 30.
    Swartout, W., Moore, J.: Explanation in Second-Generation Expert Systems. In: Second-Generation Expert Systems, pp. 543–585. Springer, New York (1993)CrossRefGoogle Scholar
  31. 31.
    Sycara, K., Lewis, M.: Integrating intelligent agents into human teams. In: Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (2002)Google Scholar
  32. 32.
    Tintarev, N., Masthoff, J.: A survey of explanations in recommender systems. In: Proceeding of the International Conference on Data Engineering Workshop. IEEE Computer Society, Washington, DC (2007)Google Scholar
  33. 33.
    van Diggelen, J., Beun, R.-J., Werkhoven, P.J.: Intelligent assistants in crisis management: from PDA to TDA. In: Proceedings of BNAIC 2009 (2009)Google Scholar
  34. 34.
    Van Lent, M., Fisher, W., Mancuso, M.: An explainable artificial intelligence system for small-unit tactical behavior. In: Proc. of IAAA 2004. AAAI Press, Menlo Park (2004)Google Scholar
  35. 35.
    Wegner, D.M.: Transactive memory: A contemporary analysis of the group mind. In: Mullen, B., Goethals, G.R. (eds.) Theories of Group Behavior, pp. 185–208 (1987)Google Scholar
  36. 36.
    Ye, R., Johnson, P.: The impact of explanation facilities on user acceptance of expert systems advice. MIS Quarterly 19(2), 157–172 (1995)CrossRefGoogle Scholar
  37. 37.
    Yen, J., Fan, X., Sun, S., Hanratty, T., Dumer, J.: Agents with shared mental models for enhancing team decision-makings. Deision Support Systems 41, 634–653 (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Maaike Harbers
    • 1
  • Jeffrey M. Bradshaw
    • 2
  • Matthew Johnson
    • 2
  • Paul Feltovich
    • 2
  • Karel van den Bosch
    • 3
  • John-Jules Meyer
    • 4
  1. 1.TU DelftDelftThe Netherlands
  2. 2.IHMCPensacolaUnited States
  3. 3.TNOSoesterbergThe Netherlands
  4. 4.Utrecht UniversityThe Netherlands

Personalised recommendations