Towards a Theory of Longitudinal Trust Calibration in Human–Robot Teams

  • Ewart J. de VisserEmail author
  • Marieke M. M. Peeters
  • Malte F. Jung
  • Spencer Kohn
  • Tyler H. Shaw
  • Richard Pak
  • Mark A. Neerincx


The introduction of artificial teammates in the form of autonomous social robots, with fewer social abilities compared to humans, presents new challenges for human–robot team dynamics. A key characteristic of high performing human-only teams is their ability to establish, develop, and calibrate trust over long periods of time, making the establishment of longitudinal human–robot team trust calibration a crucial part of these challenges. This paper presents a novel integrative model that takes a longitudinal perspective on trust development and calibration in human–robot teams. A key new proposed factor in this model is the introduction of the concept relationship equity. Relationship equity is an emotional resource that predicts the degree of goodwill between two actors. Relationship equity can help predict the future health of a long-term relationship. Our model is descriptive of current trust dynamics, predictive of the impact on trust of interactions within a human–robot team, and prescriptive with respect to the types of interventions and transparency methods promoting trust calibration. We describe the interplay between team trust dynamics and the establishment of work agreements that guide and improve human–robot collaboration. Furthermore, we introduce methods for dampening (reducing overtrust) and repairing (reducing undertrust) mis-calibrated trust between team members as well as methods for transparency and explanation. We conclude with a description of the implications of our model and a research agenda to jump-start a new comprehensive research program in this area.


Relationship equity Social autonomy Trust repair Trust calibration Work agreements Agents Social abilities Human–robot interaction Collaboration Team 



This material is based upon work supported by the Air Force Office of Scientific Research under award numbers 16RT0881 and FA9550-18-1-0455, as well as the Dutch Ministry of Defence’s exploratory research program (project Human-AI Teaming).

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Abubshait A, Wiese E (2017) You look human, but act like a machine: agent appearance and behavior modulate different aspects of human–robot interaction. Front Psychol 8:1393CrossRefGoogle Scholar
  2. 2.
    Andersson LM, Pearson CM (1999) Tit for tat? the spiraling effect of incivility in the workplace. Acad Manag Rev 24(3):452–471CrossRefGoogle Scholar
  3. 3.
    Alonso V, de la Puente P (2018) System transparency in shared autonomy: a mini review. Front Neurorobot 12:83CrossRefGoogle Scholar
  4. 4.
    Atkinson DJ, Clancey WJ, Clark MH (2014) Shared awareness, autonomy and trust in human-robot teamwork. In: 2014 AAAI fall symposium seriesGoogle Scholar
  5. 5.
    Bagosi T, Hindriks KV, Neerincx MA (2016) Ontological reasoning for human-robot teaming in search and rescue missions. In: 2016 11th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 595–596Google Scholar
  6. 6.
    Bahner JE, Hüper AD, Manzey D (2008) Misuse of automated decision aids: Complacency, automation bias and the impact of training experience. Int J Hum–Comput Stud 66(9):688–699CrossRefGoogle Scholar
  7. 7.
    Baker AL, Phillips EK, Ullman D, Keebler JR (2018) Toward an understanding of trust repair in human–robot interaction: current research and future directions. ACM Trans Interact Intell Syst (TiiS) 8(4):30Google Scholar
  8. 8.
    Beller J, Heesen M, Vollrath M (2013) Improving the driver-automation interaction: an approach using automation uncertainty. Hum Factors 55(6):1130–1141CrossRefGoogle Scholar
  9. 9.
    Billings DR, Schaefer KE, Chen JYC, Hancock PA (2012) Human-robot interaction: developing trust in robots. In: 2012 7th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 109–110Google Scholar
  10. 10.
    Byrne EA, Parasuraman R (1996) Psychophysiology and adaptive automation. Biol Psychol 42(3):249–268CrossRefGoogle Scholar
  11. 11.
    Castelfranchi C, Falcone R (2010) Trust theory: a socio-cognitive and computational model, vol 18. Wiley, HobokenzbMATHCrossRefGoogle Scholar
  12. 12.
    Chen JY, Barnes MJ (2014) Human-agent teaming for multirobot control: a review of human factors issues. IEEE Trans Hum–Mach Syst 44(1):13–29CrossRefGoogle Scholar
  13. 13.
    Chen JYC, Barnes MJ, Wright JL, Stowers K, Lakhmani SG (2017) Situation awareness-based agent transparency for human-autonomy teaming effectiveness. In: Micro-and nanotechnology sensors, systems, and applications IX, vol 10194. International Society for Optics and Photonics, p 101941VGoogle Scholar
  14. 14.
    Chen JY, Lakhmani SG, Stowers K, Selkowitz AR, Wright JL, Barnes M (2018) Situation awareness-based agent transparency and human-autonomy teaming effectiveness. Theor Issues Ergon Sci 19(3):259–282CrossRefGoogle Scholar
  15. 15.
    Chen M, Nikolaidis S, Soh H, Hsu D, Srinivasa S (2018) Planning with trust for human-robot collaboration. In: Proceedings of the 2018 ACM/IEEE international conference on human–robot interaction. ACM, pp 307–315Google Scholar
  16. 16.
    Chien SY, Lewis M, Sycara K, Kumru A, Liu JS (2019) Influence of culture, transparency, trust, and degree of automation on automation use. IEEE Trans Hum Mach Syst (submitted)Google Scholar
  17. 17.
    Chopra AK, Singh MP (2016) From social machines to social protocols: Software engineering foundations for sociotechnical systems. In: Proceedings of the 25th international conference on world wide web. International World Wide Web Conferences Steering Committee, pp 903–914Google Scholar
  18. 18.
    Cohen MS, Parasuraman R, Freeman JT (1998) Trust in decision aids: a model and its training implications. In: Proceedings of the command and control research and technology symposium. CiteseerGoogle Scholar
  19. 19.
    Correia F, Mascarenhas S, Prada R, Melo FS, Paiva A (2018) Group-based emotions in teams of humans and robots. In: Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction. ACM, pp 261–269Google Scholar
  20. 20.
    Correia F, Mascarenhas SF, Gomes S, Arriaga P, Leite I, Prada R, Melo FS, Paiva A (2019) Exploring prosociality in human-robot teams. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 143–151Google Scholar
  21. 21.
    Correia F, Melo FS, Paiva A (2019) Group intelligence on social robots. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 703–705Google Scholar
  22. 22.
    Dastani M, van der Torre L, Yorke-Smith N (2017) Commitments and interaction norms in organisations. Auton Agents Multi-Agent Syst 31(2):207–249CrossRefGoogle Scholar
  23. 23.
    de Graaf MM, Malle BF (2019) People’s explanations of robot behavior subtly reveal mental state Inferences. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 239–248Google Scholar
  24. 24.
    de Visser EJ, Parasuraman R (2011) Adaptive aiding of human–robot teaming: effects of imperfect automation on performance, trust, and workload. J Cognit Eng Decis Mak 5(2):209–231CrossRefGoogle Scholar
  25. 25.
    de Visser EJ, Cohen M, Freedy A, Parasuraman R (2014) A design methodology for trust cue calibration in cognitive agents. In: International conference on virtual, augmented and mixed reality. Springer, pp 251–262Google Scholar
  26. 26.
    de Visser EJ, Monfort SS, McKendrick R, Smith MAB, McKnight PE, Krueger F, Parasuraman R (2016) Almost human: anthropomorphism increases trust resilience in cognitive agents. J Exp Psychol Appl 22(3):331CrossRefGoogle Scholar
  27. 27.
    de Visser EJ, Pak R, Neerincx MA (2017) Trust development and repair in human-robot teams. In: Proceedings of the companion of the 2017 ACM/IEEE international conference on human–robot interaction. ACM, pp 103–104Google Scholar
  28. 28.
    de Visser EJ, Beatty PJ, Estepp JR, Kohn S, Abubshait A, Fedota JR, McDonald CG (2018) Learning from the slips of others: neural correlates of trust in automated agents. Front Hum Neurosci. CrossRefGoogle Scholar
  29. 29.
    de Visser EJ, Pak R, Shaw TH (2018) From ‘automation’to ‘autonomy’: the importance of trust repair in human-machine interaction. Ergonomics 61(10):1409–1427CrossRefGoogle Scholar
  30. 30.
    Degani A, Shafto M, Kirlik A (1999) Modes in human-machine systems: constructs, representation, and classification. Int J Aviat Psychol 9(2):125–138CrossRefGoogle Scholar
  31. 31.
    Demir M, McNeese NJ, Cooke NJ (2017) Team situation awareness within the context of human-autonomy teaming. Cognit Syst Res 46:3–12CrossRefGoogle Scholar
  32. 32.
    Demir M, McNeese NJ, Johnson C, Gorman JC, Grimm D, Cooke NJ (2019) Effective team interaction for adaptive training and situation awareness in human-autonomy teaming. In: 2019 IEEE conference on cognitive and computational aspects of situation management (CogSIMA). IEEE, pp 122–126Google Scholar
  33. 33.
    Deutsch M (1960) The effect of motivational orientation upon trust and suspicion. Hum Relat 13(2):123–139CrossRefGoogle Scholar
  34. 34.
    Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A., & Yanco, H. (2013, March). Impact of robot failures and feedback on real-time trust. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction (pp. 251–258). IEEE PressGoogle Scholar
  35. 35.
    Desai, M., Medvedev, M., Vázquez, M., McSheehy, S., Gadea-Omelchenko, S., Bruggeman, C., ... & Yanco, H. (2012, March). Effects of changing reliability on trust of robot systems. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction (pp. 73–80). ACMGoogle Scholar
  36. 36.
    Du N, Huang KY, Yang XJ (2019) Not all information is equal: effects of disclosing different types of likelihood information on trust, compliance and reliance, and task performance in human-automation teaming. Hum Factors. CrossRefGoogle Scholar
  37. 37.
    Duhigg C (2016) What google learned from its quest to build the perfect team. The New York Times Magazine.
  38. 38.
    Dzindolet MT, Peterson SA, Pomranky RA, Pierce LG, Beck HP (2003) The role of trust in automation reliance. Int J Hum–Comput Stud 58(6):697–718CrossRefGoogle Scholar
  39. 39.
    Edmondson AC, Kramer RM, Cook KS (2004) Psychological safety, trust, and learning in organizations: a group-level lens. Trust Distrust in Organ Dilemmas Approaches 12:239–272Google Scholar
  40. 40.
    Evans JSB, Frankish KE (2009) In two minds: dual processes and beyond. Oxford University Press, OxfordCrossRefGoogle Scholar
  41. 41.
    Falcone R, Castelfranchi C (2001) The socio-cognitive dynamics of trust: Does trust create trust? In: Trust in cyber-societies. Springer, pp 55–72Google Scholar
  42. 42.
    Feigh KM, Dorneich MC, Hayes CC (2012) Toward a characterization of adaptive systems: a framework for researchers and system designers. Hum Factors 54(6):1008–1024CrossRefGoogle Scholar
  43. 43.
    Fredrickson BL, Losada MF (2005) Positive affect and the complex dynamics of human flourishing. Am Ppsychol 60(7):678CrossRefGoogle Scholar
  44. 44.
    Freedy A, de Visser E, Weltman G, Coeyman N (2007) Measurement of trust in human-robot collaboration. In: International symposium on collaborative technologies and systems, 2007 (CTS 2007). IEEE, pp 106–114Google Scholar
  45. 45.
    Goddard K, Roudsari A, Wyatt JC (2011) Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc 19(1):121–127CrossRefGoogle Scholar
  46. 46.
    Goodrich MA, Crandall JW, Oudah M, Mathema N (2018) Using narrative to enable longitudinal human-robot interactions. In: Proceedings of the HRI2018 workshop on longitudinal human–robot teaming, Chicago, ILGoogle Scholar
  47. 47.
    Goodyear K, Parasuraman R, Chernyak S, de Visser EJ, Madhavan P, Deshpande G, Krueger F (2017) An fMRI and effective connectivity study investigating miss errors during advice utilization from human and machine agents. Soc Neurosci 12(5):570–581CrossRefGoogle Scholar
  48. 48.
    Gottman JM, Levenson RW (1992) Marital processes predictive of later dissolution: behavior, physiology, and health. J Personal Soc Psychol 63(2):221CrossRefGoogle Scholar
  49. 49.
    Greczek J, Kaszubski E, Atrash A, Matarić M (2014) Graded cueing feedback in robot-mediated imitation practice for children with autism spectrum disorders. In: The 23rd IEEE international symposium on robot and human interactive communication (2014 RO–MAN). IEEE, pp 561–566Google Scholar
  50. 50.
    Groom V, Nass C (2007) Can robots be teammates? Benchmarks in human–robot teams. Interact Stud 8(3):483–500CrossRefGoogle Scholar
  51. 51.
    Guidotti R, Monreale A, Turini F, Pedreschi D, Giannotti F (2018) A survey of methods for explaining black box models. arXiv preprint arXiv:1802.01933
  52. 52.
    Guznov S, Lyons J, Pfahler M, Heironimus A, Woolley M, Friedman J, Neimeier A (2019) Robot transparency and team orientation effects on human–robot teaming. Int J Hum Comput Interact. CrossRefGoogle Scholar
  53. 53.
    Hancock PA, Billings DR, Schaefer KE, Chen JYC, De Visser EJ, Parasuraman R (2011) A meta-analysis of factors affecting trust in human–robot interaction. Hum Factors 53(5):517–527CrossRefGoogle Scholar
  54. 54.
    Haring KS, Watanabe K, Velonaki M, Tossell CC, Finomore V (2018) FFAB-The form function attribution bias in human–robot interaction. IEEE Trans Cognit Dev Syst 10(4):843–851CrossRefGoogle Scholar
  55. 55.
    Haring KS, Mosley A, Pruznick S, Fleming J, Satterfield K, de Visser EJ, Tossell CC, Funke G, (2019) Robot authority in human-machine teams: effects of human-like appearance on compliance. In: Chen J, Fragomeni G (eds) Virtual, augmented and mixed reality. Applications and case studies. HCII, (2019) Lecture notes in computer science, vol 11575. Springer, Cham, pp 63–78Google Scholar
  56. 56.
    Hayes CC, Miller CA (2010) Human-computer etiquette: Cultural expectations and the design implications they place on computers and technology. CRC Press, Boca RatonCrossRefGoogle Scholar
  57. 57.
    Helldin T, Falkman G, Riveiro M, Davidsson S (2013) Presenting system uncertainty in automotive UIs for supporting trust calibration in autonomous driving. In: International conference on automotive user interfaces and interactive vehicular applications. ACM, pp 210–217Google Scholar
  58. 58.
    Hertz N, Shaw T, de Visser EJ, Wiese E (2019) Mixing it up: how mixed groups of humans and machines modulate conformity. J Cogn Eng Decis Mak. CrossRefGoogle Scholar
  59. 59.
    Hoff KA, Bashir M (2015) Trust in automation: Integrating empirical evidence on factors that influence trust. Hum Factors 57(3):407–434CrossRefGoogle Scholar
  60. 60.
    Huang SH, Bhatia K, Abbeel P, Dragan AD (2018) Establishing appropriate trust via critical states. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 3929–3936Google Scholar
  61. 61.
    Huseman RC, Hatfield JD, Miles EW (1987) A new perspective on equity theory: the equity sensitivity construct. Acad Mmanag Rev 12(2):222–234CrossRefGoogle Scholar
  62. 62.
    Hutchins AR, Cummings ML, Draper M, Hughes T (2015) Representing autonomous systems’ self-confidence through competency boundaries. In: The Human factors and ergonomics society annual meeting, vol 59. SAGE Publications Sage CA, Los Angeles, CA, pp 279–283CrossRefGoogle Scholar
  63. 63.
    Inagaki T et al (2003) Adaptive automation: sharing and trading of control. Handb Cognit Task Des 8:147–169CrossRefGoogle Scholar
  64. 64.
    Inbar O, Meyer J (2019) Politeness counts: perceptions of peacekeeping robots. IEEE Trans Hum Mach Syst 49(3):232–240CrossRefGoogle Scholar
  65. 65.
    Israelsen BW, Ahmed NR (2019) “Dave... I can assure you... that it’s going to be all right...” A definition, case for, and survey of algorithmic assurances in human-autonomy trust relationships. ACM Comput Surv 51(6):113CrossRefGoogle Scholar
  66. 66.
    Gottman JM (1994) What predicts divorce?. L. Erlbaum, USAGoogle Scholar
  67. 67.
    Gottman JM (2005) The mathematics of marriage: dynamic nonlinear models. MIT Press, BostonGoogle Scholar
  68. 68.
    Gottman JM (2011) The science of trust: emotional attunement for couples. WW Norton & Company, New YorkGoogle Scholar
  69. 69.
    Jackson RB, Williams T (2019) Language-capable robots may inadvertently weaken human moral norms. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 401–410Google Scholar
  70. 70.
    Jarrold W, Yeh PZ (2016) The social-emotional turing challenge. AI magazine 37(1):31–39Google Scholar
  71. 71.
    Jensen T, Albayram Y, Khan MMH, Fahim MAA, Buck R, Coman E (2019) The apple does fall far from the tree: user separation of a system from its developers in human-automation trust repair. In: Proceedings of the 2019 on designing interactive systems conference. ACM, pp 1071–1082Google Scholar
  72. 72.
    Jung MF (2016) Coupling interactions and performance: predicting team performance from thin slices of conflict. ACM Trans Comput–Hum Interact (TOCHI) 23(3):18CrossRefGoogle Scholar
  73. 73.
    Jung MF (2017) Affective grounding in human-robot interaction. In: Proceedings of the 2017 ACM/IEEE international conference on human–robot interaction. ACM, pp 263–273Google Scholar
  74. 74.
    Jung MF, Beane M, Forlizzi J, Murphy R, Vertesi J (2017) Robots in group context: rethinking design, development and deployment. In: Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems. ACM, pp 1283–1288Google Scholar
  75. 75.
    Jung MF, Martelaro N, Hinds PJ (2015) Using robots to moderate team conflict: the case of repairing violations. In: Proceedings of the tenth annual ACM/IEEE international conference on human–robot interaction. ACM, pp 229–236Google Scholar
  76. 76.
    Jung MF, Šabanović S, Eyssel F, Fraune M (2017) Robots in groups and teams. In: Companion of the 2017 ACM conference on computer supported cooperative work and social computing. ACM, pp 401–407Google Scholar
  77. 77.
    Juvina I, Collins MG, Larue O, Kennedy WG, Visser ED, Melo CD (2019) Toward a unified theory of learned trust in interpersonal and human–machine interactions. ACM Trans Interact Intell Syst 9(4):24CrossRefGoogle Scholar
  78. 78.
    Kaber DB, Endsley MR (2004) The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theor Issues Ergon Sci 5(2):113–153CrossRefGoogle Scholar
  79. 79.
    Kahneman D (2011) Thinking, fast and slow. Macmillan, LondonGoogle Scholar
  80. 80.
    Kaptein F, Broekens J, Hindriks K, Neerincx M (2017) Personalised self-explanation by robots: the role of goals versus beliefs in robot-action explanation for children and adults. In: 2017 26th IEEE international symposium on robot and human interactive communication (RO–MAN). IEEE, pp 676–682Google Scholar
  81. 81.
    Kaptein F, Broekens J, Hindriks K, Neerincx M (2017) The role of emotion in self-explanations by cognitive agents. In: 2017 seventh international conference on affective computing and intelligent interaction workshops and demos (ACIIW). IEEE, pp 88–93Google Scholar
  82. 82.
    Kayal A (2017) Normative social applications: user-centered models for sharing location in the family life domain. Ph.D. thesis, Delft University of TechnologyGoogle Scholar
  83. 83.
    Keltner D, Young RC, Buswell BN (1997) Appeasement in human emotion, social practice, and personality. Aggress Behav 23(5):359–374CrossRefGoogle Scholar
  84. 84.
    Kiesler S (2005) Fostering common ground in human–robot interaction. In: ROMAN 2005. IEEE international workshop on robot and human interactive communication. IEEE, pp 729–734Google Scholar
  85. 85.
    Kohn SC, Quinn D, Pak R, de Visser EJ, Shaw TH (2018) Trust repair strategies with self-driving vehicles: an exploratory study. In: Proceedings of the human factors and ergonomics society annual meeting, vol 62. Sage, Los Angeles, pp 1108–1112CrossRefGoogle Scholar
  86. 86.
    Kunze A, Summerskill SJ, Marshall R, Filtness AJ (2019) Automation transparency: implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics 62(3):345–360CrossRefGoogle Scholar
  87. 87.
    Lee JD, See KA (2004) Trust in automation: designing for appropriate reliance. Hum Factors 46(1):50–80CrossRefGoogle Scholar
  88. 88.
    Lewicki RJ, Tomlinson EC, Gillespie N (2006) Models of interpersonal trust development: theoretical approaches, empirical evidence, and future directions. J Mnag 32(6):991–1022Google Scholar
  89. 89.
    Losada M, Heaphy E (2004) The role of positivity and connectivity in the performance of business teams: a nonlinear dynamics model. Am Behav Sci 47(6):740–765CrossRefGoogle Scholar
  90. 90.
    Lyons JB, Guznov SY (2019) Individual differences in human-machine trust: a multi-study look at the perfect automation schema. Theor Issues Ergon Sci 20(4):440–458CrossRefGoogle Scholar
  91. 91.
    Lyons JB, Clark MA, Wagner AR, Schuelke MJ (2017) Certifiable trust in autonomous systems: making the intractable tangible. AI Mag 38(3):37–49CrossRefGoogle Scholar
  92. 92.
    Malle BF, Scheutz M (2018) Learning how to behave: moral competence for social robots. In: Handbuch Maschinenethik, pp 1–24Google Scholar
  93. 93.
    Malle BF, Scheutz M, Forlizzi J, Voiklis J (2016) Which robot am i thinking about? The impact of action and appearance on people’s evaluations of a moral robot. In: The eleventh ACM/IEEE international conference on human robot interaction. IEEE Press, pp 125–132Google Scholar
  94. 94.
    Marinaccio K, Kohn S, Parasuraman R, De Visser EJ (2015) A framework for rebuilding trust in social automation across health-care domains. In: Proceedings of the international symposium on human factors and ergonomics in health care, vol 4. Sage, New Delhi, pp 201–205CrossRefGoogle Scholar
  95. 95.
    Masalonis AJ, Parasuraman R (2003) Effects of situation-specific reliability on trust and usage of automated air traffic control decision aids. In: The human factors and ergonomics society annual meeting, vol 47. SAGE Publications Sage, Los Angeles, CA, pp 533–537CrossRefGoogle Scholar
  96. 96.
    Mayer RC, Davis JH, Schoorman FD (1995) An integrative model of organizational trust. Acad Manag Rev 20(3):709–734CrossRefGoogle Scholar
  97. 97.
    McGuirl JM, Sarter NB (2006) Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Hum Factors 48(4):656–665CrossRefGoogle Scholar
  98. 98.
    McNeese N, Demir M, Chiou E, Cooke N, Yanikian G (2019) Understanding the role of trust in human-autonomy teaming. In: Proceedings of the 52nd Hawaii international conference on system sciencesGoogle Scholar
  99. 99.
    Mercado JE, Rupp MA, Chen JY, Barnes MJ, Barber D, Procci K (2016) Intelligent agent transparency in human-agent teaming for Multi-UxV management. Hum Factors 58(3):401–415CrossRefGoogle Scholar
  100. 100.
    Meyer J, Miller C, Hancock P, de Visser EJ, Dorneich M (2016). Politeness in machine–human and human–human interaction. In: Proceedings of the human factors and ergonomics society annual meeting, vol 60. Sage, Los Angeles, pp 279–283CrossRefGoogle Scholar
  101. 101.
    Miller T (2017) Explanation in artificial intelligence: Insights from the social sciences. arXiv preprint arXiv:1706.07269
  102. 102.
    Mioch T, Peeters MMM, Neerincx MA (2018) Improving adaptive human-robot cooperation through work agreements. In: 27th IEEE international symposium on robot and human interactive communication (RO–MAN 2018), Nanjing, China, August 27–31, 2018, pp 1105–1110.
  103. 103.
    Morris MW, Keltner D (2000) How emotions work: the social functions of emotional expression in negotiations. Res Organ Behav 22:1–50CrossRefGoogle Scholar
  104. 104.
    Mueller ST, Hoffman RR, Clancey W, Emrey A, Klein G (2019) Explanation in human–AI systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. arXiv preprint arXiv:1902.01876
  105. 105.
    Nayyar M, Wagner AR (2018) When should a robot apologize? Understanding how timing affects human–robot trust repair. International conference on social robotics. Springer, Cham, pp 265–274CrossRefGoogle Scholar
  106. 106.
    Neerincx M, Van der Waa J, Kaptein F, Van Diggelen J (2018) Using perceptual and cognitive explanations for enhanced human-agent team performance. In: Engineering psychology and cognitive ergonomics. SpringerGoogle Scholar
  107. 107.
    Neerincx MA, van Diggelen J, van Breda L (2016) Interaction design patterns for adaptive human–agent–robot teamwork in high-risk domains. In: International conference on engineering psychology and cognitive ergonomics, pp 211–220. SpringerGoogle Scholar
  108. 108.
    Ososky S, Schuster D, Phillips E, Jentsch FG (2013) Building appropriate trust in human-robot teams. In: AAAI spring symposium: trust and autonomous systemsGoogle Scholar
  109. 109.
    Oudah M, Rahwan T, Crandall T, Crandall JW (2018) How AI wins friends and influences people in repeated games with cheap talk. In: Thirty-second AAAI conference on artificial intelligenceGoogle Scholar
  110. 110.
    Pak R, Fink N, Price M, Bass B, Sturre L (2012) Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults. Ergonomics 55(9):1059–1072CrossRefGoogle Scholar
  111. 111.
    Parasuraman R, Manzey DH (2010) Complacency and bias in human use of automation: an attentional integration. Hum Factors 52(3):381–410CrossRefGoogle Scholar
  112. 112.
    Parasuraman R, Miller CA (2004) Trust and etiquette in high-criticality automated systems. Commun ACM 47(4):51–55CrossRefGoogle Scholar
  113. 113.
    Parasuraman R, Riley V (1997) Humans and automation: use, misuse, disuse, abuse. Hum Factors 39(2):230–253CrossRefGoogle Scholar
  114. 114.
    Peeters MMM (2016) ReMindMe: agent-based support for self-disclosure of personal memories in people with alzheimer’s disease. In: Proceedings of the ICT4AWE. ScitePress, Rome, pp 61–66Google Scholar
  115. 115.
    Peeters MMM, Neerincx MA (2016) Human-agent experience sharing: creating social agents for elderly people with dementia. In: UMAP (extended proceedings)Google Scholar
  116. 116.
    Peeters MMM, van den Bosch K, Neerincx MA, Meyer JJC (2014) An ontology for automated scenario-based training. Int J Technol Enhanc Learn 6(3):195–211CrossRefGoogle Scholar
  117. 117.
    Phillips E, Ososky S, Grove J, Jentsch F (2011) From tools to teammates: toward the development of appropriate mental models for intelligent robots. In: Proceedings of the human factors and ergonomics society annual meeting, vol 55. Sage, Los Angeles, pp 1491–1495CrossRefGoogle Scholar
  118. 118.
    Phillips E, Zhao X, Ullman D, Malle BF (2018) What is human-like?: Decomposing robots’ human-like appearance using the anthropomorphic robot (abot) database. In: Proceedings of the 2018 ACM/IEEE international conference on human–robot interaction. ACM, pp 105–113Google Scholar
  119. 119.
    Robinette P, Howard AM, Wagner AR (2015) Timing is key for robot trust repair. International conference on social robotics. Springer, Cham, pp 574–583CrossRefGoogle Scholar
  120. 120.
    Robinette P, Howard AM, Wagner AR (2017) Effect of robot performance on human–robot trust in time-critical situations. IEEE Trans Hum Mach Syst 47(4):425–436CrossRefGoogle Scholar
  121. 121.
    Robinette P, Li W, Allen R, Howard AM, Wagner AR (2016) Overtrust of robots in emergency evacuation scenarios. In: The eleventh ACM/IEEE international conference on human robot interaction. IEEE Press, pp 101–108Google Scholar
  122. 122.
    Rossi A, Dautenhahn K, Koay KL, Saunders J (2017) Investigating human perceptions of trust in robots for safe HRI in home environments. In: Proceedings of the companion of the 2017 ACM/IEEE international conference on human–robot interaction. ACM, pp 375–376Google Scholar
  123. 123.
    Salas E, Dickinson TL, Converse SA, Tannenbaum SI (1992) Toward an understanding of team performance and training. In: Teams: their training and performance. Ablex PublishingGoogle Scholar
  124. 124.
    Salas E, Sims DE, Burke CS (2005) Is there a “big five” in teamwork? Small group research 36(5):555–599CrossRefGoogle Scholar
  125. 125.
    Salas E, Bisbey TM, Traylor AM, Rosen MA (2019) Can teamwork promote safety in organizations? Annu Rev Organ Psychol Organ Behav. CrossRefGoogle Scholar
  126. 126.
    Salem M, Lakatos G, Amirabdollahian F, Dautenhahn K (2015) Would you trust a (faulty) robot?: Effects of error, task type and personality on human-robot cooperation and trust. In: Proceedings of the tenth annual ACM/IEEE international conference on human–robot interaction. ACM, pp 141–148Google Scholar
  127. 127.
    Salomons N, van der Linden M, Strohkorb Sebo S, Scassellati B (2018) Humans conform to robots: disambiguating trust, truth, and conformity. In: Proceedings of the 2018 ACM/IEEE international conference on human–robot interaction. ACM, pp 187–195Google Scholar
  128. 128.
    Sauer J, Schmutz S, Sonderegger A, Messerli N (2019) Social stress and performance in human-machine interaction: a neglected research field. Ergonomics 62(11):1377–1391CrossRefGoogle Scholar
  129. 129.
    Scerbo MW (1996) Theoretical perspectives on adaptive automation. Theory and applications, automation and human performance, pp 37–63Google Scholar
  130. 130.
    Schulte A, Donath D, Lange DS (2016) Design patterns for human-cognitive agent teaming. In: International conference on engineering psychology and cognitive ergonomics. Springer, pp 231–243Google Scholar
  131. 131.
    Sebo SS, Krishnamurthi P, Scassellati B (2019) “I don’t believe you”: investigating the effects of robot trust violation and repair. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 57–65Google Scholar
  132. 132.
    Shively RJ, Lachter J, Brandt SL, Matessa M, Battiste V, Johnson WW (2017) Why human-autonomy teaming? In: International conference on applied human factors and ergonomics. Springer, pp 3–11Google Scholar
  133. 133.
    Singhvi A, Russel K (2016) Inside the self-driving tesla fatal accident. The New York Times Magazine.
  134. 134.
    Soh H, Shu P, Chen M, Hsu D (2018) The transfer of human trust in robot capabilities across tasks. arXiv preprint arXiv:1807.01866
  135. 135.
    Strohkorb Sebo S, Traeger M, Jung M, Scassellati B (2018) The ripple effects of vulnerability: the effects of a robot’s vulnerable behavior on trust in human–robot teams. In: Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction. ACM, pp 178–186Google Scholar
  136. 136.
    Tenhundfeld NL, de Visser EJ, Haring KS, Ries AJ, Finomore VS, Tossell CC (2019) Calibrating trust in automation through familiarity with the autoparking feature of a Tesla Model X. J Cognit Eng Decis Mak. CrossRefGoogle Scholar
  137. 137.
    Tenhundfeld NL, de Visser EJ, Ries AJ, Finomore VS, Tossell CC (2019) Trust and distrust of automated parking in a Tesla Model X. Hum Factors. CrossRefGoogle Scholar
  138. 138.
    van der Vecht B, van Diggelen J, Peeters MMM, van Staal W, van der Waa J (2018) The SAIL framework for implementing human-machine teaming concepts. International conference on practical applications of agents and multi-agent systems. Springer, Cham, pp 361–365Google Scholar
  139. 139.
    van der Vecht B, van Diggelen J, Peeters MMM, Barnhoorn J, van der Waa J (2018) SAIL: a social artificial intelligence layer for human–machine teaming. International conference on practical applications of agents and multi-agent systems. Springer, Cham, pp 262–274Google Scholar
  140. 140.
    Van der Waa J, van Diggelen J, Neerincx M (2018) The design and validation of an intuitive certainty measure. IUI 2018 workshop on explainable smart systems. In: IUI 2018 workshop on explainable smart systems. ACMGoogle Scholar
  141. 141.
    Van der Waa J, van Diggelen J, Neerincx M, Raaijmakers S (2018) ICM: an intuitive, model independent and accurate certainty measure for machine learning. In: 10th internantional conference on agents and AI. ICAARTGoogle Scholar
  142. 142.
    Van der Waa J, Robeer M, van Diggelen J, Brinkhuis M, Neerincx M (2018) Contrastive explanations with local foil trees. In: IJCAI 2018 workshop on human interpretability in machine learning. WHIGoogle Scholar
  143. 143.
    van der Waa J, van Diggelen J, van den Bosch K, Neerincx M (2018) Contrastive explanations for reinforcement learning in terms of expected consequences. Retrieved from arXiv:1807.08706
  144. 144.
    Van Diggelen J, Neerincx M, Peeters M, Schraagen JM (2018) Developing effective and resilient human-agent teamwork using team design patterns. IEEE Intell Syst 34(2):15–24CrossRefGoogle Scholar
  145. 145.
    van Diggelen J, Barnhoorn JS, Peeters MMM, van Staal W, van Stolk M, van der Vecht B, van der Waa J, Schraagen JM (2019) Pluggable social artificial intelligence for enabling human-agent teaming. arXiv preprint arXiv:1909.04492
  146. 146.
    Verberne FMF, Ham J, Midden CJH (2012) Trust in smart systems: sharing driving goals and giving information to increase trustworthiness and acceptability of smart systems in cars. Hum Factors 54(5):799–810CrossRefGoogle Scholar
  147. 147.
    Verberne FMF, Ham J, Ponnada A, Midden CJH (2013) Trusting digital chameleons: the effect of mimicry by a virtual social agent on user trust. In: International conference on persuasive technology. Springer, pp 234–245Google Scholar
  148. 148.
    Walliser JC, de Visser EJ, Wiese E, Shaw TH (2019) Team structure and team building improve human–machine teaming with autonomous agents. J Cognit Eng Decis Mak. CrossRefGoogle Scholar
  149. 149.
    Wang N, Pynadath DV, Hill SG (2016) Trust calibration within a human-robot team: comparing automatically generated explanations. In: The eleventh ACM/IEEE international conference on human robot interaction. IEEE Press, pp 109–116Google Scholar
  150. 150.
    Wen J, Stewart A, Billinghurst M, Dey A, Tossell C, Finomore V (2018) He who hesitates is lost (... in thoughts over a robot). In: Proceedings of the technology, mind, and society. ACM, p 43Google Scholar
  151. 151.
    Williams M (2007) Building genuine trust through interpersonal emotion management: a threat regulation model of trust and collaboration across boundaries. Acad Manag Rev 32(2):595–621CrossRefGoogle Scholar
  152. 152.
    Wiltshire TJ, Barber D, Fiore SM (2013) Towards modeling social-cognitive mechanisms in robots to facilitate human-robot teaming. In: Proceedings of the human factors and ergonomics society annual meeting, vol 57. SAGE Publications Sage, Los Angeles, CA, pp 1278–1282CrossRefGoogle Scholar
  153. 153.
    Wright JL, Chen JY, Lakhmani SG (2019) Agent transparency and reliability in human–robot interaction: the influence on user confidence and perceived reliability. IEEE Trans Hum Mach Syst. CrossRefGoogle Scholar
  154. 154.
    Wynne KT, Lyons JB (2018) An integrative model of autonomous agent teammate-likeness. Theor Issues Ergon Sci 19(3):353–374CrossRefGoogle Scholar
  155. 155.
    Xie Y, Bodala IP, Ong DC, Hsu D, Soh H (2019) Robot capability and intention in trust-based decisions across tasks. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 39–47Google Scholar
  156. 156.
    Xu A, Dudek G (2015) Optimo: Online probabilistic trust inference model for asymmetric human–robot collaborations. In: Proceedings of the tenth annual ACM/IEEE international conference on human–robot interaction. ACM, pp 221–228Google Scholar
  157. 157.
    Yang XJ, Unhelkar VV, Li K, Shah JA (2017) Evaluating effects of user experience and system transparency on trust in automation. In: 2017 12th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 408–416Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Warfighter Effectiveness Research CenterUnited States Air Force AcademyColorado SpringsUSA
  2. 2.Department of PsychologyClemson UniversityClemsonUSA
  3. 3.George Mason UniversityFairfaxUSA
  4. 4.TNO, Perceptual and Cognitive SystemsDE SoesterbergThe Netherlands
  5. 5.Delft University of TechnologyDelftThe Netherlands
  6. 6.Department of Information ScienceCornell UniversityIthacaUSA

Personalised recommendations