Skip to main content

Towards a Theory of Longitudinal Trust Calibration in Human–Robot Teams

International Journal of Social Robotics volume 12, pages 459–478 (2020)Cite this article

Abstract

The introduction of artificial teammates in the form of autonomous social robots, with fewer social abilities compared to humans, presents new challenges for human–robot team dynamics. A key characteristic of high performing human-only teams is their ability to establish, develop, and calibrate trust over long periods of time, making the establishment of longitudinal human–robot team trust calibration a crucial part of these challenges. This paper presents a novel integrative model that takes a longitudinal perspective on trust development and calibration in human–robot teams. A key new proposed factor in this model is the introduction of the concept relationship equity. Relationship equity is an emotional resource that predicts the degree of goodwill between two actors. Relationship equity can help predict the future health of a long-term relationship. Our model is descriptive of current trust dynamics, predictive of the impact on trust of interactions within a human–robot team, and prescriptive with respect to the types of interventions and transparency methods promoting trust calibration. We describe the interplay between team trust dynamics and the establishment of work agreements that guide and improve human–robot collaboration. Furthermore, we introduce methods for dampening (reducing overtrust) and repairing (reducing undertrust) mis-calibrated trust between team members as well as methods for transparency and explanation. We conclude with a description of the implications of our model and a research agenda to jump-start a new comprehensive research program in this area.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. Abubshait A, Wiese E (2017) You look human, but act like a machine: agent appearance and behavior modulate different aspects of human–robot interaction. Front Psychol 8:1393

    Google Scholar 

  2. Andersson LM, Pearson CM (1999) Tit for tat? the spiraling effect of incivility in the workplace. Acad Manag Rev 24(3):452–471

    Google Scholar 

  3. Alonso V, de la Puente P (2018) System transparency in shared autonomy: a mini review. Front Neurorobot 12:83

    Google Scholar 

  4. Atkinson DJ, Clancey WJ, Clark MH (2014) Shared awareness, autonomy and trust in human-robot teamwork. In: 2014 AAAI fall symposium series

  5. Bagosi T, Hindriks KV, Neerincx MA (2016) Ontological reasoning for human-robot teaming in search and rescue missions. In: 2016 11th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 595–596

  6. Bahner JE, Hüper AD, Manzey D (2008) Misuse of automated decision aids: Complacency, automation bias and the impact of training experience. Int J Hum–Comput Stud 66(9):688–699

    Google Scholar 

  7. Baker AL, Phillips EK, Ullman D, Keebler JR (2018) Toward an understanding of trust repair in human–robot interaction: current research and future directions. ACM Trans Interact Intell Syst (TiiS) 8(4):30

    Google Scholar 

  8. Beller J, Heesen M, Vollrath M (2013) Improving the driver-automation interaction: an approach using automation uncertainty. Hum Factors 55(6):1130–1141

    Google Scholar 

  9. Billings DR, Schaefer KE, Chen JYC, Hancock PA (2012) Human-robot interaction: developing trust in robots. In: 2012 7th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 109–110

  10. Byrne EA, Parasuraman R (1996) Psychophysiology and adaptive automation. Biol Psychol 42(3):249–268

    Google Scholar 

  11. Castelfranchi C, Falcone R (2010) Trust theory: a socio-cognitive and computational model, vol 18. Wiley, Hoboken

    MATH  Google Scholar 

  12. Chen JY, Barnes MJ (2014) Human-agent teaming for multirobot control: a review of human factors issues. IEEE Trans Hum–Mach Syst 44(1):13–29

    Google Scholar 

  13. Chen JYC, Barnes MJ, Wright JL, Stowers K, Lakhmani SG (2017) Situation awareness-based agent transparency for human-autonomy teaming effectiveness. In: Micro-and nanotechnology sensors, systems, and applications IX, vol 10194. International Society for Optics and Photonics, p 101941V

  14. Chen JY, Lakhmani SG, Stowers K, Selkowitz AR, Wright JL, Barnes M (2018) Situation awareness-based agent transparency and human-autonomy teaming effectiveness. Theor Issues Ergon Sci 19(3):259–282

    Google Scholar 

  15. Chen M, Nikolaidis S, Soh H, Hsu D, Srinivasa S (2018) Planning with trust for human-robot collaboration. In: Proceedings of the 2018 ACM/IEEE international conference on human–robot interaction. ACM, pp 307–315

  16. Chien SY, Lewis M, Sycara K, Kumru A, Liu JS (2019) Influence of culture, transparency, trust, and degree of automation on automation use. IEEE Trans Hum Mach Syst (submitted)

  17. Chopra AK, Singh MP (2016) From social machines to social protocols: Software engineering foundations for sociotechnical systems. In: Proceedings of the 25th international conference on world wide web. International World Wide Web Conferences Steering Committee, pp 903–914

  18. Cohen MS, Parasuraman R, Freeman JT (1998) Trust in decision aids: a model and its training implications. In: Proceedings of the command and control research and technology symposium. Citeseer

  19. Correia F, Mascarenhas S, Prada R, Melo FS, Paiva A (2018) Group-based emotions in teams of humans and robots. In: Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction. ACM, pp 261–269

  20. Correia F, Mascarenhas SF, Gomes S, Arriaga P, Leite I, Prada R, Melo FS, Paiva A (2019) Exploring prosociality in human-robot teams. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 143–151

  21. Correia F, Melo FS, Paiva A (2019) Group intelligence on social robots. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 703–705

  22. Dastani M, van der Torre L, Yorke-Smith N (2017) Commitments and interaction norms in organisations. Auton Agents Multi-Agent Syst 31(2):207–249

    Google Scholar 

  23. de Graaf MM, Malle BF (2019) People’s explanations of robot behavior subtly reveal mental state Inferences. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 239–248

  24. de Visser EJ, Parasuraman R (2011) Adaptive aiding of human–robot teaming: effects of imperfect automation on performance, trust, and workload. J Cognit Eng Decis Mak 5(2):209–231

    Google Scholar 

  25. de Visser EJ, Cohen M, Freedy A, Parasuraman R (2014) A design methodology for trust cue calibration in cognitive agents. In: International conference on virtual, augmented and mixed reality. Springer, pp 251–262

  26. de Visser EJ, Monfort SS, McKendrick R, Smith MAB, McKnight PE, Krueger F, Parasuraman R (2016) Almost human: anthropomorphism increases trust resilience in cognitive agents. J Exp Psychol Appl 22(3):331

    Google Scholar 

  27. de Visser EJ, Pak R, Neerincx MA (2017) Trust development and repair in human-robot teams. In: Proceedings of the companion of the 2017 ACM/IEEE international conference on human–robot interaction. ACM, pp 103–104

  28. de Visser EJ, Beatty PJ, Estepp JR, Kohn S, Abubshait A, Fedota JR, McDonald CG (2018) Learning from the slips of others: neural correlates of trust in automated agents. Front Hum Neurosci. https://doi.org/10.3389/fnhum.2018.00309

    Article  Google Scholar 

  29. de Visser EJ, Pak R, Shaw TH (2018) From ‘automation’to ‘autonomy’: the importance of trust repair in human-machine interaction. Ergonomics 61(10):1409–1427

    Google Scholar 

  30. Degani A, Shafto M, Kirlik A (1999) Modes in human-machine systems: constructs, representation, and classification. Int J Aviat Psychol 9(2):125–138

    Google Scholar 

  31. Demir M, McNeese NJ, Cooke NJ (2017) Team situation awareness within the context of human-autonomy teaming. Cognit Syst Res 46:3–12

    Google Scholar 

  32. Demir M, McNeese NJ, Johnson C, Gorman JC, Grimm D, Cooke NJ (2019) Effective team interaction for adaptive training and situation awareness in human-autonomy teaming. In: 2019 IEEE conference on cognitive and computational aspects of situation management (CogSIMA). IEEE, pp 122–126

  33. Deutsch M (1960) The effect of motivational orientation upon trust and suspicion. Hum Relat 13(2):123–139

    Google Scholar 

  34. Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A., & Yanco, H. (2013, March). Impact of robot failures and feedback on real-time trust. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction (pp. 251–258). IEEE Press

  35. Desai, M., Medvedev, M., Vázquez, M., McSheehy, S., Gadea-Omelchenko, S., Bruggeman, C., ... & Yanco, H. (2012, March). Effects of changing reliability on trust of robot systems. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction (pp. 73–80). ACM

  36. Du N, Huang KY, Yang XJ (2019) Not all information is equal: effects of disclosing different types of likelihood information on trust, compliance and reliance, and task performance in human-automation teaming. Hum Factors. https://doi.org/10.1177/0018720819862916

    Article  Google Scholar 

  37. Duhigg C (2016) What google learned from its quest to build the perfect team. The New York Times Magazine. https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html

  38. Dzindolet MT, Peterson SA, Pomranky RA, Pierce LG, Beck HP (2003) The role of trust in automation reliance. Int J Hum–Comput Stud 58(6):697–718

    Google Scholar 

  39. Edmondson AC, Kramer RM, Cook KS (2004) Psychological safety, trust, and learning in organizations: a group-level lens. Trust Distrust in Organ Dilemmas Approaches 12:239–272

    Google Scholar 

  40. Evans JSB, Frankish KE (2009) In two minds: dual processes and beyond. Oxford University Press, Oxford

    Google Scholar 

  41. Falcone R, Castelfranchi C (2001) The socio-cognitive dynamics of trust: Does trust create trust? In: Trust in cyber-societies. Springer, pp 55–72

  42. Feigh KM, Dorneich MC, Hayes CC (2012) Toward a characterization of adaptive systems: a framework for researchers and system designers. Hum Factors 54(6):1008–1024

    Google Scholar 

  43. Fredrickson BL, Losada MF (2005) Positive affect and the complex dynamics of human flourishing. Am Ppsychol 60(7):678

    Google Scholar 

  44. Freedy A, de Visser E, Weltman G, Coeyman N (2007) Measurement of trust in human-robot collaboration. In: International symposium on collaborative technologies and systems, 2007 (CTS 2007). IEEE, pp 106–114

  45. Goddard K, Roudsari A, Wyatt JC (2011) Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc 19(1):121–127

    Google Scholar 

  46. Goodrich MA, Crandall JW, Oudah M, Mathema N (2018) Using narrative to enable longitudinal human-robot interactions. In: Proceedings of the HRI2018 workshop on longitudinal human–robot teaming, Chicago, IL

  47. Goodyear K, Parasuraman R, Chernyak S, de Visser EJ, Madhavan P, Deshpande G, Krueger F (2017) An fMRI and effective connectivity study investigating miss errors during advice utilization from human and machine agents. Soc Neurosci 12(5):570–581

    Google Scholar 

  48. Gottman JM, Levenson RW (1992) Marital processes predictive of later dissolution: behavior, physiology, and health. J Personal Soc Psychol 63(2):221

    Google Scholar 

  49. Greczek J, Kaszubski E, Atrash A, Matarić M (2014) Graded cueing feedback in robot-mediated imitation practice for children with autism spectrum disorders. In: The 23rd IEEE international symposium on robot and human interactive communication (2014 RO–MAN). IEEE, pp 561–566

  50. Groom V, Nass C (2007) Can robots be teammates? Benchmarks in human–robot teams. Interact Stud 8(3):483–500

    Google Scholar 

  51. Guidotti R, Monreale A, Turini F, Pedreschi D, Giannotti F (2018) A survey of methods for explaining black box models. arXiv preprint arXiv:1802.01933

  52. Guznov S, Lyons J, Pfahler M, Heironimus A, Woolley M, Friedman J, Neimeier A (2019) Robot transparency and team orientation effects on human–robot teaming. Int J Hum Comput Interact. https://doi.org/10.1080/10447318.2019.1676519

    Article  Google Scholar 

  53. Hancock PA, Billings DR, Schaefer KE, Chen JYC, De Visser EJ, Parasuraman R (2011) A meta-analysis of factors affecting trust in human–robot interaction. Hum Factors 53(5):517–527

    Google Scholar 

  54. Haring KS, Watanabe K, Velonaki M, Tossell CC, Finomore V (2018) FFAB-The form function attribution bias in human–robot interaction. IEEE Trans Cognit Dev Syst 10(4):843–851

    Google Scholar 

  55. Haring KS, Mosley A, Pruznick S, Fleming J, Satterfield K, de Visser EJ, Tossell CC, Funke G, (2019) Robot authority in human-machine teams: effects of human-like appearance on compliance. In: Chen J, Fragomeni G (eds) Virtual, augmented and mixed reality. Applications and case studies. HCII, (2019) Lecture notes in computer science, vol 11575. Springer, Cham, pp 63–78

  56. Hayes CC, Miller CA (2010) Human-computer etiquette: Cultural expectations and the design implications they place on computers and technology. CRC Press, Boca Raton

    Google Scholar 

  57. Helldin T, Falkman G, Riveiro M, Davidsson S (2013) Presenting system uncertainty in automotive UIs for supporting trust calibration in autonomous driving. In: International conference on automotive user interfaces and interactive vehicular applications. ACM, pp 210–217

  58. Hertz N, Shaw T, de Visser EJ, Wiese E (2019) Mixing it up: how mixed groups of humans and machines modulate conformity. J Cogn Eng Decis Mak. https://doi.org/10.1177/1555343419869465

    Article  Google Scholar 

  59. Hoff KA, Bashir M (2015) Trust in automation: Integrating empirical evidence on factors that influence trust. Hum Factors 57(3):407–434

    Google Scholar 

  60. Huang SH, Bhatia K, Abbeel P, Dragan AD (2018) Establishing appropriate trust via critical states. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 3929–3936

  61. Huseman RC, Hatfield JD, Miles EW (1987) A new perspective on equity theory: the equity sensitivity construct. Acad Mmanag Rev 12(2):222–234

    Google Scholar 

  62. Hutchins AR, Cummings ML, Draper M, Hughes T (2015) Representing autonomous systems’ self-confidence through competency boundaries. In: The Human factors and ergonomics society annual meeting, vol 59. SAGE Publications Sage CA, Los Angeles, CA, pp 279–283

  63. Inagaki T et al (2003) Adaptive automation: sharing and trading of control. Handb Cognit Task Des 8:147–169

    Google Scholar 

  64. Inbar O, Meyer J (2019) Politeness counts: perceptions of peacekeeping robots. IEEE Trans Hum Mach Syst 49(3):232–240

    Google Scholar 

  65. Israelsen BW, Ahmed NR (2019) “Dave... I can assure you... that it’s going to be all right...” A definition, case for, and survey of algorithmic assurances in human-autonomy trust relationships. ACM Comput Surv 51(6):113

    Google Scholar 

  66. Gottman JM (1994) What predicts divorce?. L. Erlbaum, USA

    Google Scholar 

  67. Gottman JM (2005) The mathematics of marriage: dynamic nonlinear models. MIT Press, Boston

    Google Scholar 

  68. Gottman JM (2011) The science of trust: emotional attunement for couples. WW Norton & Company, New York

    Google Scholar 

  69. Jackson RB, Williams T (2019) Language-capable robots may inadvertently weaken human moral norms. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 401–410

  70. Jarrold W, Yeh PZ (2016) The social-emotional turing challenge. AI magazine 37(1):31–39

    Google Scholar 

  71. Jensen T, Albayram Y, Khan MMH, Fahim MAA, Buck R, Coman E (2019) The apple does fall far from the tree: user separation of a system from its developers in human-automation trust repair. In: Proceedings of the 2019 on designing interactive systems conference. ACM, pp 1071–1082

  72. Jung MF (2016) Coupling interactions and performance: predicting team performance from thin slices of conflict. ACM Trans Comput–Hum Interact (TOCHI) 23(3):18

    Google Scholar 

  73. Jung MF (2017) Affective grounding in human-robot interaction. In: Proceedings of the 2017 ACM/IEEE international conference on human–robot interaction. ACM, pp 263–273

  74. Jung MF, Beane M, Forlizzi J, Murphy R, Vertesi J (2017) Robots in group context: rethinking design, development and deployment. In: Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems. ACM, pp 1283–1288

  75. Jung MF, Martelaro N, Hinds PJ (2015) Using robots to moderate team conflict: the case of repairing violations. In: Proceedings of the tenth annual ACM/IEEE international conference on human–robot interaction. ACM, pp 229–236

  76. Jung MF, Šabanović S, Eyssel F, Fraune M (2017) Robots in groups and teams. In: Companion of the 2017 ACM conference on computer supported cooperative work and social computing. ACM, pp 401–407

  77. Juvina I, Collins MG, Larue O, Kennedy WG, Visser ED, Melo CD (2019) Toward a unified theory of learned trust in interpersonal and human–machine interactions. ACM Trans Interact Intell Syst 9(4):24

    Google Scholar 

  78. Kaber DB, Endsley MR (2004) The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theor Issues Ergon Sci 5(2):113–153

    Google Scholar 

  79. Kahneman D (2011) Thinking, fast and slow. Macmillan, London

    Google Scholar 

  80. Kaptein F, Broekens J, Hindriks K, Neerincx M (2017) Personalised self-explanation by robots: the role of goals versus beliefs in robot-action explanation for children and adults. In: 2017 26th IEEE international symposium on robot and human interactive communication (RO–MAN). IEEE, pp 676–682

  81. Kaptein F, Broekens J, Hindriks K, Neerincx M (2017) The role of emotion in self-explanations by cognitive agents. In: 2017 seventh international conference on affective computing and intelligent interaction workshops and demos (ACIIW). IEEE, pp 88–93

  82. Kayal A (2017) Normative social applications: user-centered models for sharing location in the family life domain. Ph.D. thesis, Delft University of Technology

  83. Keltner D, Young RC, Buswell BN (1997) Appeasement in human emotion, social practice, and personality. Aggress Behav 23(5):359–374

    Google Scholar 

  84. Kiesler S (2005) Fostering common ground in human–robot interaction. In: ROMAN 2005. IEEE international workshop on robot and human interactive communication. IEEE, pp 729–734

  85. Kohn SC, Quinn D, Pak R, de Visser EJ, Shaw TH (2018) Trust repair strategies with self-driving vehicles: an exploratory study. In: Proceedings of the human factors and ergonomics society annual meeting, vol 62. Sage, Los Angeles, pp 1108–1112

  86. Kunze A, Summerskill SJ, Marshall R, Filtness AJ (2019) Automation transparency: implications of uncertainty communication for human-automation interaction and interfaces. Ergonomics 62(3):345–360

    Google Scholar 

  87. Lee JD, See KA (2004) Trust in automation: designing for appropriate reliance. Hum Factors 46(1):50–80

    Google Scholar 

  88. Lewicki RJ, Tomlinson EC, Gillespie N (2006) Models of interpersonal trust development: theoretical approaches, empirical evidence, and future directions. J Mnag 32(6):991–1022

    Google Scholar 

  89. Losada M, Heaphy E (2004) The role of positivity and connectivity in the performance of business teams: a nonlinear dynamics model. Am Behav Sci 47(6):740–765

    Google Scholar 

  90. Lyons JB, Guznov SY (2019) Individual differences in human-machine trust: a multi-study look at the perfect automation schema. Theor Issues Ergon Sci 20(4):440–458

    Google Scholar 

  91. Lyons JB, Clark MA, Wagner AR, Schuelke MJ (2017) Certifiable trust in autonomous systems: making the intractable tangible. AI Mag 38(3):37–49

    Google Scholar 

  92. Malle BF, Scheutz M (2018) Learning how to behave: moral competence for social robots. In: Handbuch Maschinenethik, pp 1–24

  93. Malle BF, Scheutz M, Forlizzi J, Voiklis J (2016) Which robot am i thinking about? The impact of action and appearance on people’s evaluations of a moral robot. In: The eleventh ACM/IEEE international conference on human robot interaction. IEEE Press, pp 125–132

  94. Marinaccio K, Kohn S, Parasuraman R, De Visser EJ (2015) A framework for rebuilding trust in social automation across health-care domains. In: Proceedings of the international symposium on human factors and ergonomics in health care, vol 4. Sage, New Delhi, pp 201–205

  95. Masalonis AJ, Parasuraman R (2003) Effects of situation-specific reliability on trust and usage of automated air traffic control decision aids. In: The human factors and ergonomics society annual meeting, vol 47. SAGE Publications Sage, Los Angeles, CA, pp 533–537

  96. Mayer RC, Davis JH, Schoorman FD (1995) An integrative model of organizational trust. Acad Manag Rev 20(3):709–734

    Google Scholar 

  97. McGuirl JM, Sarter NB (2006) Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Hum Factors 48(4):656–665

    Google Scholar 

  98. McNeese N, Demir M, Chiou E, Cooke N, Yanikian G (2019) Understanding the role of trust in human-autonomy teaming. In: Proceedings of the 52nd Hawaii international conference on system sciences

  99. Mercado JE, Rupp MA, Chen JY, Barnes MJ, Barber D, Procci K (2016) Intelligent agent transparency in human-agent teaming for Multi-UxV management. Hum Factors 58(3):401–415

    Google Scholar 

  100. Meyer J, Miller C, Hancock P, de Visser EJ, Dorneich M (2016). Politeness in machine–human and human–human interaction. In: Proceedings of the human factors and ergonomics society annual meeting, vol 60. Sage, Los Angeles, pp 279–283

  101. Miller T (2017) Explanation in artificial intelligence: Insights from the social sciences. arXiv preprint arXiv:1706.07269

  102. Mioch T, Peeters MMM, Neerincx MA (2018) Improving adaptive human-robot cooperation through work agreements. In: 27th IEEE international symposium on robot and human interactive communication (RO–MAN 2018), Nanjing, China, August 27–31, 2018, pp 1105–1110. https://doi.org/10.1109/ROMAN.2018.8525776

  103. Morris MW, Keltner D (2000) How emotions work: the social functions of emotional expression in negotiations. Res Organ Behav 22:1–50

    Google Scholar 

  104. Mueller ST, Hoffman RR, Clancey W, Emrey A, Klein G (2019) Explanation in human–AI systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. arXiv preprint arXiv:1902.01876

  105. Nayyar M, Wagner AR (2018) When should a robot apologize? Understanding how timing affects human–robot trust repair. International conference on social robotics. Springer, Cham, pp 265–274

    Google Scholar 

  106. Neerincx M, Van der Waa J, Kaptein F, Van Diggelen J (2018) Using perceptual and cognitive explanations for enhanced human-agent team performance. In: Engineering psychology and cognitive ergonomics. Springer

  107. Neerincx MA, van Diggelen J, van Breda L (2016) Interaction design patterns for adaptive human–agent–robot teamwork in high-risk domains. In: International conference on engineering psychology and cognitive ergonomics, pp 211–220. Springer

  108. Ososky S, Schuster D, Phillips E, Jentsch FG (2013) Building appropriate trust in human-robot teams. In: AAAI spring symposium: trust and autonomous systems

  109. Oudah M, Rahwan T, Crandall T, Crandall JW (2018) How AI wins friends and influences people in repeated games with cheap talk. In: Thirty-second AAAI conference on artificial intelligence

  110. Pak R, Fink N, Price M, Bass B, Sturre L (2012) Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults. Ergonomics 55(9):1059–1072

    Google Scholar 

  111. Parasuraman R, Manzey DH (2010) Complacency and bias in human use of automation: an attentional integration. Hum Factors 52(3):381–410

    Google Scholar 

  112. Parasuraman R, Miller CA (2004) Trust and etiquette in high-criticality automated systems. Commun ACM 47(4):51–55

    Google Scholar 

  113. Parasuraman R, Riley V (1997) Humans and automation: use, misuse, disuse, abuse. Hum Factors 39(2):230–253

    Google Scholar 

  114. Peeters MMM (2016) ReMindMe: agent-based support for self-disclosure of personal memories in people with alzheimer’s disease. In: Proceedings of the ICT4AWE. ScitePress, Rome, pp 61–66

  115. Peeters MMM, Neerincx MA (2016) Human-agent experience sharing: creating social agents for elderly people with dementia. In: UMAP (extended proceedings)

  116. Peeters MMM, van den Bosch K, Neerincx MA, Meyer JJC (2014) An ontology for automated scenario-based training. Int J Technol Enhanc Learn 6(3):195–211

    Google Scholar 

  117. Phillips E, Ososky S, Grove J, Jentsch F (2011) From tools to teammates: toward the development of appropriate mental models for intelligent robots. In: Proceedings of the human factors and ergonomics society annual meeting, vol 55. Sage, Los Angeles, pp 1491–1495

  118. Phillips E, Zhao X, Ullman D, Malle BF (2018) What is human-like?: Decomposing robots’ human-like appearance using the anthropomorphic robot (abot) database. In: Proceedings of the 2018 ACM/IEEE international conference on human–robot interaction. ACM, pp 105–113

  119. Robinette P, Howard AM, Wagner AR (2015) Timing is key for robot trust repair. International conference on social robotics. Springer, Cham, pp 574–583

    Google Scholar 

  120. Robinette P, Howard AM, Wagner AR (2017) Effect of robot performance on human–robot trust in time-critical situations. IEEE Trans Hum Mach Syst 47(4):425–436

    Google Scholar 

  121. Robinette P, Li W, Allen R, Howard AM, Wagner AR (2016) Overtrust of robots in emergency evacuation scenarios. In: The eleventh ACM/IEEE international conference on human robot interaction. IEEE Press, pp 101–108

  122. Rossi A, Dautenhahn K, Koay KL, Saunders J (2017) Investigating human perceptions of trust in robots for safe HRI in home environments. In: Proceedings of the companion of the 2017 ACM/IEEE international conference on human–robot interaction. ACM, pp 375–376

  123. Salas E, Dickinson TL, Converse SA, Tannenbaum SI (1992) Toward an understanding of team performance and training. In: Teams: their training and performance. Ablex Publishing

  124. Salas E, Sims DE, Burke CS (2005) Is there a “big five” in teamwork? Small group research 36(5):555–599

    Google Scholar 

  125. Salas E, Bisbey TM, Traylor AM, Rosen MA (2019) Can teamwork promote safety in organizations? Annu Rev Organ Psychol Organ Behav. https://doi.org/10.1146/annurev-orgpsych-012119-045411

    Article  Google Scholar 

  126. Salem M, Lakatos G, Amirabdollahian F, Dautenhahn K (2015) Would you trust a (faulty) robot?: Effects of error, task type and personality on human-robot cooperation and trust. In: Proceedings of the tenth annual ACM/IEEE international conference on human–robot interaction. ACM, pp 141–148

  127. Salomons N, van der Linden M, Strohkorb Sebo S, Scassellati B (2018) Humans conform to robots: disambiguating trust, truth, and conformity. In: Proceedings of the 2018 ACM/IEEE international conference on human–robot interaction. ACM, pp 187–195

  128. Sauer J, Schmutz S, Sonderegger A, Messerli N (2019) Social stress and performance in human-machine interaction: a neglected research field. Ergonomics 62(11):1377–1391

    Google Scholar 

  129. Scerbo MW (1996) Theoretical perspectives on adaptive automation. Theory and applications, automation and human performance, pp 37–63

  130. Schulte A, Donath D, Lange DS (2016) Design patterns for human-cognitive agent teaming. In: International conference on engineering psychology and cognitive ergonomics. Springer, pp 231–243

  131. Sebo SS, Krishnamurthi P, Scassellati B (2019) “I don’t believe you”: investigating the effects of robot trust violation and repair. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 57–65

  132. Shively RJ, Lachter J, Brandt SL, Matessa M, Battiste V, Johnson WW (2017) Why human-autonomy teaming? In: International conference on applied human factors and ergonomics. Springer, pp 3–11

  133. Singhvi A, Russel K (2016) Inside the self-driving tesla fatal accident. The New York Times Magazine. https://www.nytimes.com/interactive/2016/07/01/business/inside-tesla-accident.html

  134. Soh H, Shu P, Chen M, Hsu D (2018) The transfer of human trust in robot capabilities across tasks. arXiv preprint arXiv:1807.01866

  135. Strohkorb Sebo S, Traeger M, Jung M, Scassellati B (2018) The ripple effects of vulnerability: the effects of a robot’s vulnerable behavior on trust in human–robot teams. In: Proceedings of the 2018 ACM/IEEE international conference on human-robot interaction. ACM, pp 178–186

  136. Tenhundfeld NL, de Visser EJ, Haring KS, Ries AJ, Finomore VS, Tossell CC (2019) Calibrating trust in automation through familiarity with the autoparking feature of a Tesla Model X. J Cognit Eng Decis Mak. https://doi.org/10.1177/0018720819865412

    Article  Google Scholar 

  137. Tenhundfeld NL, de Visser EJ, Ries AJ, Finomore VS, Tossell CC (2019) Trust and distrust of automated parking in a Tesla Model X. Hum Factors. https://doi.org/10.1177/0018720819865412

    Article  Google Scholar 

  138. van der Vecht B, van Diggelen J, Peeters MMM, van Staal W, van der Waa J (2018) The SAIL framework for implementing human-machine teaming concepts. International conference on practical applications of agents and multi-agent systems. Springer, Cham, pp 361–365

    Google Scholar 

  139. van der Vecht B, van Diggelen J, Peeters MMM, Barnhoorn J, van der Waa J (2018) SAIL: a social artificial intelligence layer for human–machine teaming. International conference on practical applications of agents and multi-agent systems. Springer, Cham, pp 262–274

    Google Scholar 

  140. Van der Waa J, van Diggelen J, Neerincx M (2018) The design and validation of an intuitive certainty measure. IUI 2018 workshop on explainable smart systems. In: IUI 2018 workshop on explainable smart systems. ACM

  141. Van der Waa J, van Diggelen J, Neerincx M, Raaijmakers S (2018) ICM: an intuitive, model independent and accurate certainty measure for machine learning. In: 10th internantional conference on agents and AI. ICAART

  142. Van der Waa J, Robeer M, van Diggelen J, Brinkhuis M, Neerincx M (2018) Contrastive explanations with local foil trees. In: IJCAI 2018 workshop on human interpretability in machine learning. WHI

  143. van der Waa J, van Diggelen J, van den Bosch K, Neerincx M (2018) Contrastive explanations for reinforcement learning in terms of expected consequences. Retrieved from arXiv:1807.08706

  144. Van Diggelen J, Neerincx M, Peeters M, Schraagen JM (2018) Developing effective and resilient human-agent teamwork using team design patterns. IEEE Intell Syst 34(2):15–24

    Google Scholar 

  145. van Diggelen J, Barnhoorn JS, Peeters MMM, van Staal W, van Stolk M, van der Vecht B, van der Waa J, Schraagen JM (2019) Pluggable social artificial intelligence for enabling human-agent teaming. arXiv preprint arXiv:1909.04492

  146. Verberne FMF, Ham J, Midden CJH (2012) Trust in smart systems: sharing driving goals and giving information to increase trustworthiness and acceptability of smart systems in cars. Hum Factors 54(5):799–810

    Google Scholar 

  147. Verberne FMF, Ham J, Ponnada A, Midden CJH (2013) Trusting digital chameleons: the effect of mimicry by a virtual social agent on user trust. In: International conference on persuasive technology. Springer, pp 234–245

  148. Walliser JC, de Visser EJ, Wiese E, Shaw TH (2019) Team structure and team building improve human–machine teaming with autonomous agents. J Cognit Eng Decis Mak. https://doi.org/10.1177/1555343419867563

    Article  Google Scholar 

  149. Wang N, Pynadath DV, Hill SG (2016) Trust calibration within a human-robot team: comparing automatically generated explanations. In: The eleventh ACM/IEEE international conference on human robot interaction. IEEE Press, pp 109–116

  150. Wen J, Stewart A, Billinghurst M, Dey A, Tossell C, Finomore V (2018) He who hesitates is lost (... in thoughts over a robot). In: Proceedings of the technology, mind, and society. ACM, p 43

  151. Williams M (2007) Building genuine trust through interpersonal emotion management: a threat regulation model of trust and collaboration across boundaries. Acad Manag Rev 32(2):595–621

    Google Scholar 

  152. Wiltshire TJ, Barber D, Fiore SM (2013) Towards modeling social-cognitive mechanisms in robots to facilitate human-robot teaming. In: Proceedings of the human factors and ergonomics society annual meeting, vol 57. SAGE Publications Sage, Los Angeles, CA, pp 1278–1282

  153. Wright JL, Chen JY, Lakhmani SG (2019) Agent transparency and reliability in human–robot interaction: the influence on user confidence and perceived reliability. IEEE Trans Hum Mach Syst. https://doi.org/10.1109/THMS.2019.2925717

    Article  Google Scholar 

  154. Wynne KT, Lyons JB (2018) An integrative model of autonomous agent teammate-likeness. Theor Issues Ergon Sci 19(3):353–374

    Google Scholar 

  155. Xie Y, Bodala IP, Ong DC, Hsu D, Soh H (2019) Robot capability and intention in trust-based decisions across tasks. In: 2019 14th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 39–47

  156. Xu A, Dudek G (2015) Optimo: Online probabilistic trust inference model for asymmetric human–robot collaborations. In: Proceedings of the tenth annual ACM/IEEE international conference on human–robot interaction. ACM, pp 221–228

  157. Yang XJ, Unhelkar VV, Li K, Shah JA (2017) Evaluating effects of user experience and system transparency on trust in automation. In: 2017 12th ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 408–416

Download references

Acknowledgements

This material is based upon work supported by the Air Force Office of Scientific Research under award numbers 16RT0881 and FA9550-18-1-0455, as well as the Dutch Ministry of Defence’s exploratory research program (project Human-AI Teaming).

Author information

Authors and Affiliations

  1. Warfighter Effectiveness Research Center, United States Air Force Academy, 2354 Fairchild Drive Suite 5L-50, Colorado Springs, CO, 80840, USA

    Ewart J. de Visser

  2. Department of Psychology, Clemson University, 418 Brackett Hall, Clemson, SC, 29634, USA

    Richard Pak

  3. George Mason University, 4400 University Drive 3F5, Fairfax, VA, 22030, USA

    Spencer Kohn & Tyler H. Shaw

  4. TNO, Perceptual and Cognitive Systems, Kampweg 55, 3769, DE Soesterberg, The Netherlands

    Marieke M. M. Peeters & Mark A. Neerincx

  5. Delft University of Technology, Delft, The Netherlands

    Mark A. Neerincx

  6. Department of Information Science, Cornell University, Gates Hall, 220A, Ithaca, NY, 14853, USA

    Malte F. Jung

Authors
  1. Ewart J. de Visser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marieke M. M. Peeters
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Malte F. Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Spencer Kohn
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tyler H. Shaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Richard Pak
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Mark A. Neerincx
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ewart J. de Visser.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

de Visser, E.J., Peeters, M.M.M., Jung, M.F. et al. Towards a Theory of Longitudinal Trust Calibration in Human–Robot Teams. Int J of Soc Robotics 12, 459–478 (2020). https://doi.org/10.1007/s12369-019-00596-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-019-00596-x

Keywords

  • Relationship equity
  • Social autonomy
  • Trust repair
  • Trust calibration
  • Work agreements
  • Agents
  • Social abilities
  • Human–robot interaction
  • Collaboration
  • Team