Skip to main content
SpringerLink
Log in

The Emotional, Cognitive, Physiological, and Performance Effects of Variable Time Delay in Robotic Teleoperation

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

The effects of intermittent and variable time delay were investigated to understand the cognitive and physical consequences of gaps between an input from an operator and the corresponding feedback response from the system. Time delay has been shown to disrupt task performance in various areas including psychology and telerobotics. Previous research in multiple domains has focused on the performance effects of time delay and overcoming technological limitations that cause time delay. However, robotics researchers have yet to study the effects of variable time delay on specific operator emotions, usability, and physiological activation in teleoperations. This study investigates the influence of variable time delay not only on task performance, but also operator emotions, physiological arousal, cognitive workload, and usability in teleoperation. Time delay was manipulated by introducing lag into the system feedback. Participants were asked to navigate a remote-control robot vehicle through mazes of differing levels of task complexity in a remote location and simultaneously identify targets. Results showed that operator frustration, anger, and workload increased while usability and task performance decreased when intermittent and variable feedback lag was introduced to a robotic navigation task. The effect of the variable time delay was greater than the effect of task complexity. Furthermore, results suggest that the effects are of time delay and task complexity are additive. A better understanding of the emotional experiences of human operators and the corresponding physiological signals is of crucial importance to designing affect-aware robotic systems that have the ability to appropriately mitigate negative operator emotional states.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Abramson LY, Seligman ME, Teasdale JD (1978) Learned helplessness in humans: critique and reformulation. J Abnorm Psychol 87(1):49

    Article  Google Scholar 

  2. Allison RS, Harris LR, Jenkin M, Jasiobedzka U, Zacher JE (2001) Tolerance of temporal delay in virtual environments. In: IEEE proceedings of virtual reality, pp 247–254

  3. Anderson CA, Bushman BJ (2002) Human aggression. Annu Rev Psychol 53:27–51

    Article  Google Scholar 

  4. Arcara P, Melchiorri C (2002) Control schemes for teleoperation with time delay: a comparative study. Rob Auton Syst 38(1):49–64

    Article  MATH  Google Scholar 

  5. Ash A, Palmisano S, Govan DG, Kim J (2011) Display lag and gain effects on vection experienced by active observers. Aviat Space Environ Med 82(8):763–769

    Article  Google Scholar 

  6. Baddeley A (1992) Working memory. Science 255(5044):556

  7. Barber RE, Lucas HC Jr (1983) System response time operator productivity, and job satisfaction. Commun ACM 26(11):972–986

  8. Bechara A (2004) The role of emotion in decision-making: Evidence from neurological patients with orbitofrontal damage. Brain Cogn 55(1):30–40

    Article  Google Scholar 

  9. Benţa KI, Van Kuilenburg H, Eligio UX, Den Uyl M, Cremene M, Hoszu A, Creţ O (2009) Evaluation of a system for realtime valence assessment of spontaneous facial expressions. In: Distributed environments adaptability, semantics and security issues international romanian-french workshop, Cluj-Napoca, Romania, pp 17–18

  10. Bergman H, Brinkman A, Koelega HS (1981) System response time and problem solving behavior. In: Proceedings of the human factors and ergonomics society annual meeting, vol 25, No. 1. SAGE Publications, Thousand Oaks, pp 749–753

  11. Boucsein W (2012) Electrodermal activity. Springer Science & Business Media, New York

    Book  Google Scholar 

  12. Bourne LE Jr (1957) Effects of delay of information feedback and task complexity on the identification of concepts. J Exp Psychol 54(3):201

    Article  Google Scholar 

  13. Bradley MM, Lang PJ (2000) Measuring emotion: behavior, feeling, and physiology. Cogn Neurosci Emot 25:49–59

    Google Scholar 

  14. Braithwaite JJ, Watson DG, Jones R, Rowe M (2013) A guide for analysing electrodermal activity (EDA) and skin conductance responses (SCRs) for psychological experiments. Psychophysiology 49:1017–1034

    Google Scholar 

  15. Burleson W, Picard RW (2004) Affective agents: sustaining motivation to learn through failure and a state of stuck. In: Workshop on social and emotional intelligence in learning environments

  16. Burridge RR, Hambuchen KA (2009) Using prediction to enhance remote robot supervision across time delay. In IEEE/RSJ international conference on intelligent robots and systems, pp 5628–5634

  17. Carlson NR (2013) Physiology of behavior. Pearson, Boston

    Google Scholar 

  18. Casals A (1998) Robots in surgery. In: Autonomous robotic systems. Springer, London, pp 222–234

  19. Ceaparu I, Lazar J, Bessiere K, Robinson J, Shneiderman B (2004) Determining causes and severity of end-user frustration. IntJ Human Comput Interact 17(3):333–356

    Article  Google Scholar 

  20. Chen JY, Haas EC, Barnes MJ (2007) Human performance issues and user interface design for teleoperated robots. IEEE Trans Syst Man Cybern C 37(6):1231–1245

    Article  Google Scholar 

  21. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Erlbaum, New Jersey

  22. Conesa J (1995) Electrodermal palmar asymmetry and nostril dominance. Percept Mot Skills 80(1):211–216

    Article  Google Scholar 

  23. Cootes TF, Taylor CJ (2004) Statistical models of appearance for computer vision. Technical report, University of Manchester, Wolfson Image Analysis Unit, Imaging Science and Biomedical Engineering

  24. Corde Lane J, Carignan CR, Sullivan BR, Akin DL, Hunt T, Cohen R (2002) Effects of time delay on telerobotic control of neutral buoyancy vehicles. In: Proceedings of IEEE international conference on robotics and automation, ICRA’02, vol 3, pp 2874–2879

  25. Davis J, Smyth C, McDowell K (2010) The effects of time lag on driving performance and a possible mitigation. IEEE Trans Rob 26(3):590–593

    Article  Google Scholar 

  26. De Silva LC, Miyasato T, Nakatsu R (1997) Facial emotion recognition using multi-modal information. In: Proceedings of 1997 international conference on information, communications and signal processing, ICICS, vol 1, pp 397–401

  27. De Silva LC, Ng PC (2000) Bimodal emotion recognition. In: Proceedings of fourth IEEE international conference on automatic face and gesture recognition, pp 332–335

  28. Den Uyl MJ, Van Kuilenburg H (2008) The FaceReader: online facial expression recognition. In: Proceedings of measuring behavior 2005, Wageningen, The Netherlands, August 30–September 2, 2008, pp 589–590

  29. Dollard J, Miller NE, Doob LW, Mowrer OH, Sears RR (1939) Frustration and aggression. Yale University Press, New Haven

    Book  Google Scholar 

  30. Dorneich MC, Ververs PM, Mathan S, Whitlow S, Hayes CC (2012) Considering etiquette in the design of an adaptive system. J Cogn Eng Decis Mak 6(2):243–265

    Article  Google Scholar 

  31. Draper JV (1993) Human factors in telemanipulation: perspectives from the Oak Ridge National Laboratory experience. In: International society for optics and photonics optical tools for manufacturing and advanced automation, pp 162–174

  32. Drascic D, Milgram P, Grodski J (1989) Learning effects in telemanipulation with monoscopic versus stereoscopic remote viewing. In: Proceedings of IEEE international conference on systems, man and cybernetics, pp 1244–1249

  33. Endsley MR, Kiris EO (1995) The out-of-the-loop performance problem and level of control in automation. Hum Factors 37(2):381–394

    Article  Google Scholar 

  34. Ekman P (1970) Universal facial expressions of emotion. California Mental Health Research Digest 8:151–158

    Google Scholar 

  35. Feigh KM, Dorneich MC, Hayes CC (2012) Towards a characterization of adaptive systems: a framework for researchers and system designers. J Human Factors Ergon 54(6):1008–1024

    Article  Google Scholar 

  36. Ferrell WR (1965) Remote manipulation with transmission delay. IEEE Trans Human Factors Electron 1:24–32

    Article  Google Scholar 

  37. Fisher RA (1921) On the probable error of a coefficient of correlation deduced from a small sample. Metron 1:3–32

    Google Scholar 

  38. Fogg BJ (1998) Persuasive computers: perspectives and research directions. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press/Addison-Wesley Publishing Co, New York, pp 225–232

  39. Fogg BJ (2002) Persuasive technology: using computers to change what we think and do. Ubiquity 2002 (December)

  40. Fong T, Thorpe C, Baur C (2003) Multi-robot remote driving with collaborative control. IEEE Trans Industr Electron 50(4):699–704

    Article  Google Scholar 

  41. Frijda NH (1986) The emotions. Cambridge University Press, Cambridge

    Google Scholar 

  42. Gelman A (2013) Commentary: P values and statistical practice. Epidemiology 24(1):69–72

    Article  Google Scholar 

  43. Gross J J (1998) Antecedent-and response-focused emotion regulation: divergent consequences for experience, expression, and physiology. J Personal Soc Psychol 74(1):224

  44. Gross JJ (2002) Emotion regulation: affective, cognitive, and social consequences. Psychophysiology 39(3):281–291

    Article  Google Scholar 

  45. Guynes JL (1988) Impact of system response time on state anxiety. Commun ACM 31(3):342–347

  46. Hambuchen K, Bluethmann W, Goza M, Ambrose R, Rabe K, Allan M (2006) Supervising remote humanoids across intermediate time delay. In: 2006 6th IEEE-RAS international conference on humanoid robots, pp 246–251

  47. Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol 52:139–183

    Article  Google Scholar 

  48. Hazlett R (2003) Measurement of user frustration: a biologic approach. In: CHI’03 extended abstracts on Human factors in computing systems. ACM, New York, pp 734–735

  49. Held R, Durlach N (1991) Telepresence, time delay and adaptation. Pictorial communication in virtual and real environments, pp 232–246

  50. Hoxmeier JA, DiCesare C (2000) System response time and user satisfaction: An experimental study of browser-based applications. In: Proceedings of AMCIS 2000

  51. Hokayem PF, Spong MW (2006) Bilateral teleoperation: an historical survey. Automatica 42(12):2035–2057

    Article  MathSciNet  MATH  Google Scholar 

  52. Imaida T, Yokokohji Y, Doi T, Oda M, Yoshikawa T (2004) Ground-space bilateral teleoperation of ETS-VII robot arm by direct bilateral coupling under 7-s time delay condition. IEEE Trans Robot Autom 20(3):499–511

    Article  Google Scholar 

  53. Jeon M, Yim JB, Walker BN (2011) An angry driver is not the same as a fearful driver: effects of specific negative emotions on risk perception, driving performance, and workload. In Proceedings of the 3rd international conference on automotive user interfaces and interactive vehicular applications. ACM, New York, pp 137–142

  54. Kaber DB, Riley JM, Zhou R, Draper J (2000) Effects of visual interface design, and control mode and latency on performance, telepresence and workload in a teleoperation task. In: Proceedings of the human factors and ergonomics society annual meeting, vol 44, No. 5. SAGE Publications, Thousand Oaks, pp 503–506

  55. Kiesler S, Zubrow D, Moses AM, Geller V (1985) Affect in computer-mediated communication: an experiment in synchronous terminal-to-terminal discussion. Hum Comput Interact 1(1):77–104

    Article  Google Scholar 

  56. Klein J, Moon Y, Picard RW (2002) This computer responds to user frustration: theory, design, and results. Interact Comput 14(2):119–140

    Article  Google Scholar 

  57. Kramer AF (1991) Physiological metrics of mental workload: a review of recent progress. Multiple-task performance, pp 279–328

  58. Kuhmann W (1989) Experimental investigation of stress inducing properties of system response times. Ergonomics 32(3):271–280

    Article  Google Scholar 

  59. Kuhmann W, Boucsein W, Schaefer F, Alexander J (1987) Experimental investigation of psychophysiological stress-reactions induced by different system response times in human-computer interaction*. Ergonomics 30(6):933–943

    Article  Google Scholar 

  60. Kulic D, Croft E (2007) Pre-collision safety strategies for human robot interaction. Auton Robots 22(2):149–164

    Article  Google Scholar 

  61. Lang PJ, Greenwald MK, Bradley MM, Hamm AO (1993) Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30(3):261–273

    Article  Google Scholar 

  62. Lang P (1995) The emotion probe: studies of motivation and attention. Am Psychol 50(5):372

    Article  Google Scholar 

  63. Lawson R (1965) Frustration the development of a scientific concept. Macmillan, New York

    Google Scholar 

  64. Lazar J, Jones A, Bessiere K, Ceaparu I, Shneiderman B (2005) User frustration with technology in the workplace

  65. LeDoux J (1998) The emotional brain: the mysterious underpinnings of emotional life. Simon and Schuster, New York

    Google Scholar 

  66. Leon E, Clarke G, Callaghan V, Doctor F (2010) Affect-aware behaviour modelling and control inside an intelligent environment. Pervasive Mob Comput 6(5):559–574

    Article  Google Scholar 

  67. Li L, Chen JH (2006) Emotion recognition using physiological signals. In: Advances in artificial reality and tele-existence. Springer, Berlin, pp 437–446

  68. Lindquist KA, Barrett LF (2008) Emotional complexity. Handbook of emotions, pp 513–530

  69. Loijens L, Krips O, Van Kuilenbug H, Den Uyl M, Ivan P, Theuws H, Spink A (2012) FaceReader reference manual version 5.0, Noldus Information Technology b.v

  70. Luck JP, McDermott PL, Allender L, Russell DC (2006) An investigation of real world control of robotic assets under communication latency. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction. ACM, New York, pp 202–209

  71. Maier SF, Seligman ME (1976) Learned helplessness: theory and evidence. J Exp Psychol Gen 105(1):3

    Article  Google Scholar 

  72. Martin GL, Corl KG (1986) System response time effects on user productivity. Behav Inform Technol 5(1):3–13

    Article  Google Scholar 

  73. Mata-Cervantes G, Westerman S, Burke MR, Hill A, Wyatt J (2014) Using psychophysiology to study the emotional impact of words used in behaviour change text messages. Int J Integr Care 14(8)

  74. Meehan M, Razzaque S, Whitton MC, Brooks FP Jr (2003) Effect of latency on presence in stressful virtual environments. In: IEEE Proceedings of virtual reality, pp 141–148

  75. Myers L, Sirois MJ (2006) Spearman correlation coefficients, differences between. Encyclopedia of Statistical Sciences, Wiley, New York. doi:10.1002/9781118445112.stat02802

  76. Miller CA, Dorneich MC (2006) From associate systems to augmented cognition: 25 years of user adaptation in high criticality systems. In: Proceedings of the 2nd augmented cognition international, San Francisco, CA

  77. Miller RB (1968) Response time in man-computer conversational transactions. In: Proceedings of the December 9–11, 1968, fall joint computer conference, part I. ACM, New York, pp 267–277

  78. Nasoz F, Alvarez K, Lisetti CL, Finkelstein N (2004) Emotion recognition from physiological signals using wireless sensors for presence technologies. Cogn Technol Work 6(1):4–14

    Article  Google Scholar 

  79. Nass C, Fogg BJ, Moon Y (1996) Can computers be teammates? Int J Hum Comput Stud 45(6):669–678

    Article  Google Scholar 

  80. Nass C, Jonsson IM, Harris H, Reaves B, Endo J, Brave S, Takayama L (2005, April) Improving automotive safety by pairing driver emotion and car voice emotion. In: CHI’05 extended abstracts on human factors in computing systems. ACM, New York, pp 1973–1976

  81. Nass C, Moon Y, Fogg BJ, Reeves B, Dryer DC (1995) Can computer personalities be human personalities? Int J Hum Comput Stud 43(2):223–239

    Article  Google Scholar 

  82. Nelson WT, Roe MM, Bolia RS, Morley RM (2000) Assessing simulator sickness in a see-through HMD: Effects of time delay, time on task, and task complexity (No. ASC-00-1047). Air Force Research Lab Wright-Patterson AFB OH

  83. Nonami K, Shimoi N, Huang QJ, Komizo D, Uchida H (2000) Development of teleoperated six-legged walking robot for mine detection and mapping of mine field. In: IEEE/RSJ international conference on intelligent robots and systems (IROS 2000), vol 1, pp 775–779

  84. Octavia JR, Raymaekers C, Coninx K (2011) Adaptation in virtual environments: conceptual framework and user models. Multimed Tools Appl 54(1):121–142

    Article  Google Scholar 

  85. Owen-Hill A, Suárez-Ruiz F, Ferre M, Aracil R (2014) Effect of video quality and buffering delay on telemanipulation performance. In: ROBOT2013: first iberian robotics conference. Springer International Publishing, pp 555–568

  86. Patten CJ, Kircher A, Östlund J, Nilsson L (2004) Using mobile telephones: cognitive workload and attention resource allocation. Accid Anal Prev 36(3):341–350

    Article  Google Scholar 

  87. Picard RW, Klein J (2002) Computers that recognize and respond to user emotion: theoretical and practical implications. Interact Comput 14(2):141–169

    Article  Google Scholar 

  88. Prewett MS, Johnson RC, Saboe KN, Elliott LR, Coovert MD (2010) Managing workload in human-robot interaction: a review of empirical studies. Comput Hum Behav 26(5):840–856

    Article  Google Scholar 

  89. Ridao P, Carreras M, Hernandez E, Palomeras N (2007) Underwater telerobotics for collaborative research. In: Advances in telerobotics. Springer, Berlin, pp 347–359

  90. Rosenthal-von der Pütten AM, Krämer NC, Hoffmann L, Sobieraj S, Eimler SC (2013) An experimental study on emotional reactions towards a robot. Int J Soc Robot 5(1):17–34

  91. Russell JA, Fehr B (1994) Fuzzy concepts in a fuzzy hierarchy: varieties of anger. J Pers Soc Psychol 67:186–205

    Article  Google Scholar 

  92. Satava RM, Simon IB (1993) Teleoperation, telerobotics, and telepresence in surgery. Endosc Surg Allied Technol 1(3):151–153

    Google Scholar 

  93. Scerbo MW, Freeman FG, Mikulka PJ (2003) A brain-based system for adaptive automation. Theor Issues Ergon Sci 4:200–219

    Article  Google Scholar 

  94. Scheirer J, Fernandez R, Klein J, Picard RW (2002) Frustrating the user on purpose: a step toward building an affective computer. Interact Comput 14(2):93–118

    Article  Google Scholar 

  95. Scherer KR (2001) The nature and study of appraisal: a review of the issues. Theory, methods, research, Appraisal processes in emotion, pp 369–391

  96. Schleifer LM, Amick III BC (1989) System response time and method of pay: stress effects in computer-based tasks. Int J Hum Comput Interact 1(1):23–39

  97. Schlosberg H (1954) Three dimensions of emotion. Psychol Rev 61(2):81

    Article  Google Scholar 

  98. Schwarz N (2000) Emotion, cognition, and decision making. Cogn Emot 14(4):433–440

    Article  MathSciNet  Google Scholar 

  99. Selvidge PR, Chaparro BS, Bender GT (2002) The world wide wait: effects of delays on user performance. Int J Ind Ergon 29(1):15–20

    Article  Google Scholar 

  100. Sheik-Nainar MA, Kaber DB, Chow MY (2005) Control gain adaptation in virtual reality mediated human-telerobot interaction. Human Fact Ergon Manuf Serv Ind 15(3):259–274

    Article  Google Scholar 

  101. Sheridan TB (1992) Telerobotics, automation, and human supervisory control. MIT Press, Cambridge

    Google Scholar 

  102. Sheridan TB (1993) Space teleoperation through time delay: review and prognosis. IEEE Trans Robot Autom 9(5):592–606

    Article  Google Scholar 

  103. Solkoff N, Todd GA, Screven CG (1964) Effects of frustration on perceptual-motor performance. Child Development, pp 569–575

  104. Spector PE (1975) Relationships of organizational frustration with reported behavioral reactions of employees. J Appl Psychol 60(5):635–637

    Article  Google Scholar 

  105. Squire LR (1987) Memory and brain. New York

  106. Stokes MT (1990) Time in human-computer interaction: performance as a function of delay type, delay duration, and task difficulty (Doctoral dissertation, Texas Tech University)

  107. Szameitat AJ, Rummel J, Szameitat DP, Sterr A (2009) Behavioral and emotional consequences of brief delays in human-computer interaction. Int J Hum Comput Stud 67(7):561–570

    Article  Google Scholar 

  108. Thum M, Boucsein W, Kuhmann W (1995) Standardized task strain and system response times in human-computer interaction. Ergonomics 38(7):1342–1351

    Article  Google Scholar 

  109. Voeffray S (2011). Emotion-sensitive human-computer interaction (HCI): state of the art-seminar paper. Emotion Recognition, pp 1–4

  110. Waterhouse IK, Child IL (1953) Frustration and the quality of performance. J Pers 21(3):298–311

    Article  Google Scholar 

  111. Wiethoff MAAG, Arnold AG, Houwing EM (1991) The value of psychophysiological measures in human-computer interaction. Human-aspects in computing: design and use of interactive systems and work with terminals. Elsevier, Amsterdam, pp 661–665

  112. Woolf B, Burleson W, Arroyo I, Dragon T, Cooper D, Picard R (2009) Affect-aware tutors: recognising and responding to student affect. Int J Learn Technol 4(3–4):129–164

    Article  Google Scholar 

  113. Yang E, Dorneich MC (2015) The effect of time delay on emotion, arousal, and satisfaction in human-robot interaction. In: Proceedings of the human factors and ergonomics society annual meeting, vol 59, No. 1. SAGE Publications, pp 443–447

  114. Zhong P (2013) Perception in remote navigation. Ph.D. Dissertation, Iowa State University

  115. Zoghbi S, Croft E, Kulic D, Van der Loos M (2009) Evaluation of affective state estimations using an on-line reporting device during human-robot interactions. In: Proceedings of the 2009 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 3742–3749

Download references

Acknowledgements

The authors would like to acknowledge the efforts of Dr. Peihan Zhong and Dr. Richard Stone for supporting the experiment robot devices. In addition, the authors would like to thank Chase Meusel for providing advice on the EDA sensor data analysis. Finally, the authors would like to thank Leslie Potter, Dr. Richard Stone, and the reviewers for feedback on drafts of the manuscript. Portions of this work appeared in preliminary form in [113].

Author information

Authors and Affiliations

  1. Industrial and Manufacturing Systems Department, Iowa State University, 3004 Black Engineering Building, Ames, IA, 50011-2164, USA

    Euijung Yang & Michael C. Dorneich

Authors
  1. Euijung Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael C. Dorneich
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael C. Dorneich.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, E., Dorneich, M.C. The Emotional, Cognitive, Physiological, and Performance Effects of Variable Time Delay in Robotic Teleoperation. Int J of Soc Robotics 9, 491–508 (2017). https://doi.org/10.1007/s12369-017-0407-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-017-0407-x

Keywords

Search

Navigation