Advertisement

Evaluation Metrics Regarding Human Well-Being and System Performance in Human-Robot Interaction – A Literature Review

  • Jochen Nelles
  • Sonja Th. Kwee-Meier
  • Alexander Mertens
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 825)

Abstract

This literature review provides an overview on evaluation metrics regarding human well-being and system performance in human-robot interaction. In this context a systematic literature search in the Web of Science and IEEE Xplore databases was carried out. Thus, 30 relevant contributions out of 3854 studies were analyzed by multistage filtering. To gain an overview on and compare the different approaches and results, the studies are summarized in tables according to the following criteria: author, year, title, task, study design, measurement methods, population, and results. The evaluation metrics presented in this contribution in principle can be divided into questionnaire-based surveys and psychophysiological measurement methods. In addition, the studies are classified into evaluation metrics for measuring well-being and performance.

The research was carried out with regard to an industrial engineering and ergonomics context, but is independent of specific application areas.

Keywords

Evaluation Human-robot interaction Literature review Metrics Performance Taxonomy Well-Being 

Notes

Acknowledgments

The literature review was performed within the research project “DUCHBLICK” (Grant No. 13N14329), funded by the German Federal Ministry of Education and Research (BMBF) in the context of the national program “Research for Civil Security” and the call “Civil Security – Aspects and Measures of Coping with Terrorism”. In addition, this publication is part of the research project “MeRoSy” (Grant No. 16SV7190), which is also funded by the German Federal Ministry of Education and Research (BMBF).

References

  1. 1.
    Onnasch L, Maier X, Jürgensohn T (2016) Mensch-Roboter-Interaktion – Eine Taxonomie für alle Anwendungsfälle. 1. Edition. Dortmund: Bundesanstand für Arbeitsschutz und Arbeitsmedizin (baua Fokus), project number: F 2369, PDF file.  https://doi.org/10.21934/baua:fokus20160630 (in german)
  2. 2.
    ISO 9241-110 (2006) Ergonomics of human-system interaction – Part 110: Dialogue principlesGoogle Scholar
  3. 3.
    ISO 9241-210 (2010) Ergonomics of human-system interaction – Part 210: Human-centered design for interactive systemsGoogle Scholar
  4. 4.
    Nelles J, Brandl C, Mertens A (2018, in German) Regelkreismodell für die menschzentrierte Gestaltung und Evaluierung einer Mensch-Roboter-Interaktion am Beispiel eines Mensch-Roboter-Arbeitsplatzes. In: Arbeit(s)Wissenschaf(f)t – Grundlage für Management und Kompetenzentwicklung. 64. Kongress der Gesellschaft für ArbeitswissenschaftGoogle Scholar
  5. 5.
    Schlick CM, Bruder R, Luczak H (2010, in German) Arbeitswissenschaft. Springer, HeidelbergGoogle Scholar
  6. 6.
    Dul J, Bruder R, Buckle P, Carayon P, Falzon P, Marras WS et al (2012) A strategy for human factors/ergonomics: developing the discipline and profession. Ergonomics 55(4):377–395.  https://doi.org/10.1080/00140139.2012.661087CrossRefGoogle Scholar
  7. 7.
    Bortot D, Born M, Bengler K (2013) Directly or on detours? How should industrial robots approximate humans? In: Kuzuoka H (ed) 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 3–6 March 2013, Tokyo, Japan (including workshop papers). IEEE, Piscataway, pp 89–90Google Scholar
  8. 8.
    Charalambous G, Fletcher S, Webb P (2016) The development of a scale to evaluate trust in industrial human-robot collaboration. Int J Soc Robot 8(2):193–209Google Scholar
  9. 9.
    Daniel B, Thomessen T, Korondi P (2013) Simplified human-robot interaction: modeling and evaluation. Model. Identif. Control Norw. Res. Bull. 34(4):199–211Google Scholar
  10. 10.
    Dehais F, Sisbot EA, Alami R, Causse M (2011) Physiological and subjective evaluation of a human-robot object hand-over task. Appl Ergon 42(6):785–791Google Scholar
  11. 11.
    Hancock PA, Billings DR, Schaefer KE, Chen JYC, de Visser EJ, Parasuraman R (2011) A meta-analysis of factors affecting trust in human-robot interaction. Hum Factors 53(5):517–527Google Scholar
  12. 12.
    Kessler TT, Larios C, Walker T, Yerdon V, Hancock PA (2017) A comparison of trust measures in human–robot interaction scenarios. In: Savage-Knepshield P, Chen J (eds) Advances in human factors in robots and un-manned systems, Proceedings of the AHFE 2016 international conference on human factors in robots and unmanned systems, 27–30 July 2016, Walt Disney World®, Florida, USA. (Advances in Intelligent Systems and Computing, vol 499, pp 353–364. Springer, ChamGoogle Scholar
  13. 13.
    Liu D, Kinugawa J, Kosuge K (2016) A projection-based making-human-feel-safe system for human-robot cooperation. In: Proceedings of 2016 IEEE international conference on mechatronics and automation, 7–10 August 2016, Harbin, China, pp 1101–1106. IEEE, PiscatawayGoogle Scholar
  14. 14.
    Maurtua I, Pedrocchi N, Orlandini A, Fernandez JG, Vogel C, Geenen A, Althoefer K, Shafti A (2016) FourByThree: imagine humans and robots working hand in hand. In: 2016 IEEE 21st international conference on emerging technologies and factory automation (ETFA), 6–9 September 2016, Berlin, Germany, pp 1–8. IEEE, PiscatawayGoogle Scholar
  15. 15.
    Murata A (2000) Ergonomics and cognitive engineering for robot-human cooperation. In: Proceedings of the 9th IEEE international workshop on robot and human interactive communication, IEEE RO-MAN 2000, 27–29 September 2000, pp 206–211. Osaka Institute of Technology, Osaka, IEEE Service Center, PiscatawayGoogle Scholar
  16. 16.
    Murphy RR, Schreckenghost D (2013) Survey of metrics for human-robot interaction. In: Kuzuoka H (ed) 2013 8th ACM/IEEE international conference on human-robot interaction (HRI) (including workshop papers), 3–6 March 2013, Tokyo, Japan, pp 197–198. IEEE, PiscatawayGoogle Scholar
  17. 17.
    Nomura T, Kanda T, Suzuki T, Kato K (2004) Psychology in human-robot communication: an attempt through investigation of negative attitudes and anxiety toward robots. In: Proceedings of the 13th IEEE international workshop on robot and human interactive communication, RO-MAN 2004, 20–22 September 2004, Kurashiki, Okayama, Japan, pp 35–40. IEEE Operations Center, PiscatawayGoogle Scholar
  18. 18.
    Novak D, Mihelj M, Munih M (2011) Psychophysiological responses to different levels of cognitive and physical workload in haptic interaction. Robotica 29(03):367–374Google Scholar
  19. 19.
    Ogorodnikova O (2008) Methodology of safety for a human robot interaction designing stage. In: Conference on human system interactions, HSI 2008, 25–27 May 2008, Krakow, Poland, pp 452–457. IEEE, PiscatawayGoogle Scholar
  20. 20.
    Pini F, Ansaloni M, Leali F (2016) Evaluation of operator relief for an effective design of HRC workcells. In: 2016 IEEE 21st international conference on emerging technologies and factory automation (ETFA), 6–9 September 2016, Berlin, Germany, pp 1–6. IEEE, PiscatawayGoogle Scholar
  21. 21.
    Profanter S, Perzylo A, Somani N, Rickert M, Knoll A (2015) Analysis and semantic modeling of modality preferences in industrial human-robot interaction. In: Burgard W (ed) 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), 28 September 2015–2 October 2015, Hamburg, Germany, pp 1812–1818. IEEE, PiscatawayGoogle Scholar
  22. 22.
    Rahman SMM, Sadrfaridpour B, Wang Y (2016) Trust-based optimal subtask allocation and model predictive control for human-robot collaborative assembly in manufacturing. In: Proceedings of the ASME 8th annual dynamic systems and control conference – 2015, presented at ASME 2015 8th annual dynamic systems and control conference, 28–30 October 2015, Columbus, OH, USA. The American Society of Mechanical Engineers, New YorkGoogle Scholar
  23. 23.
    Sadrfaridpour B, Saeidi H, Wang Y (2016) An integrated framework for human-robot collaborative assembly in hybrid manufacturing cells. In: 2016 IEEE international conference on automation science and engineering (CASE), pp 462–467. IEEEGoogle Scholar
  24. 24.
    Sanders TL, Wixon T, Schafer KE, Chen JYC, Hancock PA (2014) The influence of modality and transparency on trust in human-robot interaction. In: 2014 IEEE international inter-disciplinary conference on cognitive methods in situation awareness and decision support (CogSIMA), 3–6 March 2014, San Antonio, TX, USA, pp 156–159. IEEE, PiscatawayGoogle Scholar
  25. 25.
    Sim DYY, Loo CK (2015) Extensive assessment and evaluation methodologies on assistive social robots for modelling human–robot interaction – a review. Inf Sci 301:305–344Google Scholar
  26. 26.
    Tan JTC, Duan F, Zhang Y, Arai T (2008) Task decomposition of cell production assembly operation for man-machine collaboration by HTA. In: IEEE international conference on automation and logistics, ICAL 2008, 1–3 September 2008, Qingdao, China, pp 1066–1071. IEEE, PiscatawayGoogle Scholar
  27. 27.
    Tan J, Duan F, Zhang Y, Arai T (2009) Extending task analysis in HTA to model man-machine collaboration in cell production. In: IEEE international conference on robotics and biomimetics, ROBIO 2008, 22–25 February 2009, Bangkok, Thailand (rescheduled from December 2008, original conference date: 14–17 December 2008), pp 542–547. IEEE, PiscatawayGoogle Scholar
  28. 28.
    Tan JTC, Duan F, Zhang Y, Watanabe K, Kato R, Arai T (2009) Human-robot collaboration in cellular manufacturing: design and development. In: 2009 IEEE/RSJ international conference on intelligent robots and systems, IROS 2009, St. Louis, Missouri, USA, 10–15 (i.e. 11–15) October 2009, pp 29–34. IEEE, PiscatawayGoogle Scholar
  29. 29.
    Tan JTC, Zhang Y, Duan F, Watanabe K, Kato R, Arai T (2009) Human factors studies in information support development for human-robot collaborative cellular manufacturing system. In: The 18th IEEE international symposium on robot and human interactive communication, RO-MAN 2009, 27 September 2009–2 October 2009, Toyama, Japan, pp 334–339. IEEE, PiscatawayGoogle Scholar
  30. 30.
    Tan JTC, Arai T (2010) Analytic evaluation of human-robot system for collaboration in cellular manufacturing system. In: IEEE/ASME international conference on advanced intelligent mechatronics (AIM), 6–9 July 2010, Montreal, QC, Canada, pp 515–520. IEEE, PiscatawayGoogle Scholar
  31. 31.
    Tan JTC, Arai T (2010) Information support development with human-centered approach for human-robot collaboration in cellular manufacturing. In: IEEE RO-MAN, 19th IEEE international symposium on robot and human interactive communication, 13–15 September 2010, Viareggio, Italy, pp 767–772. IEEE, PiscatawayGoogle Scholar
  32. 32.
    Weiss A, Bernhaupt R, Lankes M (eds) (2009) The USUS evaluation framework for human-robot interaction AISB2009: proceedings of the symposium on new frontiers in human-robot interactionGoogle Scholar
  33. 33.
    Weiss A, Wurhofer D, Bernhaupt R, Altmaninger M, Tscheligi M (2010) A methodological adaptation for heuristic evaluation of HRI. In: IEEE RO-MAN, 19th IEEE international symposium on robot and human interactive communication, 13–15 September 2010, Viareggio, Italy, pp 1–6. IEEE, PiscatawayGoogle Scholar
  34. 34.
    Weiss A, Huber A, Minichberger J, Ikeda M (2016) First application of robot teaching in an existing industry 4.0 environment. Does it really work? Societies 6(3):20Google Scholar
  35. 35.
    Weistroffer V, Paljic A, Fuchs P, Hugues O, Chodacki J-P, Ligot P, Morais A (2014) Assessing the acceptability of human-robot co-presence on assembly lines: a comparison between actual situations and their virtual reality counterparts. In: RO-MAN, The 23rd IEEE international symposium on robot and human interactive communication, 25–29 August 2014, Edinburgh, Scotland, UK, pp 377–384. IEEE, PiscatawayGoogle Scholar
  36. 36.
    Yagoda RE, Gillan DJ (2012) You want me to trust a ROBOT? The development of a human–robot interaction trust scale. Int J Soc Robot 4(3):235–248Google Scholar
  37. 37.
    Brooke J (2013) SUS: a retrospective. J Usability Stud 8(2):29–40Google Scholar
  38. 38.
    Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol 52:139–183Google Scholar
  39. 39.
    Jian J-Y, Bisantz AM, Drury CG (2000) Foundations for an empirically deter-mined scale of trust in automated systems. Int J Cognit Ergon 4(1):53–71Google Scholar
  40. 40.
    McBridge SE (2010) The effect of workload and age on compliance with and reliance on an automated system. Georgia Institute of Technology, Ph.D. thesisGoogle Scholar
  41. 41.
    Schaefer KE (2013) The perception and measurement of human-robot trust. Ph.D. thesis, University of Central Florida, FloridaGoogle Scholar
  42. 42.
    Spielberger C, Gorusch R (1970) State-trait anxiety inventory, manual for the state-trait anxiety inventory. Consulting Psychologist PressGoogle Scholar
  43. 43.
    Kwee-Meier S, Nelles J, Mertens A, Schlick C (2017, in German) Integrierte Informationsvisualisierungen bei der Mensch-Roboter-Kooperation zur Unterstützung von Einsatzkräften unter erhöhter psychischer Belastung. In: Soziotechnische Gestaltung des digitalen Wandels – kreativ, innovative, sinnhaft. 63. Kongress der Gesellschaft für ArbeitswissenschaftGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Institute of Industrial Engineering and ErgonomicsRWTH Aachen UniversityAachenGermany

Personalised recommendations