Skip to main content

HRI Physio Lib: A Software Framework to Support the Integration of Physiological Adaptation in HRI

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12483)

Abstract

The rise of available physiological sensors in recent years can be attributed to the widespread popularity of commercial sensing technologies equipped with health monitoring technology. To make more engaging human-robot interaction (HRI), social robots should have some ability to infer their social partner’s affective state. Measurements of the autonomic nervous system via non-invasive physiological sensors provide a convenient window into a person’s affective state, namely their emotions, behavior, stress, and engagement. HRI research has included physiological sensors in-the-loop, however implementations are often specific to studies, and do not lend well for reusability. To address this gap, we propose a modular, flexible and extensible framework designed to work with popular robot platforms. Our framework will be compatible with both lab and consumer grade sensors, and includes essential tools and processing algorithms for affective state estimation geared towards real-time HRI applications.

Keywords

  • Physiological sensors
  • Software framework
  • Human-robot interaction
  • Affective computing

This research was undertaken, in part, thanks to funding from the Canada 150 Research Chairs Program and thanks to Alexander Graham Bell Canada Graduate Scholarships.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-62056-1_4
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-62056-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   139.99
Price excludes VAT (USA)
Fig. 1.

References

  1. EventIDE (2020). http://www.okazolab.com/

  2. iMotions: Unpack Human Behaviour (2020). https://imotions.com/

  3. Barco, A., Albo-Canals, J., Garriga, C.: Engagement based on a customization of an ipod-lego robot for a long-term interaction for an educational purpose. In: Proceeding of ACM/IEEE HRI 2014, pp. 124–125. IEEE (2014)

    Google Scholar 

  4. Batista, D., da Silva, H.P., Fred, A., Moreira, C., Reis, M., Ferreira, H.A.: Benchmarking of the bitalino biomedical toolkit against an established gold standard. Healthc. Technol. Lett. 6(2), 32–36 (2019)

    CrossRef  Google Scholar 

  5. Bian, D., Wade, J., Swanson, A., Weitlauf, A., Warren, Z., Sarkar, N.: Design of a physiology-based adaptive virtual reality driving platform for individuals with ASD. ACM Trans. Accessible Comput. (TACCESS) 12(1), 2 (2019)

    Google Scholar 

  6. Cacioppo, J.T., Tassinary, L.G., Berntson, G.: Handbook of Psychophysiology. Cambridge University Press, Cambridge (2007)

    Google Scholar 

  7. Caine, K., Šabanovic, S., Carter, M.: The effect of monitoring by cameras and robots on the privacy enhancing behaviors of older adults. In: Proceedings of ACM/IEEE HRI, vol. 2012, pp. 343–350 (2012)

    Google Scholar 

  8. De Waal, F.B.: Putting the altruism back into altruism: the evolution of empathy. Annu. Rev. Psychol. 59, 279–300 (2008)

    CrossRef  Google Scholar 

  9. Del Duchetto, F., Baxter, P., Hanheide, M.: Are you still with me? continuous engagement assessment from a robot’s point of view. arXiv preprint arXiv:2001.03515 (2020)

  10. Dillen, N., Ilievski, M., Law, E., Nacke, L.E., Czarnecki, K., Schneider, O.: Keep calm and ride along: passenger comfort and anxiety as physiological responses to autonomous driving styles. In: Proceeding of CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2020)

    Google Scholar 

  11. D’Mello, S., et al.: A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In: International Conference on Intelligent Tutoring Systems, pp. 245–254 (2010)

    Google Scholar 

  12. Essa, I.A., Pentland, A.P.: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 757–763 (1997)

    CrossRef  Google Scholar 

  13. Fairclough, S., Gilleade, K.: Construction of the biocybernetic loop: a case study. In: Proceeding of ACM ICMI, pp. 571–578 (2012)

    Google Scholar 

  14. Fairclough, S.H.: Fundamentals of physiological computing. Interact. Comput. 21(1–2), 133–145 (2009)

    CrossRef  Google Scholar 

  15. Fan, J., et al.: A robotic coach architecture for elder care (rocare) based on multi-user engagement models. IEEE Trans. Neural Syst. Rehabil. Eng. 25(8), 1153–1163 (2016)

    CrossRef  Google Scholar 

  16. Fan, Y., Lu, X., Li, D., Liu, Y.: Video-based emotion recognition using CNN-RNN and c3d hybrid networks. In: Proceeding of ACM ICMI, vol. 2016, pp. 445–450 (2016)

    Google Scholar 

  17. Fiorini, L., Mancioppi, G., Semeraro, F., Fujita, H., Cavallo, F.: Unsupervised emotional state classification through physiological parameters for social robotics applications. Knowl.-Based Syst. 190, 105217 (2020)

    CrossRef  Google Scholar 

  18. Foster, M.E., Gaschler, A., Giuliani, M.: Automatically classifying user engagement for dynamic multi-party human-robot interaction. Int. J. Soc. Robot. 9(5), 659–674 (2017)

    CrossRef  Google Scholar 

  19. Gonzalez Billandon, J., et al.: Can a robot catch you lying? a machine learning system to detect lies during interactions. Front. Robot. AI 6, 64 (2019)

    CrossRef  Google Scholar 

  20. Gunes, H., Piccardi, M.: Bi-modal emotion recognition from expressive face and body gestures. J. Netw. Comput. Appl. 30(4), 1334–1345 (2007)

    CrossRef  Google Scholar 

  21. Heath, S., et al.: Spatiotemporal aspects of engagement during dialogic storytelling child-robot interaction. Front. Robot. AI 4, 27 (2017)

    CrossRef  Google Scholar 

  22. Kaniusas, E.: Biomedical Signals and Sensors I: Linking Physiological Phenomena and Biosignals. Springer Science & Business Media, Berlin (2012)

    CrossRef  Google Scholar 

  23. Leite, I., Henriques, R., Martinho, C., Paiva, A.: Sensors in the wild: exploring electrodermal activity in child-robot interaction. In: Proceeding of ACM/IEEE HRI 2013, pp. 41–48. IEEE (2013)

    Google Scholar 

  24. Loewe, N., Nadj, M.: Physio-adaptive systems-a state-of-the-art review and future research directions. In: ECIS (2020)

    Google Scholar 

  25. McDuff, D., Blackford, E.: iphys: an open non-contact imaging-based physiological measurement toolbox. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6521–6524. IEEE (2019)

    Google Scholar 

  26. Metta, G., Fitzpatrick, P., Natale, L.: Yarp: yet another robot platform. Int. J. Adv. Robot. Syst. 3(1), 8 (2006)

    CrossRef  Google Scholar 

  27. Motti, V.G., Caine, K.: Users’ privacy concerns about wearables. In: Brenner, M., Christin, N., Johnson, B., Rohloff, K. (eds.) FC 2015. LNCS, vol. 8976, pp. 231–244. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-48051-9_17

    CrossRef  Google Scholar 

  28. Mukherjee, S.S., Robertson, N.M.: Deep head pose: gaze-direction estimation in multimodal video. IEEE Trans. Multimedia 17(11), 2094–2107 (2015)

    CrossRef  Google Scholar 

  29. Muñoz, J., Rubio, E., Cameirao, M., Bermúdez, S.: The biocybernetic loop engine: an integrated tool for creating physiologically adaptive videogames. In: Proceeding of International Conference on Physiological Computing Systems, pp. 45–54 (2017)

    Google Scholar 

  30. Muñoz, J.E., Cameirão, M., Bermúdez i Badia, S., Gouveia, E.R.: Closing the loop in exergaming-health benefits of biocybernetic adaptation in senior adults. In: Proceeding of Annual Symposium on Computer-Human Interaction in Play, pp. 329–339 (2018)

    Google Scholar 

  31. Ng, H.W., Nguyen, V.D., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: Proceeding of ACM ICMI, vol. 2015, pp. 443–449 (2015)

    Google Scholar 

  32. Novak, D.: Engineering issues in physiological computing. In: Fairclough, S.H., Gilleade, K. (eds.) Advances in Physiological Computing. HIS, pp. 17–38. Springer, London (2014). https://doi.org/10.1007/978-1-4471-6392-3_2

    CrossRef  Google Scholar 

  33. Novak, D., Mihelj, M., Munih, M.: A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing. Interact. Comput. 24(3), 154–172 (2012)

    CrossRef  Google Scholar 

  34. Paiva, A., Leite, I., Boukricha, H., Wachsmuth, I.: Empathy in virtual agents and robots: a survey. ACM Trans. Interact. Intell. Syst. (TiiS) 7(3), 1–40 (2017)

    CrossRef  Google Scholar 

  35. Pasquali, D., Aroyo, A.M., Gonzalez-Billandon, J., Rea, F., Sandini, G., Sciutti, A.: Your eyes never lie: a robot magician can tell if you are lying. In: Proceeding of companion of ACM/IEEE HRI, vol. 2020, pp. 392–394 (2020)

    Google Scholar 

  36. Peck, E.M., Easse, E., Marshall, N., Stratton, W., Perrone, L.F.: Flyloop: a micro framework for rapid development of physiological computing systems. In: Proceeding of ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 152–157 (2015)

    Google Scholar 

  37. Pineda, J., Hecht, E.: Mirroring and mu rhythm involvement in social cognition: are there dissociable subcomponents of theory of mind? Biol. Psychol. 80(3), 306–314 (2009)

    CrossRef  Google Scholar 

  38. Plews, D.J., Scott, B., Altini, M., Wood, M., Kilding, A.E., Laursen, P.B.: Comparison of heart-rate-variability recording with smartphone photoplethysmography, polar h7 chest strap, and electrocardiography. Int. J. Sports Physiol. Perform. 12(10), 1324–1328 (2017)

    CrossRef  Google Scholar 

  39. Quigley, M., et al.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3, p. 5. Kobe, Japan (2009)

    Google Scholar 

  40. Renard, Y., et al.: Openvibe: an open-source software platform to design, test, and use brain-computer interfaces in real and virtual environments. Presence: Teleoperators Virtual Environ. 19(1), 35–53 (2010)

    MathSciNet  CrossRef  Google Scholar 

  41. Rich, C., Ponsler, B., Holroyd, A., Sidner, C.L.: Recognizing engagement in human-robot interaction. In: Proceeding of ACM/IEEE HRI 2010), pp. 375–382. IEEE (2010)

    Google Scholar 

  42. Rudovic, O., Lee, J., Mascarell-Maricic, L., Schuller, B.W., Picard, R.W.: Measuring engagement in robot-assisted autism therapy: a cross-cultural study. Front. Robot. AI 4, 36 (2017)

    CrossRef  Google Scholar 

  43. Shan, C., Gong, S., McOwan, P.W.: Beyond facial expressions: learning human emotion from body gestures. In: BMVC, pp. 1–10 (2007)

    Google Scholar 

  44. Sjak-Shie, E.E.: Physiodata toolbox (version 0.5) [computer software] (2019). https://PhysioDataToolbox.leidenuniv.nl

  45. Szafir, D., Mutlu, B.: Pay attention! designing adaptive agents that monitor and improve user engagement. In: Proceeding of IGCHI Conference on Human Factors in Computing Systems, pp. 11–20 (2012)

    Google Scholar 

  46. Tsiakas, K., Abujelala, M., Makedon, F.: Task engagement as personalization feedback for socially-assistive robots and cognitive training. Technologies 6(2), 49 (2018)

    CrossRef  Google Scholar 

  47. Vogel, E.A.: About one-in-five americans use a smart watch or fitness tracker (2020). https://www.pewresearch.org/fact-tank/2020/01/09/about-one-in-five-americans-use-a-smart-watch-or-fitness-tracker/

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Austin Kothig .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Kothig, A., Muñoz, J., Mahdi, H., Aroyo, A.M., Dautenhahn, K. (2020). HRI Physio Lib: A Software Framework to Support the Integration of Physiological Adaptation in HRI. In: , et al. Social Robotics. ICSR 2020. Lecture Notes in Computer Science(), vol 12483. Springer, Cham. https://doi.org/10.1007/978-3-030-62056-1_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62056-1_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62055-4

  • Online ISBN: 978-3-030-62056-1

  • eBook Packages: Computer ScienceComputer Science (R0)