Advertisement

A Feasibility Study for Validating Robot Actions Using EEG-Based Error-Related Potentials

  • Stefan K. Ehrlich
  • Gordon Cheng
Article

Abstract

Validating human–robot interaction can be a challenging task, especially in cases in which the robot designer is interested in the assessment of individual robot actions within an ongoing interaction that should not be interrupted by intermittent surveys. In this paper, we propose a neuro-based method for real-time quantitative assessment of robot actions. The method encompasses the decoding of error-related potentials (ErrPs) from the electroencephalogram (EEG) of a human during interaction with a robot, which could be a useful and intuitive complement to existing methods for validating human–robot interaction in the future. To demonstrate usability, we conducted a study in which we examined EEG-based ErrPs in response to a humanoid robot displaying semantically incorrect actions in a simplistic HRI task. Furthermore, we conducted a procedurally identical control experiment with computer screen-based symbolic cursor action. The results of our study confirmed decodeability of ErrPs in response to incorrect robot actions with an average accuracy of \(69.0\pm 7.9\%\) across 11 subjects. Cross-comparisons of ErrPs between experimental tasks revealed high temporal and topographical similarity, but more distinct signals in response to the cursor action and, as a result, better decodeability with a mean accuracy of \(90.6\pm 3.9\%\). This demonstrated that ErrPs can be sensitive to the stimulus eliciting them despite procedurally identical protocols. Re-using ErrP-decoders across experimental tasks without re-calibration is accompanied by significant performance losses and therefore not recommended. Overall, the outcomes of our study confirm feasibility of ErrP-decoding for human–robot validation, but also highlight challenges to overcome in order to enhance usability of the proposed method.

Keywords

Electroencephalography (EEG) Passive brain–computer interface (BCI) Error-related potentials (ErrP) Event-related potentials (ERP) Error monitoring Human–robot interaction (HRI) 

Notes

Acknowledgements

We thank Ana Alves-Pinto and Sae Franklin for helpful comments on revision and editing of the manuscript. We thank the anonymous reviewers for their detailed comments and references, which have led to significant clarification of the work in this paper. This research was partially supported by Deutsche Forschungsgemeinschaft (DFG) through the International Graduate School of Science and Engineering (IGSSE) at the Technical University of Munich (TUM).

Compliance with Ethical Standards

Conflict of interest

The authors declare no conflict of interest nor competing financial interests.

Ethics Approval

This work was approved by the ethics commission of the Faculty of Medicine, Technische Universität München (TUM) under the Reference Number 236/15s.

Informed Consent

Consent to participate and publish was obtained from the participants in verbal and written form.

Availability of Data and Material

Data and material was not made publicly available but can be obtained from the corresponding author.

Supplementary material

12369_2018_501_MOESM1_ESM.pdf (66 kb)
Supplementary material 1 (pdf 66 KB)

References

  1. 1.
    Alexander WH, Brown JW (2011) Medial prefrontal cortex as an action-outcome predictor. Nat Neurosci 14(10):1338CrossRefGoogle Scholar
  2. 2.
    Alves-Pinto A, Ehrlich S, Cheng G, Turova V, Blumenstein T, Lampe R (2017) Effects of short-term piano training on measures of finger tapping, somatosensory perception and motor-related brain activity in patients with cerebral palsy. Neuropsychiatr Dis Treat 13:2705CrossRefGoogle Scholar
  3. 3.
    Bartneck C, Croft E, Kulic D (2008) Measuring the anthropomorphism, animacy, likeability, perceived intelligence and perceived safety of robots. In: Metrics for HRI workshop, technical report, Citeseer, vol 471, pp 37–44Google Scholar
  4. 4.
    Bartneck C, Kulić D, Croft E, Zoghbi S (2009) Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot 1(1):71–81CrossRefGoogle Scholar
  5. 5.
    Blankertz B, Lemm S, Treder M, Haufe S, Müller KR (2011) Single-trial analysis and classification of ERP components—a tutorial. NeuroImage 56(2):814–825CrossRefGoogle Scholar
  6. 6.
    Botvinick MM, Braver TS, Barch DM, Carter CS, Cohen JD (2001) Conflict monitoring and cognitive control. Psychol Rev 108(3):624CrossRefGoogle Scholar
  7. 7.
    Chavarriaga R, Millán JR (2010) Learning from eeg error-related potentials in noninvasive brain–computer interfaces. IEEE Trans Neural Syst Rehabil Eng 18(4):381–388CrossRefGoogle Scholar
  8. 8.
    Chavarriaga R, Sobolewski A, Millán JR (2014) Errare machinale est: the use of error-related potentials in brain–machine interfaces. Front Neurosci 8:208CrossRefGoogle Scholar
  9. 9.
    Combrisson E, Jerbi K (2015) Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J Neurosci Methods 250:126–136CrossRefGoogle Scholar
  10. 10.
    Dautenhahn K, Woods S, Kaouri C, Walters ML, Koay KL, Werry I (2005) What is a robot companion-friend, assistant or butler? In: 2005 IEEE/RSJ international conference on intelligent robots and systems, 2005 (IROS 2005). IEEE, pp 1192–1197Google Scholar
  11. 11.
    Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. J Neurosci Methods 134(1):9–21CrossRefGoogle Scholar
  12. 12.
    Ehrlich S, Cheng G (2016) A neuro-based method for detecting context-dependent erroneous robot action. In: 2016 IEEE-RAS 16th international conference on humanoid robots (humanoids). IEEE, pp 477–482Google Scholar
  13. 13.
    Ehrlich S, Cheng G (2018) Human-agent co-adaptation using error-related potentials. J Neural Eng 15:066014CrossRefGoogle Scholar
  14. 14.
    Ehrlich S, Alves-Pinto A, Lampe R, Cheng G (2017) A simple and practical sensorimotor EEG device for recording in patients with special needs. In: Neurotechnix2017, CogNeuroEng 2017Google Scholar
  15. 15.
    Falkenstein M, Hohnsbein J, Hoormann J, Blanke L (1991) Effects of crossmodal divided attention on late erp components. II. Error processing in choice reaction tasks. Electroencephalogr Clin Neurophysiol 78(6):447–455CrossRefGoogle Scholar
  16. 16.
    Falkenstein M, Hoormann J, Christ S, Hohnsbein J (2000) ERP components on reaction errors and their functional significance: a tutorial. Biol Psychol 51(2–3):87–107CrossRefGoogle Scholar
  17. 17.
    Ferrez PW, Millán JdR (2005) You are wrong!—automatic detection of interaction errors from brain waves. In: Proceedings of the 19th international joint conference on artificial intelligence, EPFL-CONF-83269Google Scholar
  18. 18.
    Ferrez PW, Millán JR (2008) Error-related eeg potentials generated during simulated brain–computer interaction. IEEE Trans Biomed Eng 55(3):923–929CrossRefGoogle Scholar
  19. 19.
    Friedman JH (1989) Regularized discriminant analysis. J Am Stat Assoc 84(405):165–175MathSciNetCrossRefGoogle Scholar
  20. 20.
    Frischen A, Bayliss AP, Tipper SP (2007) Gaze cueing of attention: visual attention, social cognition, and individual differences. Psychol Bull 133(4):694CrossRefGoogle Scholar
  21. 21.
    Gombolay MC, Gutierrez RA, Clarke SG, Sturla GF, Shah JA (2015) Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams. Auton Robots 39(3):293–312CrossRefGoogle Scholar
  22. 22.
    Goodrich MA, Schultz AC (2007) Human–robot interaction: a survey. Found Trends Hum Comput Interact 1(3):203–275CrossRefGoogle Scholar
  23. 23.
    Gouaillier D, Hugel V, Blazevic P, Kilner C, Monceaux J, Lafourcade P, Marnier B, Serre J, Maisonnier B (2008) The nao humanoid: a combination of performance and affordability. CoRR arXiv:abs/08073223
  24. 24.
    Hairston WD, Whitaker KW, Ries AJ, Vettel JM, Bradford JC, Kerick SE, McDowell K (2014) Usability of four commercially-oriented EEG systems. J Neural Eng 11(4):046,018CrossRefGoogle Scholar
  25. 25.
    Holroyd CB, Coles MG (2002) The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol Rev 109(4):679CrossRefGoogle Scholar
  26. 26.
    Holroyd CB, Dien J, Coles MG (1998) Error-related scalp potentials elicited by hand and foot movements: evidence for an output-independent error-processing system in humans. Neurosci Lett 242(2):65–68CrossRefGoogle Scholar
  27. 27.
    Homan RW, Herman J, Purdy P (1987) Cerebral location of international 10–20 system electrode placement. Electroencephalogr Clin Neurophysiol 66(4):376–382CrossRefGoogle Scholar
  28. 28.
    Huang CM, Mutlu B (2012) Robot behavior toolkit: generating effective social behaviors for robots. In: Proceedings of the seventh annual ACM/IEEE international conference on human–robot interaction. ACM, pp 25–32Google Scholar
  29. 29.
    Huang CM, Cakmak M, Mutlu B (2015) Adaptive coordination strategies for human–robot handovers. In: Robotics: science and systemsGoogle Scholar
  30. 30.
    Iturrate I, Montesano L, Minguez J (2010) Single trial recognition of error-related potentials during observation of robot operation. In: 2010 annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 4181–4184Google Scholar
  31. 31.
    Iturrate I, Montesano L, Minguez J (2013) Task-dependent signal variations in eeg error-related potentials for brain–computer interfaces. J Neural Eng 10(2):026,024CrossRefGoogle Scholar
  32. 32.
    Iturrate I, Chavarriaga R, Montesano L, Minguez J, Millán JR (2015a) Teaching brain–machine interfaces as an alternative paradigm to neuroprosthetics control. Sci Rep 5(13):893Google Scholar
  33. 33.
    Iturrate I, Grizou J, Omedes J, Oudeyer PY, Lopes M, Montesano L (2015b) Exploiting task constraints for self-calibrated brain–machine interface control using error-related potentials. PLoS ONE 10(7):e0131,491CrossRefGoogle Scholar
  34. 34.
    Joosse M, Sardar A, Lohse M, Evers V (2013) Behave-II: the revised set of measures to assess users? attitudinal and behavioral responses to a social robot. Int J Soc Robot 5(3):379–388CrossRefGoogle Scholar
  35. 35.
    Kaan E, Harris A, Gibson E, Holcomb P (2000) The P600 as an index of syntactic integration difficulty. Lang Cogn Process 15(2):159–201CrossRefGoogle Scholar
  36. 36.
    Kim SK, Kirchner EA, Stefes A, Kirchner F (2017) Intrinsic interactive reinforcement learning-using error-related potentials for real world human–robot interaction. Sci Rep 7(1):17,562CrossRefGoogle Scholar
  37. 37.
    Kreilinger A, Neuper C, Müller-Putz GR (2012) Error potential detection during continuous movement of an artificial arm controlled by brain–computer interface. Med Biol Eng Comput 50(3):223–230CrossRefGoogle Scholar
  38. 38.
    Llera A, van Gerven MA, Gómez V, Jensen O, Kappen HJ (2011) On the use of interaction error potentials for adaptive brain computer interfaces. Neural Netw 24(10):1120–1127CrossRefGoogle Scholar
  39. 39.
    Lotte F, Guan C (2010) Learning from other subjects helps reducing brain–computer interface calibration time. In: 2010 IEEE international conference on acoustics speech and signal processing (ICASSP). IEEE, pp 614–617Google Scholar
  40. 40.
    Miltner WH, Braun CH, Coles MG (1997) Event-related brain potentials following incorrect feedback in a time-estimation task: evidence for a generic neural system for error detection. J Cogn Neurosci 9(6):788–798CrossRefGoogle Scholar
  41. 41.
    Mutlu B, Forlizzi J (2008) Robots in organizations: the role of workflow, social, and environmental factors in human–robot interaction. In: Proceedings of the 3rd ACM/IEEE international conference on human robot interaction. ACM, pp 287–294Google Scholar
  42. 42.
    Mutlu B, Shiwa T, Kanda T, Ishiguro H, Hagita N (2009) Footing in human–robot conversations: how robots might shape participant roles using gaze cues. In: Proceedings of the 4th ACM/IEEE international conference on human robot interaction. ACM, pp 61–68Google Scholar
  43. 43.
    Parra LC, Spence CD, Gerson AD, Sajda P (2003) Response error correction—a demonstration of improved human–machine performance using real-time EEG monitoring. IEEE Trans Neural Syst Rehabil Eng 11(2):173–177CrossRefGoogle Scholar
  44. 44.
    Peirce JW (2007) Psychopy—psychophysics software in python. J Neurosci Methods 162(1–2):8–13CrossRefGoogle Scholar
  45. 45.
    Ridderinkhof KR, Ullsperger M, Crone EA, Nieuwenhuis S (2004) The role of the medial frontal cortex in cognitive control. Science 306(5695):443–447CrossRefGoogle Scholar
  46. 46.
    Salazar-Gomez AF, DelPreto J, Gil S, Guenther FH, Rus D (2017) Correcting robot mistakes in real time using EEG signals. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp 6570–6577Google Scholar
  47. 47.
    Schäfer J, Strimmer K (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Mol Biol 4(1):1175–1189MathSciNetCrossRefGoogle Scholar
  48. 48.
    van Schie HT, Mars RB, Coles MG, Bekkering H (2004) Modulation of activity in medial frontal and motor cortices during error observation. Nat Neurosci 7(5):549CrossRefGoogle Scholar
  49. 49.
    Schlögl A, Keinrath C, Zimmermann D, Scherer R, Leeb R, Pfurtscheller G (2007) A fully automated correction method of EOG artifacts in EEG recordings. Clin Neurophysiol 118(1):98–104CrossRefGoogle Scholar
  50. 50.
    Schmidt NM, Blankertz B, Treder MS (2012) Online detection of error-related potentials boosts the performance of mental typewriters. BMC Neurosci 13(1):19CrossRefGoogle Scholar
  51. 51.
    Spüler M, Niethammer C (2015) Error-related potentials during continuous feedback: using EEG to detect errors of different type and severity. Front Hum Neurosci 9:155Google Scholar
  52. 52.
    Spüler M, Rosenstiel W, Bogdan M (2012) Online adaptation of a c-VEP brain–computer interface (BCI) based on error-related potentials and unsupervised learning. PLoS ONE 7(12):e51,077CrossRefGoogle Scholar
  53. 53.
    Szafir D, Mutlu B (2012) Pay attention!: designing adaptive agents that monitor and improve user engagement. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 11–20Google Scholar
  54. 54.
    Van Herten M, Kolk HH, Chwilla DJ (2005) An ERP study of P600 effects elicited by semantic anomalies. Cogn Brain Res 22(2):241–255CrossRefGoogle Scholar
  55. 55.
    Welke D, Behncke J, Hader M, Schirrmeister RT, Schönau A, Eßmann B, Müller O, Burgard W, Ball T (2017) Brain responses during robot-error observation. ArXiv preprint arXiv:170801465
  56. 56.
    Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain–computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.Chair for Cognitive Systems, Department of Electrical and Computer EngineeringTechnical University of Munich (TUM)MunichGermany

Personalised recommendations