Skip to main content
SpringerLink
Log in

A Feasibility Study for Validating Robot Actions Using EEG-Based Error-Related Potentials

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Validating human–robot interaction can be a challenging task, especially in cases in which the robot designer is interested in the assessment of individual robot actions within an ongoing interaction that should not be interrupted by intermittent surveys. In this paper, we propose a neuro-based method for real-time quantitative assessment of robot actions. The method encompasses the decoding of error-related potentials (ErrPs) from the electroencephalogram (EEG) of a human during interaction with a robot, which could be a useful and intuitive complement to existing methods for validating human–robot interaction in the future. To demonstrate usability, we conducted a study in which we examined EEG-based ErrPs in response to a humanoid robot displaying semantically incorrect actions in a simplistic HRI task. Furthermore, we conducted a procedurally identical control experiment with computer screen-based symbolic cursor action. The results of our study confirmed decodeability of ErrPs in response to incorrect robot actions with an average accuracy of \(69.0\pm 7.9\%\) across 11 subjects. Cross-comparisons of ErrPs between experimental tasks revealed high temporal and topographical similarity, but more distinct signals in response to the cursor action and, as a result, better decodeability with a mean accuracy of \(90.6\pm 3.9\%\). This demonstrated that ErrPs can be sensitive to the stimulus eliciting them despite procedurally identical protocols. Re-using ErrP-decoders across experimental tasks without re-calibration is accompanied by significant performance losses and therefore not recommended. Overall, the outcomes of our study confirm feasibility of ErrP-decoding for human–robot validation, but also highlight challenges to overcome in order to enhance usability of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Alexander WH, Brown JW (2011) Medial prefrontal cortex as an action-outcome predictor. Nat Neurosci 14(10):1338

    Article  Google Scholar 

  2. Alves-Pinto A, Ehrlich S, Cheng G, Turova V, Blumenstein T, Lampe R (2017) Effects of short-term piano training on measures of finger tapping, somatosensory perception and motor-related brain activity in patients with cerebral palsy. Neuropsychiatr Dis Treat 13:2705

    Article  Google Scholar 

  3. Bartneck C, Croft E, Kulic D (2008) Measuring the anthropomorphism, animacy, likeability, perceived intelligence and perceived safety of robots. In: Metrics for HRI workshop, technical report, Citeseer, vol 471, pp 37–44

  4. Bartneck C, Kulić D, Croft E, Zoghbi S (2009) Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot 1(1):71–81

    Article  Google Scholar 

  5. Blankertz B, Lemm S, Treder M, Haufe S, Müller KR (2011) Single-trial analysis and classification of ERP components—a tutorial. NeuroImage 56(2):814–825

    Article  Google Scholar 

  6. Botvinick MM, Braver TS, Barch DM, Carter CS, Cohen JD (2001) Conflict monitoring and cognitive control. Psychol Rev 108(3):624

    Article  Google Scholar 

  7. Chavarriaga R, Millán JR (2010) Learning from eeg error-related potentials in noninvasive brain–computer interfaces. IEEE Trans Neural Syst Rehabil Eng 18(4):381–388

    Article  Google Scholar 

  8. Chavarriaga R, Sobolewski A, Millán JR (2014) Errare machinale est: the use of error-related potentials in brain–machine interfaces. Front Neurosci 8:208

    Article  Google Scholar 

  9. Combrisson E, Jerbi K (2015) Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J Neurosci Methods 250:126–136

    Article  Google Scholar 

  10. Dautenhahn K, Woods S, Kaouri C, Walters ML, Koay KL, Werry I (2005) What is a robot companion-friend, assistant or butler? In: 2005 IEEE/RSJ international conference on intelligent robots and systems, 2005 (IROS 2005). IEEE, pp 1192–1197

  11. Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. J Neurosci Methods 134(1):9–21

    Article  Google Scholar 

  12. Ehrlich S, Cheng G (2016) A neuro-based method for detecting context-dependent erroneous robot action. In: 2016 IEEE-RAS 16th international conference on humanoid robots (humanoids). IEEE, pp 477–482

  13. Ehrlich S, Cheng G (2018) Human-agent co-adaptation using error-related potentials. J Neural Eng 15:066014

    Article  Google Scholar 

  14. Ehrlich S, Alves-Pinto A, Lampe R, Cheng G (2017) A simple and practical sensorimotor EEG device for recording in patients with special needs. In: Neurotechnix2017, CogNeuroEng 2017

  15. Falkenstein M, Hohnsbein J, Hoormann J, Blanke L (1991) Effects of crossmodal divided attention on late erp components. II. Error processing in choice reaction tasks. Electroencephalogr Clin Neurophysiol 78(6):447–455

    Article  Google Scholar 

  16. Falkenstein M, Hoormann J, Christ S, Hohnsbein J (2000) ERP components on reaction errors and their functional significance: a tutorial. Biol Psychol 51(2–3):87–107

    Article  Google Scholar 

  17. Ferrez PW, Millán JdR (2005) You are wrong!—automatic detection of interaction errors from brain waves. In: Proceedings of the 19th international joint conference on artificial intelligence, EPFL-CONF-83269

  18. Ferrez PW, Millán JR (2008) Error-related eeg potentials generated during simulated brain–computer interaction. IEEE Trans Biomed Eng 55(3):923–929

    Article  Google Scholar 

  19. Friedman JH (1989) Regularized discriminant analysis. J Am Stat Assoc 84(405):165–175

    Article  MathSciNet  Google Scholar 

  20. Frischen A, Bayliss AP, Tipper SP (2007) Gaze cueing of attention: visual attention, social cognition, and individual differences. Psychol Bull 133(4):694

    Article  Google Scholar 

  21. Gombolay MC, Gutierrez RA, Clarke SG, Sturla GF, Shah JA (2015) Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams. Auton Robots 39(3):293–312

    Article  Google Scholar 

  22. Goodrich MA, Schultz AC (2007) Human–robot interaction: a survey. Found Trends Hum Comput Interact 1(3):203–275

    Article  MATH  Google Scholar 

  23. Gouaillier D, Hugel V, Blazevic P, Kilner C, Monceaux J, Lafourcade P, Marnier B, Serre J, Maisonnier B (2008) The nao humanoid: a combination of performance and affordability. CoRR arXiv:abs/08073223

  24. Hairston WD, Whitaker KW, Ries AJ, Vettel JM, Bradford JC, Kerick SE, McDowell K (2014) Usability of four commercially-oriented EEG systems. J Neural Eng 11(4):046,018

    Article  Google Scholar 

  25. Holroyd CB, Coles MG (2002) The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol Rev 109(4):679

    Article  Google Scholar 

  26. Holroyd CB, Dien J, Coles MG (1998) Error-related scalp potentials elicited by hand and foot movements: evidence for an output-independent error-processing system in humans. Neurosci Lett 242(2):65–68

    Article  Google Scholar 

  27. Homan RW, Herman J, Purdy P (1987) Cerebral location of international 10–20 system electrode placement. Electroencephalogr Clin Neurophysiol 66(4):376–382

    Article  Google Scholar 

  28. Huang CM, Mutlu B (2012) Robot behavior toolkit: generating effective social behaviors for robots. In: Proceedings of the seventh annual ACM/IEEE international conference on human–robot interaction. ACM, pp 25–32

  29. Huang CM, Cakmak M, Mutlu B (2015) Adaptive coordination strategies for human–robot handovers. In: Robotics: science and systems

  30. Iturrate I, Montesano L, Minguez J (2010) Single trial recognition of error-related potentials during observation of robot operation. In: 2010 annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 4181–4184

  31. Iturrate I, Montesano L, Minguez J (2013) Task-dependent signal variations in eeg error-related potentials for brain–computer interfaces. J Neural Eng 10(2):026,024

    Article  Google Scholar 

  32. Iturrate I, Chavarriaga R, Montesano L, Minguez J, Millán JR (2015a) Teaching brain–machine interfaces as an alternative paradigm to neuroprosthetics control. Sci Rep 5(13):893

    Google Scholar 

  33. Iturrate I, Grizou J, Omedes J, Oudeyer PY, Lopes M, Montesano L (2015b) Exploiting task constraints for self-calibrated brain–machine interface control using error-related potentials. PLoS ONE 10(7):e0131,491

    Article  Google Scholar 

  34. Joosse M, Sardar A, Lohse M, Evers V (2013) Behave-II: the revised set of measures to assess users? attitudinal and behavioral responses to a social robot. Int J Soc Robot 5(3):379–388

    Article  Google Scholar 

  35. Kaan E, Harris A, Gibson E, Holcomb P (2000) The P600 as an index of syntactic integration difficulty. Lang Cogn Process 15(2):159–201

    Article  Google Scholar 

  36. Kim SK, Kirchner EA, Stefes A, Kirchner F (2017) Intrinsic interactive reinforcement learning-using error-related potentials for real world human–robot interaction. Sci Rep 7(1):17,562

    Article  Google Scholar 

  37. Kreilinger A, Neuper C, Müller-Putz GR (2012) Error potential detection during continuous movement of an artificial arm controlled by brain–computer interface. Med Biol Eng Comput 50(3):223–230

    Article  Google Scholar 

  38. Llera A, van Gerven MA, Gómez V, Jensen O, Kappen HJ (2011) On the use of interaction error potentials for adaptive brain computer interfaces. Neural Netw 24(10):1120–1127

    Article  Google Scholar 

  39. Lotte F, Guan C (2010) Learning from other subjects helps reducing brain–computer interface calibration time. In: 2010 IEEE international conference on acoustics speech and signal processing (ICASSP). IEEE, pp 614–617

  40. Miltner WH, Braun CH, Coles MG (1997) Event-related brain potentials following incorrect feedback in a time-estimation task: evidence for a generic neural system for error detection. J Cogn Neurosci 9(6):788–798

    Article  Google Scholar 

  41. Mutlu B, Forlizzi J (2008) Robots in organizations: the role of workflow, social, and environmental factors in human–robot interaction. In: Proceedings of the 3rd ACM/IEEE international conference on human robot interaction. ACM, pp 287–294

  42. Mutlu B, Shiwa T, Kanda T, Ishiguro H, Hagita N (2009) Footing in human–robot conversations: how robots might shape participant roles using gaze cues. In: Proceedings of the 4th ACM/IEEE international conference on human robot interaction. ACM, pp 61–68

  43. Parra LC, Spence CD, Gerson AD, Sajda P (2003) Response error correction—a demonstration of improved human–machine performance using real-time EEG monitoring. IEEE Trans Neural Syst Rehabil Eng 11(2):173–177

    Article  Google Scholar 

  44. Peirce JW (2007) Psychopy—psychophysics software in python. J Neurosci Methods 162(1–2):8–13

    Article  Google Scholar 

  45. Ridderinkhof KR, Ullsperger M, Crone EA, Nieuwenhuis S (2004) The role of the medial frontal cortex in cognitive control. Science 306(5695):443–447

    Article  Google Scholar 

  46. Salazar-Gomez AF, DelPreto J, Gil S, Guenther FH, Rus D (2017) Correcting robot mistakes in real time using EEG signals. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp 6570–6577

  47. Schäfer J, Strimmer K (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Mol Biol 4(1):1175–1189

    Article  MathSciNet  Google Scholar 

  48. van Schie HT, Mars RB, Coles MG, Bekkering H (2004) Modulation of activity in medial frontal and motor cortices during error observation. Nat Neurosci 7(5):549

    Article  Google Scholar 

  49. Schlögl A, Keinrath C, Zimmermann D, Scherer R, Leeb R, Pfurtscheller G (2007) A fully automated correction method of EOG artifacts in EEG recordings. Clin Neurophysiol 118(1):98–104

    Article  Google Scholar 

  50. Schmidt NM, Blankertz B, Treder MS (2012) Online detection of error-related potentials boosts the performance of mental typewriters. BMC Neurosci 13(1):19

    Article  Google Scholar 

  51. Spüler M, Niethammer C (2015) Error-related potentials during continuous feedback: using EEG to detect errors of different type and severity. Front Hum Neurosci 9:155

    Google Scholar 

  52. Spüler M, Rosenstiel W, Bogdan M (2012) Online adaptation of a c-VEP brain–computer interface (BCI) based on error-related potentials and unsupervised learning. PLoS ONE 7(12):e51,077

    Article  Google Scholar 

  53. Szafir D, Mutlu B (2012) Pay attention!: designing adaptive agents that monitor and improve user engagement. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 11–20

  54. Van Herten M, Kolk HH, Chwilla DJ (2005) An ERP study of P600 effects elicited by semantic anomalies. Cogn Brain Res 22(2):241–255

    Article  Google Scholar 

  55. Welke D, Behncke J, Hader M, Schirrmeister RT, Schönau A, Eßmann B, Müller O, Burgard W, Ball T (2017) Brain responses during robot-error observation. ArXiv preprint arXiv:170801465

  56. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain–computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791

    Article  Google Scholar 

Download references

Acknowledgements

We thank Ana Alves-Pinto and Sae Franklin for helpful comments on revision and editing of the manuscript. We thank the anonymous reviewers for their detailed comments and references, which have led to significant clarification of the work in this paper. This research was partially supported by Deutsche Forschungsgemeinschaft (DFG) through the International Graduate School of Science and Engineering (IGSSE) at the Technical University of Munich (TUM).

Author information

Authors and Affiliations

  1. Chair for Cognitive Systems, Department of Electrical and Computer Engineering, Technical University of Munich (TUM), Arcisstrasse 21, 80333, Munich, Germany

    Stefan K. Ehrlich & Gordon Cheng

Authors
  1. Stefan K. Ehrlich
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gordon Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan K. Ehrlich.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest nor competing financial interests.

Ethics Approval

This work was approved by the ethics commission of the Faculty of Medicine, Technische Universität München (TUM) under the Reference Number 236/15s.

Informed Consent

Consent to participate and publish was obtained from the participants in verbal and written form.

Availability of Data and Material

Data and material was not made publicly available but can be obtained from the corresponding author.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 66 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ehrlich, S.K., Cheng, G. A Feasibility Study for Validating Robot Actions Using EEG-Based Error-Related Potentials. Int J of Soc Robotics 11, 271–283 (2019). https://doi.org/10.1007/s12369-018-0501-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-018-0501-8

Keywords

Search

Navigation