Validating human–robot interaction can be a challenging task, especially in cases in which the robot designer is interested in the assessment of individual robot actions within an ongoing interaction that should not be interrupted by intermittent surveys. In this paper, we propose a neuro-based method for real-time quantitative assessment of robot actions. The method encompasses the decoding of error-related potentials (ErrPs) from the electroencephalogram (EEG) of a human during interaction with a robot, which could be a useful and intuitive complement to existing methods for validating human–robot interaction in the future. To demonstrate usability, we conducted a study in which we examined EEG-based ErrPs in response to a humanoid robot displaying semantically incorrect actions in a simplistic HRI task. Furthermore, we conducted a procedurally identical control experiment with computer screen-based symbolic cursor action. The results of our study confirmed decodeability of ErrPs in response to incorrect robot actions with an average accuracy of \(69.0\pm 7.9\%\) across 11 subjects. Cross-comparisons of ErrPs between experimental tasks revealed high temporal and topographical similarity, but more distinct signals in response to the cursor action and, as a result, better decodeability with a mean accuracy of \(90.6\pm 3.9\%\). This demonstrated that ErrPs can be sensitive to the stimulus eliciting them despite procedurally identical protocols. Re-using ErrP-decoders across experimental tasks without re-calibration is accompanied by significant performance losses and therefore not recommended. Overall, the outcomes of our study confirm feasibility of ErrP-decoding for human–robot validation, but also highlight challenges to overcome in order to enhance usability of the proposed method.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Alexander WH, Brown JW (2011) Medial prefrontal cortex as an action-outcome predictor. Nat Neurosci 14(10):1338
Alves-Pinto A, Ehrlich S, Cheng G, Turova V, Blumenstein T, Lampe R (2017) Effects of short-term piano training on measures of finger tapping, somatosensory perception and motor-related brain activity in patients with cerebral palsy. Neuropsychiatr Dis Treat 13:2705
Bartneck C, Croft E, Kulic D (2008) Measuring the anthropomorphism, animacy, likeability, perceived intelligence and perceived safety of robots. In: Metrics for HRI workshop, technical report, Citeseer, vol 471, pp 37–44
Bartneck C, Kulić D, Croft E, Zoghbi S (2009) Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot 1(1):71–81
Blankertz B, Lemm S, Treder M, Haufe S, Müller KR (2011) Single-trial analysis and classification of ERP components—a tutorial. NeuroImage 56(2):814–825
Botvinick MM, Braver TS, Barch DM, Carter CS, Cohen JD (2001) Conflict monitoring and cognitive control. Psychol Rev 108(3):624
Chavarriaga R, Millán JR (2010) Learning from eeg error-related potentials in noninvasive brain–computer interfaces. IEEE Trans Neural Syst Rehabil Eng 18(4):381–388
Chavarriaga R, Sobolewski A, Millán JR (2014) Errare machinale est: the use of error-related potentials in brain–machine interfaces. Front Neurosci 8:208
Combrisson E, Jerbi K (2015) Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J Neurosci Methods 250:126–136
Dautenhahn K, Woods S, Kaouri C, Walters ML, Koay KL, Werry I (2005) What is a robot companion-friend, assistant or butler? In: 2005 IEEE/RSJ international conference on intelligent robots and systems, 2005 (IROS 2005). IEEE, pp 1192–1197
Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. J Neurosci Methods 134(1):9–21
Ehrlich S, Cheng G (2016) A neuro-based method for detecting context-dependent erroneous robot action. In: 2016 IEEE-RAS 16th international conference on humanoid robots (humanoids). IEEE, pp 477–482
Ehrlich S, Cheng G (2018) Human-agent co-adaptation using error-related potentials. J Neural Eng 15:066014
Ehrlich S, Alves-Pinto A, Lampe R, Cheng G (2017) A simple and practical sensorimotor EEG device for recording in patients with special needs. In: Neurotechnix2017, CogNeuroEng 2017
Falkenstein M, Hohnsbein J, Hoormann J, Blanke L (1991) Effects of crossmodal divided attention on late erp components. II. Error processing in choice reaction tasks. Electroencephalogr Clin Neurophysiol 78(6):447–455
Falkenstein M, Hoormann J, Christ S, Hohnsbein J (2000) ERP components on reaction errors and their functional significance: a tutorial. Biol Psychol 51(2–3):87–107
Ferrez PW, Millán JdR (2005) You are wrong!—automatic detection of interaction errors from brain waves. In: Proceedings of the 19th international joint conference on artificial intelligence, EPFL-CONF-83269
Ferrez PW, Millán JR (2008) Error-related eeg potentials generated during simulated brain–computer interaction. IEEE Trans Biomed Eng 55(3):923–929
Friedman JH (1989) Regularized discriminant analysis. J Am Stat Assoc 84(405):165–175
Frischen A, Bayliss AP, Tipper SP (2007) Gaze cueing of attention: visual attention, social cognition, and individual differences. Psychol Bull 133(4):694
Gombolay MC, Gutierrez RA, Clarke SG, Sturla GF, Shah JA (2015) Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams. Auton Robots 39(3):293–312
Goodrich MA, Schultz AC (2007) Human–robot interaction: a survey. Found Trends Hum Comput Interact 1(3):203–275
Gouaillier D, Hugel V, Blazevic P, Kilner C, Monceaux J, Lafourcade P, Marnier B, Serre J, Maisonnier B (2008) The nao humanoid: a combination of performance and affordability. CoRR arXiv:abs/08073223
Hairston WD, Whitaker KW, Ries AJ, Vettel JM, Bradford JC, Kerick SE, McDowell K (2014) Usability of four commercially-oriented EEG systems. J Neural Eng 11(4):046,018
Holroyd CB, Coles MG (2002) The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol Rev 109(4):679
Holroyd CB, Dien J, Coles MG (1998) Error-related scalp potentials elicited by hand and foot movements: evidence for an output-independent error-processing system in humans. Neurosci Lett 242(2):65–68
Homan RW, Herman J, Purdy P (1987) Cerebral location of international 10–20 system electrode placement. Electroencephalogr Clin Neurophysiol 66(4):376–382
Huang CM, Mutlu B (2012) Robot behavior toolkit: generating effective social behaviors for robots. In: Proceedings of the seventh annual ACM/IEEE international conference on human–robot interaction. ACM, pp 25–32
Huang CM, Cakmak M, Mutlu B (2015) Adaptive coordination strategies for human–robot handovers. In: Robotics: science and systems
Iturrate I, Montesano L, Minguez J (2010) Single trial recognition of error-related potentials during observation of robot operation. In: 2010 annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 4181–4184
Iturrate I, Montesano L, Minguez J (2013) Task-dependent signal variations in eeg error-related potentials for brain–computer interfaces. J Neural Eng 10(2):026,024
Iturrate I, Chavarriaga R, Montesano L, Minguez J, Millán JR (2015a) Teaching brain–machine interfaces as an alternative paradigm to neuroprosthetics control. Sci Rep 5(13):893
Iturrate I, Grizou J, Omedes J, Oudeyer PY, Lopes M, Montesano L (2015b) Exploiting task constraints for self-calibrated brain–machine interface control using error-related potentials. PLoS ONE 10(7):e0131,491
Joosse M, Sardar A, Lohse M, Evers V (2013) Behave-II: the revised set of measures to assess users? attitudinal and behavioral responses to a social robot. Int J Soc Robot 5(3):379–388
Kaan E, Harris A, Gibson E, Holcomb P (2000) The P600 as an index of syntactic integration difficulty. Lang Cogn Process 15(2):159–201
Kim SK, Kirchner EA, Stefes A, Kirchner F (2017) Intrinsic interactive reinforcement learning-using error-related potentials for real world human–robot interaction. Sci Rep 7(1):17,562
Kreilinger A, Neuper C, Müller-Putz GR (2012) Error potential detection during continuous movement of an artificial arm controlled by brain–computer interface. Med Biol Eng Comput 50(3):223–230
Llera A, van Gerven MA, Gómez V, Jensen O, Kappen HJ (2011) On the use of interaction error potentials for adaptive brain computer interfaces. Neural Netw 24(10):1120–1127
Lotte F, Guan C (2010) Learning from other subjects helps reducing brain–computer interface calibration time. In: 2010 IEEE international conference on acoustics speech and signal processing (ICASSP). IEEE, pp 614–617
Miltner WH, Braun CH, Coles MG (1997) Event-related brain potentials following incorrect feedback in a time-estimation task: evidence for a generic neural system for error detection. J Cogn Neurosci 9(6):788–798
Mutlu B, Forlizzi J (2008) Robots in organizations: the role of workflow, social, and environmental factors in human–robot interaction. In: Proceedings of the 3rd ACM/IEEE international conference on human robot interaction. ACM, pp 287–294
Mutlu B, Shiwa T, Kanda T, Ishiguro H, Hagita N (2009) Footing in human–robot conversations: how robots might shape participant roles using gaze cues. In: Proceedings of the 4th ACM/IEEE international conference on human robot interaction. ACM, pp 61–68
Parra LC, Spence CD, Gerson AD, Sajda P (2003) Response error correction—a demonstration of improved human–machine performance using real-time EEG monitoring. IEEE Trans Neural Syst Rehabil Eng 11(2):173–177
Peirce JW (2007) Psychopy—psychophysics software in python. J Neurosci Methods 162(1–2):8–13
Ridderinkhof KR, Ullsperger M, Crone EA, Nieuwenhuis S (2004) The role of the medial frontal cortex in cognitive control. Science 306(5695):443–447
Salazar-Gomez AF, DelPreto J, Gil S, Guenther FH, Rus D (2017) Correcting robot mistakes in real time using EEG signals. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp 6570–6577
Schäfer J, Strimmer K (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Mol Biol 4(1):1175–1189
van Schie HT, Mars RB, Coles MG, Bekkering H (2004) Modulation of activity in medial frontal and motor cortices during error observation. Nat Neurosci 7(5):549
Schlögl A, Keinrath C, Zimmermann D, Scherer R, Leeb R, Pfurtscheller G (2007) A fully automated correction method of EOG artifacts in EEG recordings. Clin Neurophysiol 118(1):98–104
Schmidt NM, Blankertz B, Treder MS (2012) Online detection of error-related potentials boosts the performance of mental typewriters. BMC Neurosci 13(1):19
Spüler M, Niethammer C (2015) Error-related potentials during continuous feedback: using EEG to detect errors of different type and severity. Front Hum Neurosci 9:155
Spüler M, Rosenstiel W, Bogdan M (2012) Online adaptation of a c-VEP brain–computer interface (BCI) based on error-related potentials and unsupervised learning. PLoS ONE 7(12):e51,077
Szafir D, Mutlu B (2012) Pay attention!: designing adaptive agents that monitor and improve user engagement. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 11–20
Van Herten M, Kolk HH, Chwilla DJ (2005) An ERP study of P600 effects elicited by semantic anomalies. Cogn Brain Res 22(2):241–255
Welke D, Behncke J, Hader M, Schirrmeister RT, Schönau A, Eßmann B, Müller O, Burgard W, Ball T (2017) Brain responses during robot-error observation. ArXiv preprint arXiv:170801465
Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain–computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791
We thank Ana Alves-Pinto and Sae Franklin for helpful comments on revision and editing of the manuscript. We thank the anonymous reviewers for their detailed comments and references, which have led to significant clarification of the work in this paper. This research was partially supported by Deutsche Forschungsgemeinschaft (DFG) through the International Graduate School of Science and Engineering (IGSSE) at the Technical University of Munich (TUM).
Conflict of interest
The authors declare no conflict of interest nor competing financial interests.
This work was approved by the ethics commission of the Faculty of Medicine, Technische Universität München (TUM) under the Reference Number 236/15s.
Consent to participate and publish was obtained from the participants in verbal and written form.
Availability of Data and Material
Data and material was not made publicly available but can be obtained from the corresponding author.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Ehrlich, S.K., Cheng, G. A Feasibility Study for Validating Robot Actions Using EEG-Based Error-Related Potentials. Int J of Soc Robotics 11, 271–283 (2019). https://doi.org/10.1007/s12369-018-0501-8
- Electroencephalography (EEG)
- Passive brain–computer interface (BCI)
- Error-related potentials (ErrP)
- Event-related potentials (ERP)
- Error monitoring
- Human–robot interaction (HRI)