Soft Computing

, Volume 17, Issue 2, pp 317–331 | Cite as

Brain–computer interfacing: more than the sum of its parts

  • Reinhold Scherer
  • Josef Faller
  • David Balderas
  • Elisabeth V. C. Friedrich
  • Markus Pröll
  • Brendan Allison
  • Gernot Müller-Putz


The performance of non-invasive electroencephalogram-based (EEG) brain–computer interfaces (BCIs) has improved significantly in recent years. However, remaining challenges include the non-stationarity and the low signal-to-noise ratio of the EEG, which limit the bandwidth and hence the available applications. Optimization of both individual components of BCIs and the interrelationship between them is crucial to enhance bandwidth. In other words, neuroscientific knowledge and machine learning need to be optimized by considering concepts from human–computer interaction research and usability. In this paper, we present results of ongoing relevant research in our lab that addresses several important issues for BCIs based on the detection of transient changes in oscillatory EEG activity. First, we report on the long-term stability and robustness of detection of oscillatory EEG components modulated by distinct mental tasks, and show that the use of mental task pairs “mental subtraction versus motor imagery” achieves robust and reliable performance (Cohen’s κ > 0.6) in seven out of nine subjects over a period of 4 days. Second, we report on restricted Boltzmann machines (RBMs) as promising tools for the recognition of oscillatory EEG patterns. In an off-line BCI simulation we computed average peak accuracies, averaged over ten subjects, of 80.8 ± 7.2 %. Third, we present the basic framework of the context-aware hybrid Graz-BCI that allows interacting with the massive multiplayer online role playing game World of Warcraft. We show how a more integrated design approach that considers all components of BCIs, their interrelationships, other input signals and contextual information can increase interaction efficacy.


Hybrid context-aware brain–computer interface (BCI) Electroencephalogram (EEG) Mental imagery Restricted Boltzmann machine World of Warcraft BCI-based gaming 



This work was supported in part by the ICT Collaborative Project BrainAble (247447), the GaLA project (258169), the Wings for Life Spinal Cord Foundation, and ARO award W911NF-11-1-0307.


  1. Allison BZ, Brunner C, Kaiser V, Müller-Putz GR, Neuper C, Pfurtscheller G (2010) Toward a hybrid brain–computer interface based on imagined movement and visual attention. J Neural Eng 7:026,007. doi: 10.1088/1741-2560/7/2/026007 CrossRefGoogle Scholar
  2. Allison BZ, Leeb R, Brunner C, Müller-Putz GR, Bauernfeind G, Kelly JW, Neuper C (2012) Toward smarter BCIs: extending BCIs through hybridization and intelligent control. J Neural Eng 9(1):013001. doi: 10.1088/1741-2560/9/1/013001 Google Scholar
  3. Balderas D, Zander T, Bachl F, Neuper C, Scherer R (2011) Restricted boltzmann machines as useful tool for detecting oscillatory eeg components. In: Proc. of the 5th international brain–computer interface conference, Graz, Austria, pp 68–71Google Scholar
  4. Blankertz B, Dornhege G, Krauledat M, Müller KR, Curio G (2007) The non-invasive Berlin brain–computer interface: fast acquisition of effective performance in untrained subjects. NeuroImage 37:539–550. doi: 10.1016/j.neuroimage.2007.01.051 CrossRefGoogle Scholar
  5. Blankertz B, Tomioka R, Lemm S, Kawanabe M, Müller KR (2008) Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process Mag 25:41–56. doi: 10.1109/MSP.2008.4408441 CrossRefGoogle Scholar
  6. Carreira-Perpinan MA, Hinton GE (2005) On contrastive divergence learning. In: Proceedings of the tenth international workshop on artificial intelligence and statistics, pp 33–40Google Scholar
  7. Chung M, Cheung W, Scherer R, Rao RPN (2011) A hierarchical architecture for adaptive brain–computer interfacing. In: Proceedings of the 22nd international joint conference on artificial intelligence, IJCAI 2011, Barcelona, Catalonia, Spain, July 16–22, 2011 pp 1647–1652Google Scholar
  8. Cohen J (1960) A coefficient of agreement for nominal scales. Psychol Meas 20:37–46CrossRefGoogle Scholar
  9. Darvas F, Scherer R, Ojemann JG, Rao RP, Miller KJ, Sorensen LB (2009) High gamma mapping using EEG. NeuroImage 49:930–938. doi: 10.1016/j.neuroimage.2009.08.041 CrossRefGoogle Scholar
  10. Fatourechi M, Bashashati A, Ward RK, Birch GE (2007) EMG and EOG artifacts in brain computer interface systems: a survey. Clin Neurophysiol 118:480–494CrossRefGoogle Scholar
  11. Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Müller KR, Blankertz B (2012) Enhanced performance by a hybrid NIRSEEG brain computer interface. NeuroImage 59:519–529. doi: 10.1016/j.neuroimage.2011.07.084 CrossRefGoogle Scholar
  12. Friedrich EVC, Scherer R, Neuper C (2012) The effect of distinct mental strategies on classification performance for brain–computer interfaces. Int J Psychophysiol 84(1):86–94Google Scholar
  13. Gevins A, Smith ME, McEvoy L, Yu D (1997) High-resolution eeg mapping of cortical activation related to working memory: effects of task difficulty, type of processing, and practice. Cereb Cortex 7(4):374–385CrossRefGoogle Scholar
  14. Grozea C, Voinescu CD, Fazli S (2011) Bristle-sensors—low-cost flexible passive dry EEG electrodes for neurofeedback and BCI applications. J Neural Eng 8:025,008. doi: 10.1088/1741-2560/8/2/025008 CrossRefGoogle Scholar
  15. Hinton GE (2002) Training products of experts by minimizing contrastive divergence. Neural Comput 14(8):1771–1800Google Scholar
  16. Hinton GE (2010) A practical guide to training restricted boltzmann machines.
  17. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18:1527–1554MathSciNetzbMATHCrossRefGoogle Scholar
  18. Hjorth B (1975) An on-line transformation of EEG scalp potentials into orthogonal source derivations. Electroencephalogr Clin Neurophysiol 39:526–530. doi: 10.1016/0013-4694(75)90056-5 CrossRefGoogle Scholar
  19. Johnson RR, Popovic DP, Olmstead RE, Stikic M, Levendowski DJ, Berka C (2011) Drowsiness/alertness algorithm development and validation using synchronized eeg and cognitive performance to individualize a generalized model. Biol Psychol 87(2):241–250. doi: 10.1016/j.biopsycho.2011.03.003 Google Scholar
  20. Krauledat M, Tangermann M, Blankertz B, Müller KR (2008) Towards zero training for brain–computer interfacing. PLoS ONE 3:e2967. doi: 10.1371/journal.pone.0002967 CrossRefGoogle Scholar
  21. Larochelle H, Bengio Y (2008) Classification using discriminative restricted boltzmann machines. In: ICML 08: proceedings of the 25th international conference on machine learning. ACMGoogle Scholar
  22. Leeb R, Friedman D, Müller-Putz GR, Scherer R, Slater M, Pfurtscheller G (2007) Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: a case study with a tetraplegics. Comput Intell Neurosci 2007:79,642. doi: 10.1155/2007/79642 CrossRefGoogle Scholar
  23. Leeb R, Sagha H, Chavarriaga R, Milln JDR (2011) A hybrid brain–computer interface based on the fusion of electroencephalographic and electromyographic activities. J Neural Eng 8(2):025011. doi: 10.1088/1741-2560/8/2/025011 Google Scholar
  24. Lotte F, Congedo M, Lcuyer A, Lamarche F, Arnaldi B (2007) A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng 4:R1–R13. doi: 10.1088/1741-2560/4/2/R01 CrossRefGoogle Scholar
  25. Mason SG, Bashashati A, Fatourechi M, Navarro KF, Birch GE (2007) A comprehensive survey of brain interface technology designs. Ann Biomed Eng 35:137–169. doi: 10.1007/s10439-006-9170-0 CrossRefGoogle Scholar
  26. Millán J, Mourino J (2003) Asynchronous BCI and local neural classifiers: an overview of the adaptive brain interface project. IEEE Trans Neural Syst Rehabil Eng 11:159–161CrossRefGoogle Scholar
  27. Müller-Gerking J, Pfurtscheller G, Flyvbjerg H (1999) Designing optimal spatial filters for single-trial EEG classification in a movement task. Clin Neurophysiol 110:787–798CrossRefGoogle Scholar
  28. Müller-Putz GR, Eder E, Wriessnegger SC, Pfurtscheller G (2008) Comparison of DFT and lock-in amplifier features and search for optimal electrode positions in SSVEP-based BCI. J Neurosci Methods 168:174–181. doi: 10.1016/j.jneumeth.2007.09.024 CrossRefGoogle Scholar
  29. Müller-Putz GR, Scherer R, Pfurtscheller G, Neuper C (2010) Temporal coding of brain patterns for direct limb control in humans. Front Neurosci 4. doi: 10.3389/fnins.2010.00034
  30. Navarro NA, Ceccaroni L, Velickovski F, Torrellas S, Miralles F, Allison BZ, Scherer R, Faller J (2011) Context-awareness as an enhancement of brain–computer interfaces. In: International workshop on ambient assisted living, Malaga, Spain. Lecture notes in computer science, pp 216–223Google Scholar
  31. Navdeep J, Hinton GE (2011) Learning a better representation of speech sound waves using restricted boltzmann machines. In: Proceedings of the 12th international conference on artificial intelligence and statistics (AISTATS)Google Scholar
  32. Neuper C, Scherer R, Reiner M, Pfurtscheller G (2005) Imagery of motor actions: differential effects of kinesthetic versus visual-motor mode of imagery on single-trial EEG. Brain Res Cognit Brain Res 25:668–677CrossRefGoogle Scholar
  33. Neuper C, Scherer R, Wriessnegger S, Pfurtscheller G (2009) Motor imagery and action observation: modulation of sensorimotor brain rhythms during mental control of a brain–computer interface. Clin Neurophysiol 120:239–247. doi: 10.1016/j.clinph.2008.11.015 CrossRefGoogle Scholar
  34. Pfurtscheller G, Lopes da Silva FH (1999) Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin Neurophysiol 110:1842–1857. doi: 10.1016/S1388-2457(99)00141-8 CrossRefGoogle Scholar
  35. Pfurtscheller G, Scherer R, Müller-Putz GR, Lopes da Silva FH (2008) Short-lived brain state after cued motor imagery in naive subjects. Eur J Neurosci 28:1419–1426. doi: 10.1111/j.1460-9568.2008.06441.x CrossRefGoogle Scholar
  36. Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer R, Zander TO, Müller-Putz G, Neuper C, Birbaumer N (2010) The hybrid BCI. Front Neurosci 4:30. doi: 10.3389/fnpro.2010.00003 Google Scholar
  37. Popescu F, Fazli S, Badower Y, Blankertz B, Müller KR (2007) Single trial classification of motor imagination using 6 dry EEG electrodes. PLoS ONE 2:e637. doi: 10.1371/journal.pone.0000637 CrossRefGoogle Scholar
  38. Pregenzer M, Pfurtscheller G (1999) Frequency component selection for an EEG-based brain to computer interface. IEEE Trans Neural Syst Rehabil Eng 7:413–419CrossRefGoogle Scholar
  39. Pregenzer M, Pfurtscheller G, Flotzinger D (1996) Automated feature selection with a distinction sensitive learning vector quantizer. Neurocomputing 11:19–29. doi: 10.1016/0925-2312(94)00071-9 CrossRefGoogle Scholar
  40. Ramoser H, Müller-Gerking J, Pfurtscheller G (2000) Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 8:441–446CrossRefGoogle Scholar
  41. Rao RPN, Scherer R (2010) Brain–computer interfacing. IEEE Signal Proc Mag 27(4):150–152CrossRefGoogle Scholar
  42. Scherer R, Müller GR, Neuper C, Graimann B, Pfurtscheller G (2004) An asynchronously controlled EEG-based virtual keyboard: improvement of the spelling rate. IEEE Trans Neural Syst Rehabil Eng 51:979–984Google Scholar
  43. Scherer R, Schlögl A, Lee F, Bischof H, Jansa J, Pfurtscheller G (2007) The self-paced Graz brain–computer interface: methods and applications. Comput Intell Neurosci 2007:79,826CrossRefGoogle Scholar
  44. Scherer R, Lee F, Schlögl A, Leeb R, Bischof H, Pfurtscheller G (2008) Toward self-paced brain–computer communication: navigation through virtual worlds. IEEE Trans Biomed Eng 55:675–682. doi: 10.1109/TBME.2007.903709 CrossRefGoogle Scholar
  45. Scherer R, Pfurtscheller G, Neuper C (2008) Motor imagery induced changes in oscillatory ee components: speed vs. accuracy. In: Proceedings of the Graz brain–computer interface workshop 2008Google Scholar
  46. Scherer R, Müller-Putz GR, Pfurtscheller G (2009) Flexibility and practicality: Graz brain–computer interface approach. Int Rev Neurobiol 86:119–131. doi: 10.1016/S0074-7742(09)86009-1 CrossRefGoogle Scholar
  47. Scherer R, Friedrich EVC, Allison BZ, Pröll M, Chung M, Cheung W, Rao RPN, Neuper C (2011) Non-invasive brain–computer interfaces: enhanced gaming and robotic control. Lect Notes Comput Sci 6691:362–369CrossRefGoogle Scholar
  48. Schlögl A, Kronegg J, Huggins JE, Mason SG (2007) Evaluation criteria for BCI research. In: Toward brain–computer interfacing. MIT Press, CambridgeGoogle Scholar
  49. Shenoy P, Krauledat M, Blankertz B, Rao RP, Müller KR (2006) Towards adaptive classification for BCI. J Neural Eng 3:13–23CrossRefGoogle Scholar
  50. Trejo L, Kochavi R, Kubitz K, Montgomery L, Rosipal R, Matthews B (2005) EEG-based estimation of cognitive fatigue. In: SPIE conference proceedingsGoogle Scholar
  51. Usakli AB, Gurkan S, Aloise F, Vecchiato G, Babiloni F (2009) A hybrid platform based on EOG and EEG signals to restore communication for patients afflicted with progressive motor neuron diseases. In: Conference proceedings of the IEEE engineering in medicine and biology societyGoogle Scholar
  52. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain–computer interfaces for communication and control. Clin Neurophysiol 113:767–791. doi: 10.1016/S1388-2457(02)00057-3 CrossRefGoogle Scholar
  53. Zander TO, Jatzev S (2012) Context-aware brain–computer interfaces: exploring the information space of user, technical system and environment. J Neural Eng 9(1):016003. doi: 10.1088/1741-2560/9/1/016003
  54. Zander TO, Kothe C (2011) Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. J Neural Eng 8:025,005. doi: 10.1088/1741-2560/8/2/025005 Google Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  • Reinhold Scherer
    • 1
  • Josef Faller
    • 1
  • David Balderas
    • 1
  • Elisabeth V. C. Friedrich
    • 2
  • Markus Pröll
    • 1
  • Brendan Allison
    • 3
  • Gernot Müller-Putz
    • 1
  1. 1.Institute for Knowledge DiscoveryGraz University of TechnologyGrazAustria
  2. 2.Department of PsychologyUniversity of GrazGrazAustria
  3. 3.Cognitive Science DepartmentUniversity of California San DiegoLa JollaUSA

Personalised recommendations