Skip to main content
SpringerLink
Log in

Is It Significant? Guidelines for Reporting BCI Performance

  • Chapter
  • First Online:
Towards Practical Brain-Computer Interfaces

Part of the book series: Biological and Medical Physics, Biomedical Engineering ((BIOMEDICAL))

Abstract

Recent growth in brain-computer interface (BCI) research has increased pressure to report improved performance. However, different research groups report performance in different ways. Hence, it is essential that evaluation procedures are valid and reported in sufficient detail. In this chapter we give an overview of available performance measures such as classification accuracy, cohen’s kappa, information transfer rate (ITR), and written symbol rate. We show how to distinguish results from chance level using confidence intervals for accuracy or kappa. Furthermore, we point out common pitfalls when moving from offline to online analysis and provide a guide on how to conduct statistical tests on (BCI) results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Such randomness is not necessarily caused by the classifier alone. The BCI user failing at the task, electrode failures or inadequate features may all decrease the degree of agreement between the estimated and true class labels. The actual source of randomness is not relevant for this analysis.

References

  1. Allison, B.Z.: The I of BCIs: Next Generation Interfaces for Brain-Computer Interface Systems That Adapt to Individual Users. Human-Computer Interaction. Novel Interaction Methods and Techniques, vol. 5611, pp. 558–568. Springer Berlin/Heidelberg (2009)

    Google Scholar 

  2. Allison, B.Z.: Toward Ubiquitous BCIs. Brain-Computer Interfaces. The Frontiers Collection, pp. 357–387. Springer Berlin/Heidelberg (2010)

    Google Scholar 

  3. Allison, B.Z., Neuper, C.: Could Anyone Use a BCI? Brain-Computer Interfaces. Human-Computer Interaction Series, pp. 35–54. Springer London (2010)

    Google Scholar 

  4. Allison, B.Z., Brunner, C., Kaiser, V., Müller-Putz, G.R., Neuper, C., Pfurtscheller, G.: Toward a hybrid brain–computer interface based on imagined movement and visual attention. J. Neural Eng. 7, 026,007 (2010). DOI 10.1088/1741-2560/7/2/026007

    Google Scholar 

  5. Atum, Y., Gareis, I., Gentiletti, G., Ruben, A., Rufiner, L.: Genetic feature selection to optimally detect P300 in brain computer interfaces. In: 32nd Annual International Conference of the IEEE EMBS (2010)

    Google Scholar 

  6. Bin, G., Gao, X., Wang, Y., Li, Y., Hong, B., Gao, S.: A high-speed BCI based on code modulation VEP. J. Neural Eng. 8, 025,015 (2011). DOI 10.1088/1741-2560/8/2/025015

    Google Scholar 

  7. Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., Perelmouter, J., Taub, E., Flor, H.: A spelling device for the paralysed. Nature 398, 297–298 (1999). DOI 10.1038/18581

    Google Scholar 

  8. Boomsma, A.: Confidence intervals for a binomial proportion. Unpublished manuscript, university of Groningen, Department of Statistics & Measurement Theory (2005)

    Google Scholar 

  9. Bortz, J.: Statistik für Sozialwissenschaftler. Springer, Berlin, Heidelberg, New York (1999)

    Google Scholar 

  10. Brunner, C., Allison, B.Z., Krusienski, D.J., Kaiser, V., Müller-Putz, G.R., Pfurtscheller, G., Neuper, C.: Improved signal processing approaches in an offline simulation of a hybrid brain–computer interface. J. Neurosci. Methods 188, 165–173 (2010). DOI 10.1016/j.jneumeth.2010.02.002

    Google Scholar 

  11. Brunner, C., Allison, B.Z., Altstätter, C., Neuper, C.: A comparison of three brain–computer interfaces based on event-related desynchronization, steady state visual evoked potentials, or a hybrid approach using both signals. J. Neural Eng. 8, 025,010 (2011a). DOI 10.1088/1741-2560/8/2/025010

    Google Scholar 

  12. Brunner, P., Ritaccio, A.L., Emrich, J.F., Bischof, H., Schalk, G.: Rapid communication with a “P300” matrix speller using electrocorticographic signals (ECoG). Front. Neurosci. 5, 5 (2011b)

    Google Scholar 

  13. Cabestaing, F., Vaughan, T.M., Mcfarland, D.J., Wolpaw, J.R.: Classification of evoked potentials by Pearson’s correlation in a brain–computer interface. Matrix 67, 156–166 (2007)

    Google Scholar 

  14. Chumbley, J.R., Friston, K.J.: False discovery rate revisited: FDR and topological inference using gaussian random fields. NeuroImage 44(1), 62–70 (2009). DOI 10.1016/j.neuroimage.2008.05.021, http://www.ncbi.nlm.nih.gov/pubmed/18603449

  15. Cohen, J.: A coefficient of agreement for nominal scales. Psychol. Meas. 20, 37–46 (1960)

    Google Scholar 

  16. Cohen, J.: A power primer. Psychol. Bull. 112(1), 155–159 (1992)

    Google Scholar 

  17. Dal Seno, B. Matteucci, M., Mainardi, L.: Online detection of P300 and error potentials in a BCI speller. Computational Intelligence and Neuroscience, pp. 1–5 (2010)

    Google Scholar 

  18. Daly, I., Nasuto, S., Warwick, K.: Single tap identification for fast BCI control. Cogn. Neurodyn. 5, 21–30 (2011)

    Google Scholar 

  19. Dornhege, G., del R Millán, J., Hinterberger, T., McFarland, D.J., Müller, K.R.: (eds.) Towards Brain–Computer Interfacing. MIT Press (2007)

    Google Scholar 

  20. Eskandari, P., Erfanian, A.: Improving the performance of brain–computer interface through meditation practicing. In: Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE, pp. 662–665 (2008). DOI 10.1109/IEMBS.2008.4649239

    Google Scholar 

  21. Falk, T., Paton, K., Power, S., Chau, T.: Improving the performance of NIRS-based brain–computer interfaces in the presence of background auditory distractions. In: Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on, pp. 517–520 (2010). DOI 10.1109/ICASSP.2010.5495643

    Google Scholar 

  22. Farwell, L.A., Donchin, E.: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 70, 510–523 (1988)

    Google Scholar 

  23. Furdea, A., Halder, S., Krusienski, D.J., Bross, D., Nijboer, F., Birbaumer, N., Kübler, A.: An auditory oddball (P300) spelling system for brain–computer interfaces. Psychophysiology 46, 1–9 (2009). DOI 10.1111/j.1469-8986.2008.00783.x

    Google Scholar 

  24. Galán, F., Nuttin, M., Lew, E., Ferrez, P.W., Vanacker, G., Philips, J., del R Millán, J.: A brain-actuated wheelchair: asynchronous and non-invasive brain–computer interfaces for continuous control of robots. Clin. Neurophysiol. 119, 2159–2169 (2008). DOI 10.1016/j.clinph.2008.06.001

    Google Scholar 

  25. Gareis, I., Gentiletti, G., Acevedo, R., Rufiner, L.: Feature extraction on brain computer interfaces using discrete dyadic wavelet transform: preliminary results. Journal of Physics: Conference Series (IOP) 313, pp. 1–7 (2011)

    Google Scholar 

  26. Genovese, C., Wasserman, L.: Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. Series B Stat. Methodol. 64(3), 499–517 (2002). DOI 10.1111/1467-9868.00347, http://doi.wiley.com/10.1111/1467-9868.00347

  27. Guger, C., Ramoser, H., Pfurtscheller, G.: Real-time EEG analysis with subject-specific spatial patterns for a brain–computer interface (BCI). IEEE Trans. Neural Syst. Rehabil. Eng. 8, 447–450 (2000). DOI 10.1109/86.895947

    Google Scholar 

  28. Hemmelmann, C., Horn, M., Süsse, T., Vollandt, R., Weiss, S.: New concepts of multiple tests and their use for evaluating high-dimensional EEG data. J. Neurosci. Methods 142(2), 209–17 (2005). DOI 10.1016/j.jneumeth.2004.08.008, http://ukpmc.ac.uk/abstract/MED/15698661/reload=1

  29. Hild II, K.E., Kurimo, M., Calhoun, V.D.: The sixth annual MLSP competition, 2010. Machine Learning for Signal Proc (MLSP ’10) (2010)

    Google Scholar 

  30. Hoffmann, U., Vesin, J.M., Ebrahimi, T., Diserens, K.: An efficient P300-based brain–computer interface for disabled subjects. J. Neurosci. Methods 167, 115–125 (2008). DOI 10.1016/j.jneumeth.2007.03.005

    Google Scholar 

  31. Horki, P., Solis-Escalante, T., Neuper, C., Müller-Putz, G.: Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb. Med. Biol. Eng. Comput. (2011). DOI 10.1007/s11517-011-0750-2

    Google Scholar 

  32. Huggins, J.E., Levine, S.P., BeMent, S.L., Kushwaha, R.K., Schuh, L.A., Passaro, E.A., Rohde, M.M., Ross, D.A., Elisevich, K.V., Smith, B.J.: Detection of event-related potentials for development of a direct brain interface. J. Clin. Neurophysiol. 16(5), 448 (1999)

    Google Scholar 

  33. Jin, J., Allison, B., Sellers, E., Brunner, C., Horki, P., Wang, X., Neuper, C.: Optimized stimulus presentation patterns for an event-related potential EEG-based brain–computer interface. Med. Biol. Eng. Comput. 49, 181–191 (2011). doi:10.1007/s11517-010-0689-8

    Google Scholar 

  34. Kalcher, J., Flotzinger, D., Neuper, C., Gölly, S., Pfurtscheller, G.: Graz brain–computer interface II: towards communication between humans and computers based on online classification of three different EEG patterns. Med. Biol. Eng. Comput. 34, 382–388 (1996). DOI 10.1007/BF02520010

    Google Scholar 

  35. Karrasch, M., Laine, M., Rapinoja, P., Krause, C.M.: Effects of normal aging on event-related desynchronization/synchronization during a memory task in humans. Neurosci. Lett. 366(1), 18–23 (2004). DOI 10.1016/j.neulet.2004.05.010, http://dx.doi.org/10.1016/j.neulet.2004.05.010

  36. Krausz, G., Ortner, R., Opisso, E.: Accuracy of a brain computer interface (p300 spelling device) used by people with motor impairments. Stud. Health Technol. Inform. 167, 182–186 (2011)

    Google Scholar 

  37. Kübler, A., Birbaumer, N.: Brain-computer interfaces and communication in paralysis: extinction of goal directed thinking in completely paralysed patients? Clin. Neurophysiol. 119, 2658–2666 (2008). DOI 10.1016/j.clinph.2008.06.019

    Google Scholar 

  38. Kübler, A., Nijboer, F., Mellinger, J., Vaughan, T.M., Pawelzik, H., Schalk, G., McFarland, D.J., Birbaumer, N., Wolpaw, J.R.: Patients with ALS can use sensorimotor rhythms to operate a braincomputer interface. Neurology 64, 1775–1777 (2005)

    Google Scholar 

  39. Lemm, S., Blankertz, B. Dickhaus, T., Müller, K.R.: Introduction to machine learning for brain imaging. NeuroImage 56(2), pp. 387–399 (2011)

    Google Scholar 

  40. Lotte, F.: Generating artificial EEG signals to reduce BCI calibration time. In: Proceedings of the 5th International Brain–Computer Interface Conference 2011, pp. 176–179 (2011)

    Google Scholar 

  41. Mason, S.G., Birch, G.E.: A brain-controlled switch for asynchronous control applications. IEEE Trans. Biomed. Eng. 47, 1297–1307 (2000)

    Google Scholar 

  42. Millán, J., Mouriño, J.: Asynchronous BCI and local neural classifiers: an overview of the adaptive brain interface project. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 159–161 (2003)

    Google Scholar 

  43. Millán, J., Mouriño, J., Franzé M., Cincotti, F., Varsta, M., Heikkonen, J., Babiloni, F.: A local neural classifier for the recognition of EEG patterns associated to mental tasks. IEEE Trans. Neural Netw. 13, 678–686 (2002)

    Google Scholar 

  44. Müller, K.R., Anderson, C.W., Birch, G.E.: Linear and nonlinear methods for brain–computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 165–169 (2003)

    Google Scholar 

  45. Müller, K.R., Tangermann, M., Dornhege, G., Krauledat, M., Curio, G., Blankertz, B.: Machine learning for real-time single-trial EEG analysis: from brain–computer interfacing to mental state monitoring. J. Neurosci. Meth. 167, 82–90 (2008). DOI 10.1016/j.jneumeth.2007.09.022

    Google Scholar 

  46. Müller-Putz, G.R., Pfurtscheller, G.: Control of an electrical prosthesis with an SSVEP-based BCI. IEEE Trans. Biomed. Eng. 55, 361–364 (2008). DOI 10.1109/TBME.2007.897815

    Google Scholar 

  47. Müller-Putz, G.R., Scherer, R., Pfurtscheller, G., Rupp, R.: EEG-based neuroprosthesis control: a step towards clinical practice. Neurosci. Lett. 382, 169–174 (2005)

    Google Scholar 

  48. Müller-Putz, G.R., Scherer, R., Brunner, C., Leeb, R., Pfurtscheller, G.: Better than random? A closer look on BCI results. Int. J. Bioelectromagn. 10, 52–55 (2008)

    Google Scholar 

  49. Neuper, C., Müller, G.R., Kübler, A., Birbaumer, N., Pfurtscheller, G.: Clinical application of an EEG-based brain–computer interface: a case study in a patient with severe motor impairment. Clin. Neurophysiol. 114, 399–409 (2003)

    Google Scholar 

  50. Pfurtscheller, G., Neuper, C.: Motor imagery and direct brain–computer communication. Proc. IEEE 89, 1123–1134 (2001). DOI 10.1109/5.939829

    Google Scholar 

  51. Pfurtscheller, G., Müller, G.R., Pfurtscheller, J., Gerner, H.J., Rupp, R.: “Thought”-control of functional electrical stimulation to restore handgrasp in a patient with tetraplegia. Neurosci. Lett. 351, 33–36 (2003). DOI 10.1016/S0304-3940(03)00947-9

    Google Scholar 

  52. Piccione, F., Giorgi, F., Tonin, P., Priftis, K., Giove, S., Silvoni, S., Palmas, G., Beverina, F.: P300-based brain computer interface: reliability and performance in healthy and paralysed participants. Clin. Neurophysiol. 117, 531–537 (2006). DOI 10.1016/j.clinph.2005.07.024

    Google Scholar 

  53. Rebsamen, B., Guan, C., Zhang, H., Wang, C., Teo, C., Ang, M.H., Burdet, E.: A brain controlled wheelchair to navigate in familiar environments. IEEE Trans. Neural Syst. Rehabil. Eng. 18(6), 590–598 (2010). DOI 10.1109/TNSRE.2010.2049862, http://dx.doi.org/10.1109/TNSRE.2010.2049862

  54. Roberts, S., Penny, W., Rezek, I.: Temporal and spatial complexity measures for electroencephalogram based brain–computer interfacing. Med. Biol. Eng. Comput. 37, 93–98 (1999). doi:10.1007/BF02513272

    Google Scholar 

  55. Ryan, D.B., Frye, G.E., Townsend, G., Berry, D.R., Mesa-G, S., Gates, N.A., Sellers, E.W.: Predictive spelling with a P300-based brain–computer interface: Increasing the rate of communication. Int. J. Hum. Comput. Interact. 27, 69–84 (2011). DOI 10.1080/10447318.2011.535754

    Google Scholar 

  56. Schalk, G., Wolpaw, J.R., McFarland, D.J., Pfurtscheller, G.: EEG-based communication: presence of an error potential. Clin. Neurophysiol. 111, 2138–2144 (2000)

    Google Scholar 

  57. Schlögl, A., Kronegg, J., Huggins, J.E., Mason, S.G.: Evaluation criteria for BCI research. In: Toward brain–computer interfacing. MIT Press (2007)

    Google Scholar 

  58. Shannon, C.E., Weaver, W.: A mathematical theory of communication. University of Illinois Press (1964)

    Google Scholar 

  59. Singh, A.K., Phillips, S.: Hierarchical control of false discovery rate for phase locking measures of EEG synchrony. NeuroImage 50(1), 40–47 (2010). DOI 10.1016/j.neuroimage.2009.12.030, http://dx.doi.org/10.1016/j.neuroimage.2009.12.030

  60. Sitaram, R., Zhang, H., Guan, C., Thulasidas, M., Hoshi, Y., Ishikawa, A., Shimizu, K., Birbaumer, N.: Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface. NeuroImage 34, 1416–1427 (2007)

    Google Scholar 

  61. Townsend, G., LaPallo, B.K., Boulay, C.B., Krusienski, D.J., Frye, G.E., Hauser, C.K., Schwartz, N.E., Vaughan, T.M., Wolpaw, J.R., Sellers, E.W.: A novel P300-based brain–computer interface stimulus presentation paradigm: Moving beyond rows and columns. Clin. Neurophysiol. 121, 1109–1120 (2010)

    Google Scholar 

  62. Vidaurre, C., Blankertz, B.: Towards a cure for BCI illiteracy. Brain Topogr. 23, 194–198 (2010). DOI 10.1007/s10548-009-0121-6

    Google Scholar 

  63. Volosyak, I.: SSVEP-based Bremen-BCI interface – boosting information transfer rates. J. Neural Eng. 8, 036,020 (2011). DOI 10.1088/1741-2560/8/3/036020

    Google Scholar 

  64. Wolpaw, J.R.: Brain-computer interfaces as new brain output pathways. J. Physiol. 579, 623–619 (2007). DOI 10.1113/jphysiol.2006.125948

    Google Scholar 

  65. Wolpaw, J.R., Flotzinger, D., Pfurtscheller, G., McFarland, D.J.: Timing of EEG-based cursor control. J. Clin. Neurophysiol. 14(6), 529–538 (1997)

    Google Scholar 

  66. Wolpaw, J.R., Birbaumer, N., Heetderks, W.J., McFarland, D.J., Peckham, P.H., Schalk, G., Donchin, E., Quatrano, L.A., Robinson, C.J., Vaughan, T.M.: Brain-computer interface technology: a review of the first international meeting. IEEE Trans. Rehabil. Eng. 8, 164–173 (2000). DOI 10.1109/TRE.2000.847807

    Google Scholar 

  67. Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791 (2002). DOI 10.1016/S1388-2457(02)00057-3

    Google Scholar 

Download references

Acknowledgements

The views and the conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the corresponding funding agencies.

The research leading to these results has received funding from the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement 248320. In addition, the authors would like to acknowledge the following projects and funding sources:

\(\bullet \) Coupling Measures for BCIs, FWF project P 20848-N15

\(\bullet \) TOBI: Tools for Brain–Computer Interaction, EU project D-1709000020

\(\bullet \) Grant National Natural Science Foundation of China, grant no. 61074113.

We would like to express our gratitude towards the reviewers, who provided invaluable thorough and constructive feedback to improve the quality of this chapter.

Author information

Authors and Affiliations

  1. Institute for Knowledge Discovery, Graz University of Technology, Graz, Austria

    Martin Billinger, Ian Daly, Vera Kaiser, Brendan Z. Allison, Gernot R. Müller-Putz & Clemens Brunner

  2. Swartz Center for Computational Neuroscience, INC, UCSD, San Diego, CA, USA

    Clemens Brunner

  3. Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai, China

    Jing Jin

Authors
  1. Martin Billinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ian Daly
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vera Kaiser
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Brendan Z. Allison
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gernot R. Müller-Putz
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Clemens Brunner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin Billinger .

Editor information

Editors and Affiliations

  1. Baltimore Drive 5700Apt 167, La Mesa, 91942, California, USA

    Brendan Z. Allison

  2. Starlab Neuroscience Research, Teodor Roviralta 45, Barcelona, 08022, Spain

    Stephen Dunne

  3. , School of Engineering, Swiss Federal Institute of Technology, EPFL STI-CNBI, ELB 141, Station 11, Lausanne, 1015, Switzerland

    Robert Leeb

  4. , School of Engineering, Swiss Federal Institute of Technology, EPFL STI-CNBI, ELB 138. Station 11, Lausanne, 1015, Switzerland

    José Del R. Millán

  5. , Faculty EEMCS, University of Twente, Enschede, 7500, Netherlands

    Anton Nijholt

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Billinger, M. et al. (2012). Is It Significant? Guidelines for Reporting BCI Performance. In: Allison, B., Dunne, S., Leeb, R., Del R. Millán, J., Nijholt, A. (eds) Towards Practical Brain-Computer Interfaces. Biological and Medical Physics, Biomedical Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29746-5_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-29746-5_17

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-29745-8

  • Online ISBN: 978-3-642-29746-5

  • eBook Packages: Physics and AstronomyPhysics and Astronomy (R0)

Publish with us

Policies and ethics

Search

Navigation