Is It Significant? Guidelines for Reporting BCI Performance

  • Martin Billinger
  • Ian Daly
  • Vera Kaiser
  • Jing Jin
  • Brendan Z. Allison
  • Gernot R. Müller-Putz
  • Clemens Brunner
Chapter
Part of the Biological and Medical Physics, Biomedical Engineering book series (BIOMEDICAL)

Abstract

Recent growth in brain-computer interface (BCI) research has increased pressure to report improved performance. However, different research groups report performance in different ways. Hence, it is essential that evaluation procedures are valid and reported in sufficient detail. In this chapter we give an overview of available performance measures such as classification accuracy, cohen’s kappa, information transfer rate (ITR), and written symbol rate. We show how to distinguish results from chance level using confidence intervals for accuracy or kappa. Furthermore, we point out common pitfalls when moving from offline to online analysis and provide a guide on how to conduct statistical tests on (BCI) results.

Keywords

Classification Accuracy Confusion Matrix Chance Level Event Related Desynchronization Information Transfer Rate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgements

The views and the conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the corresponding funding agencies.

The research leading to these results has received funding from the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement 248320. In addition, the authors would like to acknowledge the following projects and funding sources:

\(\bullet \) Coupling Measures for BCIs, FWF project P 20848-N15

\(\bullet \) TOBI: Tools for Brain–Computer Interaction, EU project D-1709000020

\(\bullet \) Grant National Natural Science Foundation of China, grant no. 61074113.

We would like to express our gratitude towards the reviewers, who provided invaluable thorough and constructive feedback to improve the quality of this chapter.

References

  1. 1.
    Allison, B.Z.: The I of BCIs: Next Generation Interfaces for Brain-Computer Interface Systems That Adapt to Individual Users. Human-Computer Interaction. Novel Interaction Methods and Techniques, vol. 5611, pp. 558–568. Springer Berlin/Heidelberg (2009)Google Scholar
  2. 2.
    Allison, B.Z.: Toward Ubiquitous BCIs. Brain-Computer Interfaces. The Frontiers Collection, pp. 357–387. Springer Berlin/Heidelberg (2010)Google Scholar
  3. 3.
    Allison, B.Z., Neuper, C.: Could Anyone Use a BCI? Brain-Computer Interfaces. Human-Computer Interaction Series, pp. 35–54. Springer London (2010)Google Scholar
  4. 4.
    Allison, B.Z., Brunner, C., Kaiser, V., Müller-Putz, G.R., Neuper, C., Pfurtscheller, G.: Toward a hybrid brain–computer interface based on imagined movement and visual attention. J. Neural Eng. 7, 026,007 (2010). DOI 10.1088/1741-2560/7/2/026007Google Scholar
  5. 5.
    Atum, Y., Gareis, I., Gentiletti, G., Ruben, A., Rufiner, L.: Genetic feature selection to optimally detect P300 in brain computer interfaces. In: 32nd Annual International Conference of the IEEE EMBS (2010)Google Scholar
  6. 6.
    Bin, G., Gao, X., Wang, Y., Li, Y., Hong, B., Gao, S.: A high-speed BCI based on code modulation VEP. J. Neural Eng. 8, 025,015 (2011). DOI 10.1088/1741-2560/8/2/025015Google Scholar
  7. 7.
    Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., Perelmouter, J., Taub, E., Flor, H.: A spelling device for the paralysed. Nature 398, 297–298 (1999). DOI 10.1038/18581Google Scholar
  8. 8.
    Boomsma, A.: Confidence intervals for a binomial proportion. Unpublished manuscript, university of Groningen, Department of Statistics & Measurement Theory (2005)Google Scholar
  9. 9.
    Bortz, J.: Statistik für Sozialwissenschaftler. Springer, Berlin, Heidelberg, New York (1999)Google Scholar
  10. 10.
    Brunner, C., Allison, B.Z., Krusienski, D.J., Kaiser, V., Müller-Putz, G.R., Pfurtscheller, G., Neuper, C.: Improved signal processing approaches in an offline simulation of a hybrid brain–computer interface. J. Neurosci. Methods 188, 165–173 (2010). DOI 10.1016/j.jneumeth.2010.02.002Google Scholar
  11. 11.
    Brunner, C., Allison, B.Z., Altstätter, C., Neuper, C.: A comparison of three brain–computer interfaces based on event-related desynchronization, steady state visual evoked potentials, or a hybrid approach using both signals. J. Neural Eng. 8, 025,010 (2011a). DOI 10.1088/1741-2560/8/2/025010Google Scholar
  12. 12.
    Brunner, P., Ritaccio, A.L., Emrich, J.F., Bischof, H., Schalk, G.: Rapid communication with a “P300” matrix speller using electrocorticographic signals (ECoG). Front. Neurosci. 5, 5 (2011b)Google Scholar
  13. 13.
    Cabestaing, F., Vaughan, T.M., Mcfarland, D.J., Wolpaw, J.R.: Classification of evoked potentials by Pearson’s correlation in a brain–computer interface. Matrix 67, 156–166 (2007)Google Scholar
  14. 14.
    Chumbley, J.R., Friston, K.J.: False discovery rate revisited: FDR and topological inference using gaussian random fields. NeuroImage 44(1), 62–70 (2009). DOI 10.1016/j.neuroimage.2008.05.021, http://www.ncbi.nlm.nih.gov/pubmed/18603449
  15. 15.
    Cohen, J.: A coefficient of agreement for nominal scales. Psychol. Meas. 20, 37–46 (1960)Google Scholar
  16. 16.
    Cohen, J.: A power primer. Psychol. Bull. 112(1), 155–159 (1992)Google Scholar
  17. 17.
    Dal Seno, B. Matteucci, M., Mainardi, L.: Online detection of P300 and error potentials in a BCI speller. Computational Intelligence and Neuroscience, pp. 1–5 (2010)Google Scholar
  18. 18.
    Daly, I., Nasuto, S., Warwick, K.: Single tap identification for fast BCI control. Cogn. Neurodyn. 5, 21–30 (2011)Google Scholar
  19. 19.
    Dornhege, G., del R Millán, J., Hinterberger, T., McFarland, D.J., Müller, K.R.: (eds.) Towards Brain–Computer Interfacing. MIT Press (2007)Google Scholar
  20. 20.
    Eskandari, P., Erfanian, A.: Improving the performance of brain–computer interface through meditation practicing. In: Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE, pp. 662–665 (2008). DOI 10.1109/IEMBS.2008.4649239Google Scholar
  21. 21.
    Falk, T., Paton, K., Power, S., Chau, T.: Improving the performance of NIRS-based brain–computer interfaces in the presence of background auditory distractions. In: Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on, pp. 517–520 (2010). DOI 10.1109/ICASSP.2010.5495643Google Scholar
  22. 22.
    Farwell, L.A., Donchin, E.: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 70, 510–523 (1988)Google Scholar
  23. 23.
    Furdea, A., Halder, S., Krusienski, D.J., Bross, D., Nijboer, F., Birbaumer, N., Kübler, A.: An auditory oddball (P300) spelling system for brain–computer interfaces. Psychophysiology 46, 1–9 (2009). DOI 10.1111/j.1469-8986.2008.00783.xGoogle Scholar
  24. 24.
    Galán, F., Nuttin, M., Lew, E., Ferrez, P.W., Vanacker, G., Philips, J., del R Millán, J.: A brain-actuated wheelchair: asynchronous and non-invasive brain–computer interfaces for continuous control of robots. Clin. Neurophysiol. 119, 2159–2169 (2008). DOI 10.1016/j.clinph.2008.06.001Google Scholar
  25. 25.
    Gareis, I., Gentiletti, G., Acevedo, R., Rufiner, L.: Feature extraction on brain computer interfaces using discrete dyadic wavelet transform: preliminary results. Journal of Physics: Conference Series (IOP) 313, pp. 1–7 (2011)Google Scholar
  26. 26.
    Genovese, C., Wasserman, L.: Operating characteristics and extensions of the false discovery rate procedure. J. R. Stat. Soc. Series B Stat. Methodol. 64(3), 499–517 (2002). DOI 10.1111/1467-9868.00347, http://doi.wiley.com/10.1111/1467-9868.00347
  27. 27.
    Guger, C., Ramoser, H., Pfurtscheller, G.: Real-time EEG analysis with subject-specific spatial patterns for a brain–computer interface (BCI). IEEE Trans. Neural Syst. Rehabil. Eng. 8, 447–450 (2000). DOI 10.1109/86.895947Google Scholar
  28. 28.
    Hemmelmann, C., Horn, M., Süsse, T., Vollandt, R., Weiss, S.: New concepts of multiple tests and their use for evaluating high-dimensional EEG data. J. Neurosci. Methods 142(2), 209–17 (2005). DOI 10.1016/j.jneumeth.2004.08.008, http://ukpmc.ac.uk/abstract/MED/15698661/reload=1
  29. 29.
    Hild II, K.E., Kurimo, M., Calhoun, V.D.: The sixth annual MLSP competition, 2010. Machine Learning for Signal Proc (MLSP ’10) (2010)Google Scholar
  30. 30.
    Hoffmann, U., Vesin, J.M., Ebrahimi, T., Diserens, K.: An efficient P300-based brain–computer interface for disabled subjects. J. Neurosci. Methods 167, 115–125 (2008). DOI 10.1016/j.jneumeth.2007.03.005Google Scholar
  31. 31.
    Horki, P., Solis-Escalante, T., Neuper, C., Müller-Putz, G.: Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb. Med. Biol. Eng. Comput. (2011). DOI 10.1007/s11517-011-0750-2Google Scholar
  32. 32.
    Huggins, J.E., Levine, S.P., BeMent, S.L., Kushwaha, R.K., Schuh, L.A., Passaro, E.A., Rohde, M.M., Ross, D.A., Elisevich, K.V., Smith, B.J.: Detection of event-related potentials for development of a direct brain interface. J. Clin. Neurophysiol. 16(5), 448 (1999)Google Scholar
  33. 33.
    Jin, J., Allison, B., Sellers, E., Brunner, C., Horki, P., Wang, X., Neuper, C.: Optimized stimulus presentation patterns for an event-related potential EEG-based brain–computer interface. Med. Biol. Eng. Comput. 49, 181–191 (2011). doi:10.1007/s11517-010-0689-8Google Scholar
  34. 34.
    Kalcher, J., Flotzinger, D., Neuper, C., Gölly, S., Pfurtscheller, G.: Graz brain–computer interface II: towards communication between humans and computers based on online classification of three different EEG patterns. Med. Biol. Eng. Comput. 34, 382–388 (1996). DOI 10.1007/BF02520010Google Scholar
  35. 35.
    Karrasch, M., Laine, M., Rapinoja, P., Krause, C.M.: Effects of normal aging on event-related desynchronization/synchronization during a memory task in humans. Neurosci. Lett. 366(1), 18–23 (2004). DOI 10.1016/j.neulet.2004.05.010, http://dx.doi.org/10.1016/j.neulet.2004.05.010
  36. 36.
    Krausz, G., Ortner, R., Opisso, E.: Accuracy of a brain computer interface (p300 spelling device) used by people with motor impairments. Stud. Health Technol. Inform. 167, 182–186 (2011)Google Scholar
  37. 37.
    Kübler, A., Birbaumer, N.: Brain-computer interfaces and communication in paralysis: extinction of goal directed thinking in completely paralysed patients? Clin. Neurophysiol. 119, 2658–2666 (2008). DOI 10.1016/j.clinph.2008.06.019Google Scholar
  38. 38.
    Kübler, A., Nijboer, F., Mellinger, J., Vaughan, T.M., Pawelzik, H., Schalk, G., McFarland, D.J., Birbaumer, N., Wolpaw, J.R.: Patients with ALS can use sensorimotor rhythms to operate a braincomputer interface. Neurology 64, 1775–1777 (2005)Google Scholar
  39. 39.
    Lemm, S., Blankertz, B. Dickhaus, T., Müller, K.R.: Introduction to machine learning for brain imaging. NeuroImage 56(2), pp. 387–399 (2011)Google Scholar
  40. 40.
    Lotte, F.: Generating artificial EEG signals to reduce BCI calibration time. In: Proceedings of the 5th International Brain–Computer Interface Conference 2011, pp. 176–179 (2011)Google Scholar
  41. 41.
    Mason, S.G., Birch, G.E.: A brain-controlled switch for asynchronous control applications. IEEE Trans. Biomed. Eng. 47, 1297–1307 (2000)Google Scholar
  42. 42.
    Millán, J., Mouriño, J.: Asynchronous BCI and local neural classifiers: an overview of the adaptive brain interface project. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 159–161 (2003)Google Scholar
  43. 43.
    Millán, J., Mouriño, J., Franzé M., Cincotti, F., Varsta, M., Heikkonen, J., Babiloni, F.: A local neural classifier for the recognition of EEG patterns associated to mental tasks. IEEE Trans. Neural Netw. 13, 678–686 (2002)Google Scholar
  44. 44.
    Müller, K.R., Anderson, C.W., Birch, G.E.: Linear and nonlinear methods for brain–computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 165–169 (2003)Google Scholar
  45. 45.
    Müller, K.R., Tangermann, M., Dornhege, G., Krauledat, M., Curio, G., Blankertz, B.: Machine learning for real-time single-trial EEG analysis: from brain–computer interfacing to mental state monitoring. J. Neurosci. Meth. 167, 82–90 (2008). DOI 10.1016/j.jneumeth.2007.09.022Google Scholar
  46. 46.
    Müller-Putz, G.R., Pfurtscheller, G.: Control of an electrical prosthesis with an SSVEP-based BCI. IEEE Trans. Biomed. Eng. 55, 361–364 (2008). DOI 10.1109/TBME.2007.897815Google Scholar
  47. 47.
    Müller-Putz, G.R., Scherer, R., Pfurtscheller, G., Rupp, R.: EEG-based neuroprosthesis control: a step towards clinical practice. Neurosci. Lett. 382, 169–174 (2005)Google Scholar
  48. 48.
    Müller-Putz, G.R., Scherer, R., Brunner, C., Leeb, R., Pfurtscheller, G.: Better than random? A closer look on BCI results. Int. J. Bioelectromagn. 10, 52–55 (2008)Google Scholar
  49. 49.
    Neuper, C., Müller, G.R., Kübler, A., Birbaumer, N., Pfurtscheller, G.: Clinical application of an EEG-based brain–computer interface: a case study in a patient with severe motor impairment. Clin. Neurophysiol. 114, 399–409 (2003)Google Scholar
  50. 50.
    Pfurtscheller, G., Neuper, C.: Motor imagery and direct brain–computer communication. Proc. IEEE 89, 1123–1134 (2001). DOI 10.1109/5.939829Google Scholar
  51. 51.
    Pfurtscheller, G., Müller, G.R., Pfurtscheller, J., Gerner, H.J., Rupp, R.: “Thought”-control of functional electrical stimulation to restore handgrasp in a patient with tetraplegia. Neurosci. Lett. 351, 33–36 (2003). DOI 10.1016/S0304-3940(03)00947-9Google Scholar
  52. 52.
    Piccione, F., Giorgi, F., Tonin, P., Priftis, K., Giove, S., Silvoni, S., Palmas, G., Beverina, F.: P300-based brain computer interface: reliability and performance in healthy and paralysed participants. Clin. Neurophysiol. 117, 531–537 (2006). DOI 10.1016/j.clinph.2005.07.024Google Scholar
  53. 53.
    Rebsamen, B., Guan, C., Zhang, H., Wang, C., Teo, C., Ang, M.H., Burdet, E.: A brain controlled wheelchair to navigate in familiar environments. IEEE Trans. Neural Syst. Rehabil. Eng. 18(6), 590–598 (2010). DOI 10.1109/TNSRE.2010.2049862, http://dx.doi.org/10.1109/TNSRE.2010.2049862
  54. 54.
    Roberts, S., Penny, W., Rezek, I.: Temporal and spatial complexity measures for electroencephalogram based brain–computer interfacing. Med. Biol. Eng. Comput. 37, 93–98 (1999). doi:10.1007/BF02513272Google Scholar
  55. 55.
    Ryan, D.B., Frye, G.E., Townsend, G., Berry, D.R., Mesa-G, S., Gates, N.A., Sellers, E.W.: Predictive spelling with a P300-based brain–computer interface: Increasing the rate of communication. Int. J. Hum. Comput. Interact. 27, 69–84 (2011). DOI 10.1080/10447318.2011.535754Google Scholar
  56. 56.
    Schalk, G., Wolpaw, J.R., McFarland, D.J., Pfurtscheller, G.: EEG-based communication: presence of an error potential. Clin. Neurophysiol. 111, 2138–2144 (2000)Google Scholar
  57. 57.
    Schlögl, A., Kronegg, J., Huggins, J.E., Mason, S.G.: Evaluation criteria for BCI research. In: Toward brain–computer interfacing. MIT Press (2007)Google Scholar
  58. 58.
    Shannon, C.E., Weaver, W.: A mathematical theory of communication. University of Illinois Press (1964)Google Scholar
  59. 59.
    Singh, A.K., Phillips, S.: Hierarchical control of false discovery rate for phase locking measures of EEG synchrony. NeuroImage 50(1), 40–47 (2010). DOI 10.1016/j.neuroimage.2009.12.030, http://dx.doi.org/10.1016/j.neuroimage.2009.12.030
  60. 60.
    Sitaram, R., Zhang, H., Guan, C., Thulasidas, M., Hoshi, Y., Ishikawa, A., Shimizu, K., Birbaumer, N.: Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface. NeuroImage 34, 1416–1427 (2007)Google Scholar
  61. 61.
    Townsend, G., LaPallo, B.K., Boulay, C.B., Krusienski, D.J., Frye, G.E., Hauser, C.K., Schwartz, N.E., Vaughan, T.M., Wolpaw, J.R., Sellers, E.W.: A novel P300-based brain–computer interface stimulus presentation paradigm: Moving beyond rows and columns. Clin. Neurophysiol. 121, 1109–1120 (2010)Google Scholar
  62. 62.
    Vidaurre, C., Blankertz, B.: Towards a cure for BCI illiteracy. Brain Topogr. 23, 194–198 (2010). DOI 10.1007/s10548-009-0121-6Google Scholar
  63. 63.
    Volosyak, I.: SSVEP-based Bremen-BCI interface – boosting information transfer rates. J. Neural Eng. 8, 036,020 (2011). DOI 10.1088/1741-2560/8/3/036020Google Scholar
  64. 64.
    Wolpaw, J.R.: Brain-computer interfaces as new brain output pathways. J. Physiol. 579, 623–619 (2007). DOI 10.1113/jphysiol.2006.125948Google Scholar
  65. 65.
    Wolpaw, J.R., Flotzinger, D., Pfurtscheller, G., McFarland, D.J.: Timing of EEG-based cursor control. J. Clin. Neurophysiol. 14(6), 529–538 (1997)Google Scholar
  66. 66.
    Wolpaw, J.R., Birbaumer, N., Heetderks, W.J., McFarland, D.J., Peckham, P.H., Schalk, G., Donchin, E., Quatrano, L.A., Robinson, C.J., Vaughan, T.M.: Brain-computer interface technology: a review of the first international meeting. IEEE Trans. Rehabil. Eng. 8, 164–173 (2000). DOI 10.1109/TRE.2000.847807Google Scholar
  67. 67.
    Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791 (2002). DOI 10.1016/S1388-2457(02)00057-3Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Martin Billinger
    • 1
  • Ian Daly
    • 1
  • Vera Kaiser
    • 1
  • Jing Jin
    • 3
  • Brendan Z. Allison
    • 1
  • Gernot R. Müller-Putz
    • 1
  • Clemens Brunner
    • 1
    • 2
  1. 1.Institute for Knowledge DiscoveryGraz University of TechnologyGrazAustria
  2. 2.Swartz Center for Computational NeuroscienceINC, UCSDSan DiegoUSA
  3. 3.Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of EducationEast China University of Science and TechnologyShanghaiChina

Personalised recommendations