Skip to main content
SpringerLink
Log in

Usability Evaluation of BCI Software Applications: A systematic review of the literature

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

Brain Computer Interfaces- BCI allow users to communicate with the software system through cognitive functions measurable by brain signals, identified as Electroencephalography-EEG. User tests have been the most used method for the evaluation of BCI software applications. In user tests, the data collected comes from the opinions of users through questionnaires, these tests require a lot of time, since they include not only performing interaction task and the application of the questionnaires, but also include placing and calibrating the EEG device. All this makes the evaluation process a very heavy task for the participants and can mean that the data collected is not entirely reliable. That is why we are interested in including EEG signals in the usability evaluation process of applications with BCI. Therefore, we present in this paper the result of the systematic mapping of the literature in order to identify the relevant works in the area and future lines of research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.

Similar content being viewed by others

REFERENCES

  1. Ansari-Asl, K., Chanel, G., and Pun, T., A channel selection method for EEG classification in emotion assessment based on synchronization likelihood, Proc. 15th European Signal Processing Conf., Poznan, 2007, pp. 1241–1245.

  2. Antonenko, P., Paas, F., Grabner, R., and Van Gog, T., Using electroencephalography to measure cognitive load, Educ. Psychol. Rev., 2010, vol. 22, no. 4, pp. 425–438.

    Article  Google Scholar 

  3. Appriou, A., Cichocki, A., and Lotte, F., Towards robust neuroadaptive HCI: exploring modern machine learning methods to estimate mental workload from EEG signals, Proc. 2018 CHI Conf. on Human Factors in Computing Systems, Montreal, 2018, pp. 1–6.

  4. Arai, K. and Mardiyanto, R., Evaluation of users’ impact for using the proposed eye based HCI with moving and fixed keyboard by using eeg signals, Int. J. Res. Rev. Comput. Sci., 2011, vol. 2, no. 6, p. 1228.

    Google Scholar 

  5. Arai, K. and Mardiyanto, R., Eye based HCI with moving keyboard for reducing fatigue effects, Proc. 8th Int. Conf. on Information Technology: New Generations, Las Vegas, 2011, pp. 417–422.

  6. Bhardwaj, A., Gupta, A., Jain, P., Rani, A., and Yadav, J., Classification of human emotions from EEG signals using SVM and LDA classifiers, Proc. 2nd Int. Conf. on Signal Processing and Integrated Networks (SPIN), Noida, 2015, pp. 180–185.

  7. Bos, D.P.O., Reuderink, B., van de Laar, B., Gürkök, H., Mühl, C., Poel, M., and Nijholt, A., Human-computer interaction for BCI games: usability and user experience, Proc. IEEE Int. Conf. on Cyberworlds, Singapore, 2010, pp. 277–281.

  8. Charisis, V., Hadjidimitriou, S., Hadjileontiadis, L., Uğurca, D., and Yilmaz, E., EmoActivity-an EEG-based gamified emotion HCI for augmented artistic expression: the i-treasures paradigm, Proc. Int. Conf. on Universal Access in Human-Computer Interaction, Cham: Springer, 2015, pp. 29–40.

  9. Chin, J.P., Diehl, V.A., and Norman, K.L., Development of an instrument measuring user satisfaction of the human-computer interface, Proc. SIGCHI Conf. on Human Factors in Computing Systems, Washington, 1988, pp. 213–218.

  10. Chowdhury, A., Meena, Y.K., Raza, H., Bhushan, B., Uttam, A.K., Pandey, N., and Prasad, G., Active physical practice followed by mental practice using BCI-driven hand exoskeleton: a pilot trial for clinical effectiveness and usability, IEEE J. Biomed. Health Inf., 2018, vol. 22, no. 6, pp. 1786–1795.

    Article  Google Scholar 

  11. Crichton, N., Visual analogue scale (VAS), J. Clin. Nurs., 2001, vol. 10, no. 5, p. 706-6.

    Google Scholar 

  12. Erkan, E. and Akbaba, M., A study on performance increasing in SSVEP based BCI application, Int. J. Eng. Sci. Technol., 2018, vol. 21, no. 3, pp. 421–427.

    Google Scholar 

  13. Freidman, V. and Mielke, C., A Field Guide to Usability Testing, Smashing Media, 2013.

    Google Scholar 

  14. Frey, J., Daniel, M., Castet, J., Hachet, M., and Lotte, F., Framework for electroencephalography-based evaluation of user experience, Proc. 2016 CHI Conf. on Human Factors in Computing Systems, San Jose, CA, 2016, pp. 2283–2294.

  15. Frey, J., Pommereau, L., Lotte, F., and Hachet, M., Assessing the zone of comfort in stereoscopic displays using EEG, Proc. CHI’14 Conf. on Human Factors in Computing Systems, Toronto, 2014, pp. 2041–2046.

  16. García Ramírez, A.R., Da Silva, J.F., Savall, A.C.R., Catecati, T., and Ferreira, M.G.G., User’s emotions and usability study of a brain-computer interface applied to people with cerebral palsy, Technologies, 2018, vol. 6, no. 1, p. 28.

    Article  Google Scholar 

  17. Gentiletti, G., Tabernig, C., and Acevedo, R., Interfaz cerebro-computadora: estado del arte y desarrollo en Argentina, Rev. Argentina Bioing., Rev. SABI, 2007, vol. 13, no. 1, pp. 22–29.

    Google Scholar 

  18. Hartson, H.R., Andre, T.S., and Williges, R.C., Criteria for evaluating usability evaluation methods, Int. J. Human-Comput. Interact., 2003, vol. 15, no. 1, pp. 145–181.

    Article  Google Scholar 

  19. Hosseini, S.A. and Khalilzadeh, M.A., Emotional stress recognition system using EEG and psychophysiological signals: using new labelling process of EEG signals in emotional stress state, Proc. IEEE Int. Conf. on Biomedical Engineering and Computer Science, Wuhan, 2010, pp. 1–6.

  20. Kortelainen, J. and Seppänen, T., EEG-based recognition of video-induced emotions: selecting subject-independent feature set, Proc. 35th IEEE Annu. Int. Conf. on Engineering in Medicine and Biology Society (EMBC), Osaka, 2013, pp. 4287–4290.

  21. Kosiński, J., Szklanny, K., Wieczorkowska, A., and Wichrowski, M., An analysis of game-related emotions using Emotiv EPOC, Proc. Federated Conf. on Computer Science and Information Systems (FedCSIS), Poznan, 2018, pp. 913–917.

  22. Kumar, J., Affective modelling of users in HCI using EEG, Procedia Comput. Sci., 2016, vol. 84, pp. 107–114.

    Article  Google Scholar 

  23. Kumar, N. and Kumar, J., Measurement of cognitive load in HCI systems using EEG power spectrum: an experimental study, Procedia Comput. Sci., 2016, vol. 84, pp. 70–78.

    Article  Google Scholar 

  24. Laar, B.V.D., Gürkök, H., Bos, D.P.O., Nijboer, F., and Nijholt, A., Brain–computer interfaces and user experience evaluation, in Towards Practical Brain-Computer Interfaces, Berlin, Heidelberg: Springer, 2012, pp. 223–237.

    Google Scholar 

  25. Laubheimer, P., Beyond the NPS: Measuring Perceived Usability with the SUS, NASA-TLX, and the Single Ease Question after Tasks and Usability Tests, Nielsen Norman Group, 2018.

    Google Scholar 

  26. Lewis, J.R., IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use, Int. J. Human-Comput. Interact., 1995, vol. 7, no. 1, pp. 57–78.

    Article  Google Scholar 

  27. Liu, Y., Sourina, O., and Nguyen, M.K., Real-time EEG-based human emotion recognition and visualization, Proc. IEEE Int. Conf. on Cyberworlds, Singapore, 2010, pp. 262–269.

  28. Lührs, M. and Goebel, R., Turbo-Satori: a neurofeedback and brain–computer interface toolbox for real-time functional near-infrared spectroscopy, Neurophotonics, 2017, vol. 4, no. 4, p. 041504.

  29. Massa, S.M., De Giusti, A.E., and Pesado, P.M. Métodos de evaluación de usabilidad: una propuesta de aplicación en Objetos de Aprendizaje, Proc. Workshop de Investigadores en Ciencias de la Computación, Misiones, 2012, vol. 14.

  30. Murugappan, M., Juhari, M.R.B.M., Nagarajan, R., and Yaacob, S., An investigation on visual and audiovisual stimulus based emotion recognition using EEG, Int. J. Med. Eng. Inf., 2009, vol. 1, no. 3, pp. 342–356.

    Google Scholar 

  31. Murugappan, M. and Murugappan, S., Human emotion recognition through short time Electroencephalogram (EEG) signals using Fast Fourier Transform (FFT), Proc. 9th IEEE Int. Colloquium on Signal Processing and its Applications, Kuala Lumpur, 2013, pp. 289–294.

  32. Murugappan, M., Rizon, M., Nagarajan, R., Yaacob, S., Zunaidi, I., and Hazry, D., Lifting scheme for human emotion recognition using EEG, Proc. IEEE Int. Symp. on Information Technology, Kuala Lumpur, 2008, vol. 2, pp. 1–7.

  33. Nielsen, J., Usability inspection methods, Proc. Conf. Companion on Human Factors in Computing Systems, Boston, MA, 1994, pp. 413–414.

  34. Nielsen, J., Usability Engineering, New York: AP Professional, 1993. https://doi.org/10.1016/B978-0-08-052029-2.50007-3

    Book  MATH  Google Scholar 

  35. Ortega-Gijón, Y.N. and Mezura-Godoy, C., Usability evaluation process of brain computer interfaces: an experimental study, Proc. 9th Latin American Conf. on Human Computer Interaction, Panama, 2019, pp. 1–8.

  36. Sourina, O., Liu, Y., Wang, Q., and Nguyen, M.K., EEG-based personalized digital experience, in Proc. Int. Conf. on Universal Access in Human-Computer Interaction, Berlin, Heidelberg: Springer, 2011, pp. 591–599.

  37. Parra, L.C., Spence, C.D., Gerson, A.D., and Sajda, P., Response error correction-a demonstration of improved human-machine performance using real-time EEG monitoring, IEEE Trans. Neural Syst. Rehabil. Eng., 2003, vol. 11, no. 2, pp. 173–177.

    Article  Google Scholar 

  38. Pasqualotto, E., Federici, S., Simonetta, A., and Olivetti Belardinelli, M., Usability of brain computer interfaces, in Everyday Technology for Independence and Care, IOS Press, 2011, pp. 481–488.

    Google Scholar 

  39. Pradhapan, P., Großekathöfer, U., Schiavone, G., Grundlehner, B., and Mihajlović, V., Toward practical BCI solutions for entertainment and art performance, in Brain–Computer Interfaces Handbook: Technological and Theoretical Advances, CRC Pess, 2018.

  40. Putze, F., Scherer, M., and Schultz, T., Starring into the void? Classifying internal vs. external attention from EEG, Proc. 9th Nordic Conf. on Human-Computer Interaction, Gothenburg, 2016, pp. 1–4.

  41. Puwakpitiyage, C.A., Rao, V.R., Azizi, M.S.A., Tee, W.J., Murugesan, R.K., and Hamzah, M.D., A proposed web based real time brain computer interface (BCI) system for usability testing, Int. J. Online Biomed. Eng., 2019, vol. 15, no. 8.

  42. Ramirez, R. and Vamvakousis, Z., Detecting emotion from EEG signals using the emotive epoc device, in Proc. Int. Conf. on Brain Informatics, Berlin, Heidelberg: Springer, 2012, pp. 175–184.

  43. Research-Based User Experience, W.L.: Usability Testing 101, Nielsen Norman Group, 2020. https://www.nngroup.com/articles/usability-testing-101/.

  44. Rhiu, I., Lee, Y., Choi, I., Yun, M.H., and Nam, C.S., Toward usability evaluation for brain–computer interfaces, in Brain–Computer Interfaces Handbook, CRC Pess, 2018, pp. 563–584.

    Google Scholar 

  45. Snyder, H., Literature review as a research methodology: an overview and guidelines, J. Bus. Res., 2019, vol. 104, pp. 333–339.

    Article  Google Scholar 

  46. Spencer, D., Card Sorting. Boxes and Arrows, 2004.

    Google Scholar 

  47. Stein, A., Yotam, Y., Puzis, R., Shani, G., and Taieb-Maimon, M., EEG-triggered dynamic difficulty adjustment for multiplayer games, Entertainment Comput., 2018, vol. 25, pp. 14–25.

    Article  Google Scholar 

  48. Spüler, M., A high-speed brain-computer interface (BCI) using dry EEG electrodes, PloS One, 2017, vol. 12, no. 2, p. e0172400.

  49. Taherian, S., Selitskiy, D., Pau, J., and Claire Davies, T., Are we there yet? Evaluating commercial grade brain–computer interface for control of computer applications by individuals with cerebral palsy, Disabil. Rehabil.: Assistive Technol., 2017, vol. 12, no. 2, pp. 165–174.

    Google Scholar 

  50. Theofanos, M. and Quesenbery, W., Towards the design of effective formative test reports, J. Usability Stud., 2005, vol. 1, no. 1, pp. 27–45.

    Google Scholar 

  51. Tidoni, E., Abu-Alqumsan, M., Leonardis, D., Kapeller, C., Fusco, G., Guger, C., and Aglioti, S.M., Local and remote cooperation with virtual and robotic agents: a P300 BCI study in healthy and people living with spinal cord injury, IEEE Trans. Neural Syst. Rehabil. Eng., 2016, vol. 25, no. 9, pp. 1622–1632.

    Article  Google Scholar 

  52. Valderrama, C.E. and Ulloa, G., Spectral analysis of physiological parameters for emotion detection, Proc. 17th IEEE Symp. of Image, Signal Processing, and Artificial Vision (STSIVA), Medellin, 2012, pp. 275–280.

  53. Villegas, A., Salvatierra, E., Gubyk, A., Lugo, E., and Pacheco, J., Reconocimiento de patrones de actividad cerebral asociados a tareas mentales mediante RNA para una interfaz cerebro computador, Rev. Ing. UC, 2008, vol. 15, no. 1, pp. 88–92.

    Google Scholar 

  54. Wang, Q., Sourina, O., and Nguyen, M.K., Eeg-based “serious” games design for medical applications, Proc. IEEE Int. Conf. on Cyberworlds, Washington, 2010, pp. 270–276.

  55. Xing, X., Wang, Y., Pei, W., Guo, X., and Liu, Z., A high-speed SSVEP-based BCI using dry EEG electrodes, Sci. Rep., 2018, vol. 8, no. 1, pp. 1–10.

    Article  Google Scholar 

  56. Zhang, J., Chen, M., Hu, S., Cao, Y., and Kozma, R., PNN for EEG-based emotion recognition, Proc. IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC), Washington, 2016, pp. 002319–002323.

  57. Kurniawan, S., Interaction design: beyond human–computer interaction by Preece, Sharp and Rogers (2001), ISBN 0471492787, Univ. Access. Inf. Soc., 2004, vol. 3, p. 289. https://doi.org/10.1007/s10209-004-0102-1

    Article  Google Scholar 

  58. Kitchenham, B., Brereton, O.P., Budgen, D., Turner, M., Bailey, J., and Linkman, S., Systematic literature reviews in software engineering–a systematic literature review, Inf. Software Technol., 2009, vol. 51, no. 1, pp. 7–15.

    Article  Google Scholar 

  59. Tello-Rodríguez, M., Ocharán-Hernández, J.O., Pérez-Arriaga, J.C., Limón, X., and Sánchez-García, Á.J., A design guide for usable web APIs, Program. Comput. Software, 2020, vol. 46, no. 8, pp. 584–593.

    Article  Google Scholar 

  60. Lukin, V.N., Dzyubenko, A.L., and Chechikov, Y.B. Approaches to user interface development, Program. Comput. Software, 2020, vol. 46, no. 5, pp. 316–323.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Facultad de Estadística e Informática, Universidad Veracruzana, Av. Xalapa, Obrero Campesino, 91020, Xalapa-Enríquez, Ver., México

    Y. N. Ortega & C. Mezura-Godoy

Authors
  1. Y. N. Ortega
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. C. Mezura-Godoy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Y. N. Ortega or C. Mezura-Godoy.

Ethics declarations

The authors declare that they have no conflicts of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ortega, Y.N., Mezura-Godoy, C. Usability Evaluation of BCI Software Applications: A systematic review of the literature. Program Comput Soft 48, 646–657 (2022). https://doi.org/10.1134/S0361768822080163

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0361768822080163

Search

Navigation