Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment

Abstract

The purpose of this work is to set up a model that can estimate the mental fatigue of users based on the fusion of relevant features extracted from Positive 300 (P300) and steady state visual evoked potentials (SSVEP) measured by electroencephalogram. To this end, an experimental protocol describes the induction of P300, SSVEP and mental workload (which leads to mental fatigue by varying time-on-task) in different scenarios where environmental artifacts are controlled (obstacles number, obstacles velocities, ambient luminosity). Ten subjects took part in the experiment (with two suffering from cerebral palsy). Their mission is to navigate along a corridor from a starting point A to a goal point B where specific flickering stimuli are introduced to perform the P300 task. On the other hand, SSVEP task is elicited thanks to 10 Hz flickering lights. Correlated features are considered as inputs to fusion block which estimates mental workload. In order to deal with uncertainties and heterogeneity of P300 and SSVEP features, Dempster–Shafer (D–S) evidential reasoning is introduced. As the goal is to assess the reliability for the estimation of mental fatigue levels, D–S is compared to multi layer perception and linear discriminant analysis. The results show that D–S globally outperforms the other classifiers (although its performance significantly decreases between healthy and palsied groups). Finally we discuss the feasibility of such a fusion proposal in real life situation.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Notes

  1. 1.

    https://www.emotiv.com.

References

  1. Banerjee S, Khajanchi S, Chaudhuri S (2015) A mathematical model to elucidate brain tumor abrogation by immunotherapy with T11 target structure. PLoS ONE 40:e123611

    Google Scholar 

  2. Chew LH, Teo J, Mountstephens J (2015) Aesthetic preference recognition of 3D shapes using EEG. Cognit Neurodyn 10:165–173

    Article  Google Scholar 

  3. Cortese S, Ferrin M, Brandeis D, Buitelaar J, Daley D, Dittmann RW, Holtmann M, Santosh P, Stevenson J, Stringaris A, Zuddas A, Sonuga-Barke EJS (2015) Cognitive training for attention-deficit/hyperactivity disorder: meta-analysis of clinical and neuropsychological outcomes from randomized controlled trials. J Am Acad Child Adolesc Psychiatry 54(3):164–174

    Article  PubMed  PubMed Central  Google Scholar 

  4. Crippa A, Maurits NM, Lorist MM, Roerdink JBTM (2011) Visual computing in biology and medicine: graph averaging as a means to compare multichannel EEG coherence networks and its application to the study of mental fatigue and neurodegenerative disease. Comput Graph 35(2):265–274. https://doi.org/10.1016/j.cag.2010.12.008

    Article  Google Scholar 

  5. Crowley K, Sliney A, Pitt I, Murphy D (2010) Evaluating a brain–computer interface to categorise human emotional response. In: Proceedings of the 2010 10th IEEE International Conference on Advanced Learning Technologies, ICALT ’10. IEEE Computer Society, Washington, pp 276–278

  6. Csatho A, van der Linden D, Darnai G, Hopstaken JF (2013) The same-object benefit is influenced by time-on-task. J Cognit Psychol 25(3):319–327. https://doi.org/10.1080/20445911.2012.753875

    Article  Google Scholar 

  7. da Silva FL, Niedermeyer E (2004) lectroencphalographie: Principes de base, les applications cliniques, et les domaines connexes. Lippincott Williams and Wilkins. ISBN 0-7817-5126-8, 9780781751261

  8. Dempster AP (2008) A generalization of Bayesian inference. In: Yager RR, Liu L (eds) Classic works of the Dempster–Shafer theory of belief functions, volume 219 of studies in fuzziness and soft computing. Springer, Berlin, pp 73–104

  9. Dey AK, Mann DD (2010) A complete task analysis to measure the workload associated with operating an agricultural sprayer equipped with a navigation device. Appl Ergon 41(1):146–149

    Article  CAS  PubMed  Google Scholar 

  10. Dornhege G, Blankertz B, Curio G, Müller K-R (2003) Increase information transfer rates in BCI by CSP extension to multi-class. In: NIPS

  11. Friman O, Luth T, Volosyak I, Graser A (2007a) Spelling with steady-state visual evoked potentials. In: 3rd International IEEE/EMBS Conference on Neural Engineering, 2007. CNE ’07, pp 354–357

  12. Friman O, Volosyak I, Graser A (2007b) Multiple channel detection of steady-state visual evoked potentials for brain–computer interfaces. IEEE Trans Biomed Eng 54(4):742–750

    Article  PubMed  Google Scholar 

  13. Gao X, Xu D, Cheng M, Gao S (2003) A BCI-based environmental controller for the motion-disabled. IEEE Trans Neural Syst Rehabil Eng 11(2):137–140. ISSN 1534-4320. https://doi.org/10.1109/TNSRE.2003.814449

  14. Ghosh D, Khajanchi S, Mangiarotti S, Denis F, Dana SK, Letellier C (2017) How tumor growth can be influenced by delayed interactions between cancer cells and the microenvironment? Biosystems 158:17–30. ISSN 0303-2647

  15. Girelli M, Luck S (1997) Are the same attentional mechanisms used to detect visual search targets defined by color, orientation, and motion? J Cognit Neurosci 9:238–53

    Article  CAS  Google Scholar 

  16. Grandjean EP, Wotzka G, Schaad R, Gilgen A (1971) Fatigue and stress in air traffic controllers. Ergonomics 14(1):159–165

    Article  CAS  PubMed  Google Scholar 

  17. Haapalainen E, Kim SJ, Forlizzi JF, Dey AK (2010) Psycho-physiological measures for assessing cognitive load. In: Proceedings of the 12th ACM International Conference on Ubiquitous Computing, Ubicomp ’10. ACM, New York, pp 301–310

  18. Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload, volume 52 of advances in psychology. North-Holland, Amsterdam, pp 139–183

    Google Scholar 

  19. Ho Hwang J, Won Nam K, Jang DP, Kim I (2017) Effects of spectral smearin of stimuli on the performance of auditory steady-state response-based brain–computer interface. Cognit Neurodyn 11:515–527 07

    Article  Google Scholar 

  20. Huang M, Jin J, Zhang Y, Hu D, Wang X (2017) Usage of drip drops as stimuli in an auditory P300 BCI paradigm. Cognit Neurodyn 12:85–94. https://doi.org/10.1007/s11571-017-9456-y

    Article  Google Scholar 

  21. Jo S, Myung R, Yoon D (2012) Quantitative prediction of mental workload with the ACT-R cognitive architecture. Int J Ind Ergon 42(4):359–370

    Article  Google Scholar 

  22. Khajanchi S (2017) Uniform persistence and global stability for a brain tumor and immune system interaction. Biophys Rev Lett 12:187–208

    Article  CAS  Google Scholar 

  23. Khajanchi S, Das DK, Kar TK (2018) Dynamics of tuberculosis transmission with exogenous reinfections and endogenous reactivation. Phys A Stat Mech Appl 497:52–71. ISSN 0378-4371

  24. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Article  Google Scholar 

  25. Kota S, Miho O, Torres CCC, Sato M, Miki N (2016) Mental fatigue monitoring using a wearable transparent eye detection system. Micromachines 7(2):123

    Google Scholar 

  26. Kthner I, Wriessnegger SC, Mller-Putz GR, Kbler A, Halder S (2014) Effects of mental workload and fatigue on the p300, alpha and theta band power during operation of an erp (p300) brain–computer interface. Biol Psychol 102:118–129

    Article  Google Scholar 

  27. Lalor EC, Kelly SP, Finucane C, Burke R, Smith R, Reilly RB, McDarby G (2005) Steady-state VEP-based brain–computer interface control in an immersive 3D gaming environment. EURASIP J Adv Signal Process 2005(19):3156–3164

    Article  Google Scholar 

  28. Lamti HA, Ben Khelifa MM, Alimi AM, Gorce P (2014a) Effect of fatigue on ssvep during virtual wheelchair navigation. J Theor Appl Inf Technol 65:1–10

    Google Scholar 

  29. Lamti HA, Khelifa MMB, Alimi AM, Gorce P (2014b) Influence of mental fatigue on P300 and SSVEP during virtual wheelchair navigation. In: 2014 36th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC). IEEE, pp 1255–1258

  30. Lamti HA, Gorce P, Khelifa MMB, Alimi AM (2016) When mental fatigue maybe characterized by event related potential (p300) during virtual wheelchair navigation. Comput Methods Biomech Biomed Eng 19(16):1749–1759

    Article  Google Scholar 

  31. Lamti HA, Khelifa MMB, Hugel V (2018) Cerebral and gaze data fusion for wheelchair navigation enhancement: case of distracted users. Robotica. https://doi.org/10.1017/S0263574718000991

  32. Lazar NA, Luna B, Sweeney JA, Eddy WF (2002) Combining brains: a survey of methods for statistical pooling of information. NeuroImage 16(2):538–550

    Article  PubMed  Google Scholar 

  33. Li X-O, Zhang X-W, Feng H (2003) Real time extraction of visual evoked potentials. In: International conference on neural networks and signal processing, 2003. Proceedings of the 2003, vol 2, pp 1342–1345

  34. Liberati G, Federici S, Pasqualotto E (2015) Extracting neurophysiological signals reflecting users’ emotional and affective responses to bci use: a systematic literature review. NeuroRehabilitation 37:10

    Article  Google Scholar 

  35. Luck SJ (2005) An introduction to the event-related potential technique (cognitive neuroscience). A Bradford Book, 1 edn

  36. Mandel C, Luth T, Laue T, Rofer T, Graser A, Krieg-Bruckner B (2009) Navigating a smart wheelchair with a brain–computer interface interpreting steady-state visual evoked potentials. In: IEEE/RSJ international conference on intelligent robots and systems, 2009. IROS 2009, pp 1118–1125

  37. Meng L-F, Lu C-P, Chen B-W, Chen C-H (2006) Fatigue-induced reversed hemispheric plasticity during motor repetitions: a brain electrophysiological study. In: Proceedings of the 13 international conference on neural information processing—volume part I, ICONIP’06. Springer, Berlin, pp 65–71. ISBN 3-540-46479-4, 978-3-540-46479-2

  38. Molina GG, Tsoneva T, Nijholt A (2009) Emotional brain–computer interfaces. In: 3rd international conference on affective computing and intelligent interaction and workshops, 2009. ACII 2009, pp 87–108

  39. Murata A, Uetake A, Takasawa Y (2005) Evaluation of mental fatigue using feature parameter extracted from event-related potential. Int J Ind Ergon 35(8):761–770

    Article  Google Scholar 

  40. Myers LJ, Hunter DJ (2009) Parsimonious identification of physiological indices for monitoring cognitive fatigue. In: Proceedings of the 5th international conference on foundations of augmented cognition. Neuroergonomics and Operational Neuroscience: Held As Part of HCI International 2009, FAC ’09. Springer, Berlin, pp 495–503. ISBN 978-3-642-02811-3

  41. O’Donnell RD, Eggemeier FT (1986) Workload assessment methodology. Cognit Process Perform 2:1–49

    Google Scholar 

  42. Perez JLM, Cruz AB (2007) Linear discriminant analysis on brain computer interface. In: IEEE International Symposium on Intelligent Signal Processing, 2007. WISP 2007, pp 1–6

  43. Perlstein WM, Cole MA, Larson M, Kelly K, Seignourel P, Keil A (2003) Steady-state visual evoked potentials reveal frontally-mediated working memory activity in humans. Neurosci Lett 342(3):191–195

    Article  CAS  PubMed  Google Scholar 

  44. Pimenta A, Carneiro D, Novais P, Neves J (2013) Monitoring mental fatigue through the analysis of keyboard and mouse interaction patterns. In: Pan J-S, Polycarpou MM, Woźniak M, de Carvalho ACPLF, Quintián H, Corchado E (eds) Hybrid artificial intelligent systems. Springer, Berlin, pp 222–231

    Google Scholar 

  45. Puanhvuan D, Khemmachotikun S, Wechakarn P, Wijarn B, Wongsawat Y (2017) Navigation-synchronized multimodal control wheelchair from brain to alternative assistive technologies for persons with severe disabilities. Cognit Neurodyn 11:117–134

    Article  Google Scholar 

  46. Queteschiner D (2012) Reality factory. http://www.realityfactory.info/cms/tutorials.html

  47. Regan D (1966) Some characteristics of average steady-state and transient responses evoked by modulated light. Electroencephalogr Clin Neurophysiol 20(3):238–248

    Article  CAS  PubMed  Google Scholar 

  48. Roy RN, Bonnet S, Charbonnier S, Campagne A (2013) Mental fatigue and working memory load estimation: interaction and implications for EEG-based passive BCI. In: 2013 35th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 6607–6610

  49. Ryu K, Myung R (2005) Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. Int J Ind Ergon 35(11):991–1009

    Article  Google Scholar 

  50. Schlesinger M, Amso D, Johnson SP (2007) The neural basis for visual selective attention in young infants: a computational account. Adapt Behav 15(2):135–148

    Article  Google Scholar 

  51. Valbuena D, Cyriacks M, Friman O, Volosyak I, Graser A (2007) Brain–computer interface for high-level control of rehabilitation robotic systems. In: IEEE 10th international conference on rehabilitation robotics, 2007. ICORR 2007, pp 619–625

  52. Vialatte F-B, Maurice M, Dauwels J, Cichocki A (2010) Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol 90(4):418–438

    Article  PubMed  Google Scholar 

  53. Wang Y, Wang R, Gao X, Hong B, Gao S (2006) A practical VEP-based brain–computer interface. IEEE Trans Neural Syst Rehabil Eng 14(2):234–240

    Article  CAS  PubMed  Google Scholar 

  54. Wu Z, Lai Y, Xia Y, Wu D, Yao D (2008) Stimulator selection in SSVEP-based BCI. Med Eng Phys 30(8):1079–1088. Special Issue (part): Bioengineering in Taiwan

  55. Zhang Y, Guo D, Peng X, Zhang Y, Yao D (2016) Robust frequency recognition for ssvep-based bci with temporally local multivariate synchronization index. Cognit Neurodyn 10(6):505–511

    Article  Google Scholar 

  56. Zhang C, Zheng C, Pei X, Yu X (2007) Estimating mental fatigue based on multichannel linear descriptors analysis. In: Proceedings of the 1st international conference on digital human modeling, ICDHM’07. Springer, Berlin, pp 521–529. ISBN 978-3-540-73318-8. http://dl.acm.org/citation.cfm?id=1784074.1784136

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Hachem A. Lamti.

Ethics declarations

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lamti, H.A., Ben Khelifa, M.M. & Hugel, V. Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment. Cogn Neurodyn 13, 271–285 (2019). https://doi.org/10.1007/s11571-019-09523-2

Download citation

Keywords

  • BCI
  • Mental fatigue
  • Evidential reasoning
  • P300
  • SSVEP