Skip to main content

Advertisement

SpringerLink
Log in

Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment

  • Research Article
  • Published:
Cognitive Neurodynamics Aims and scope Submit manuscript

Abstract

The purpose of this work is to set up a model that can estimate the mental fatigue of users based on the fusion of relevant features extracted from Positive 300 (P300) and steady state visual evoked potentials (SSVEP) measured by electroencephalogram. To this end, an experimental protocol describes the induction of P300, SSVEP and mental workload (which leads to mental fatigue by varying time-on-task) in different scenarios where environmental artifacts are controlled (obstacles number, obstacles velocities, ambient luminosity). Ten subjects took part in the experiment (with two suffering from cerebral palsy). Their mission is to navigate along a corridor from a starting point A to a goal point B where specific flickering stimuli are introduced to perform the P300 task. On the other hand, SSVEP task is elicited thanks to 10 Hz flickering lights. Correlated features are considered as inputs to fusion block which estimates mental workload. In order to deal with uncertainties and heterogeneity of P300 and SSVEP features, Dempster–Shafer (D–S) evidential reasoning is introduced. As the goal is to assess the reliability for the estimation of mental fatigue levels, D–S is compared to multi layer perception and linear discriminant analysis. The results show that D–S globally outperforms the other classifiers (although its performance significantly decreases between healthy and palsied groups). Finally we discuss the feasibility of such a fusion proposal in real life situation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://www.emotiv.com.

References

  • Banerjee S, Khajanchi S, Chaudhuri S (2015) A mathematical model to elucidate brain tumor abrogation by immunotherapy with T11 target structure. PLoS ONE 40:e123611

    Google Scholar 

  • Chew LH, Teo J, Mountstephens J (2015) Aesthetic preference recognition of 3D shapes using EEG. Cognit Neurodyn 10:165–173

    Article  Google Scholar 

  • Cortese S, Ferrin M, Brandeis D, Buitelaar J, Daley D, Dittmann RW, Holtmann M, Santosh P, Stevenson J, Stringaris A, Zuddas A, Sonuga-Barke EJS (2015) Cognitive training for attention-deficit/hyperactivity disorder: meta-analysis of clinical and neuropsychological outcomes from randomized controlled trials. J Am Acad Child Adolesc Psychiatry 54(3):164–174

    Article  PubMed  PubMed Central  Google Scholar 

  • Crippa A, Maurits NM, Lorist MM, Roerdink JBTM (2011) Visual computing in biology and medicine: graph averaging as a means to compare multichannel EEG coherence networks and its application to the study of mental fatigue and neurodegenerative disease. Comput Graph 35(2):265–274. https://doi.org/10.1016/j.cag.2010.12.008

    Article  Google Scholar 

  • Crowley K, Sliney A, Pitt I, Murphy D (2010) Evaluating a brain–computer interface to categorise human emotional response. In: Proceedings of the 2010 10th IEEE International Conference on Advanced Learning Technologies, ICALT ’10. IEEE Computer Society, Washington, pp 276–278

  • Csatho A, van der Linden D, Darnai G, Hopstaken JF (2013) The same-object benefit is influenced by time-on-task. J Cognit Psychol 25(3):319–327. https://doi.org/10.1080/20445911.2012.753875

    Article  Google Scholar 

  • da Silva FL, Niedermeyer E (2004) lectroencphalographie: Principes de base, les applications cliniques, et les domaines connexes. Lippincott Williams and Wilkins. ISBN 0-7817-5126-8, 9780781751261

  • Dempster AP (2008) A generalization of Bayesian inference. In: Yager RR, Liu L (eds) Classic works of the Dempster–Shafer theory of belief functions, volume 219 of studies in fuzziness and soft computing. Springer, Berlin, pp 73–104

  • Dey AK, Mann DD (2010) A complete task analysis to measure the workload associated with operating an agricultural sprayer equipped with a navigation device. Appl Ergon 41(1):146–149

    Article  CAS  PubMed  Google Scholar 

  • Dornhege G, Blankertz B, Curio G, Müller K-R (2003) Increase information transfer rates in BCI by CSP extension to multi-class. In: NIPS

  • Friman O, Luth T, Volosyak I, Graser A (2007a) Spelling with steady-state visual evoked potentials. In: 3rd International IEEE/EMBS Conference on Neural Engineering, 2007. CNE ’07, pp 354–357

  • Friman O, Volosyak I, Graser A (2007b) Multiple channel detection of steady-state visual evoked potentials for brain–computer interfaces. IEEE Trans Biomed Eng 54(4):742–750

    Article  PubMed  Google Scholar 

  • Gao X, Xu D, Cheng M, Gao S (2003) A BCI-based environmental controller for the motion-disabled. IEEE Trans Neural Syst Rehabil Eng 11(2):137–140. ISSN 1534-4320. https://doi.org/10.1109/TNSRE.2003.814449

  • Ghosh D, Khajanchi S, Mangiarotti S, Denis F, Dana SK, Letellier C (2017) How tumor growth can be influenced by delayed interactions between cancer cells and the microenvironment? Biosystems 158:17–30. ISSN 0303-2647

  • Girelli M, Luck S (1997) Are the same attentional mechanisms used to detect visual search targets defined by color, orientation, and motion? J Cognit Neurosci 9:238–53

    Article  CAS  Google Scholar 

  • Grandjean EP, Wotzka G, Schaad R, Gilgen A (1971) Fatigue and stress in air traffic controllers. Ergonomics 14(1):159–165

    Article  CAS  PubMed  Google Scholar 

  • Haapalainen E, Kim SJ, Forlizzi JF, Dey AK (2010) Psycho-physiological measures for assessing cognitive load. In: Proceedings of the 12th ACM International Conference on Ubiquitous Computing, Ubicomp ’10. ACM, New York, pp 301–310

  • Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload, volume 52 of advances in psychology. North-Holland, Amsterdam, pp 139–183

    Chapter  Google Scholar 

  • Ho Hwang J, Won Nam K, Jang DP, Kim I (2017) Effects of spectral smearin of stimuli on the performance of auditory steady-state response-based brain–computer interface. Cognit Neurodyn 11:515–527 07

    Article  Google Scholar 

  • Huang M, Jin J, Zhang Y, Hu D, Wang X (2017) Usage of drip drops as stimuli in an auditory P300 BCI paradigm. Cognit Neurodyn 12:85–94. https://doi.org/10.1007/s11571-017-9456-y

    Article  Google Scholar 

  • Jo S, Myung R, Yoon D (2012) Quantitative prediction of mental workload with the ACT-R cognitive architecture. Int J Ind Ergon 42(4):359–370

    Article  Google Scholar 

  • Khajanchi S (2017) Uniform persistence and global stability for a brain tumor and immune system interaction. Biophys Rev Lett 12:187–208

    Article  CAS  Google Scholar 

  • Khajanchi S, Das DK, Kar TK (2018) Dynamics of tuberculosis transmission with exogenous reinfections and endogenous reactivation. Phys A Stat Mech Appl 497:52–71. ISSN 0378-4371

  • Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Article  Google Scholar 

  • Kota S, Miho O, Torres CCC, Sato M, Miki N (2016) Mental fatigue monitoring using a wearable transparent eye detection system. Micromachines 7(2):123

    Google Scholar 

  • Kthner I, Wriessnegger SC, Mller-Putz GR, Kbler A, Halder S (2014) Effects of mental workload and fatigue on the p300, alpha and theta band power during operation of an erp (p300) brain–computer interface. Biol Psychol 102:118–129

    Article  Google Scholar 

  • Lalor EC, Kelly SP, Finucane C, Burke R, Smith R, Reilly RB, McDarby G (2005) Steady-state VEP-based brain–computer interface control in an immersive 3D gaming environment. EURASIP J Adv Signal Process 2005(19):3156–3164

    Article  Google Scholar 

  • Lamti HA, Ben Khelifa MM, Alimi AM, Gorce P (2014a) Effect of fatigue on ssvep during virtual wheelchair navigation. J Theor Appl Inf Technol 65:1–10

    Google Scholar 

  • Lamti HA, Khelifa MMB, Alimi AM, Gorce P (2014b) Influence of mental fatigue on P300 and SSVEP during virtual wheelchair navigation. In: 2014 36th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC). IEEE, pp 1255–1258

  • Lamti HA, Gorce P, Khelifa MMB, Alimi AM (2016) When mental fatigue maybe characterized by event related potential (p300) during virtual wheelchair navigation. Comput Methods Biomech Biomed Eng 19(16):1749–1759

    Article  Google Scholar 

  • Lamti HA, Khelifa MMB, Hugel V (2018) Cerebral and gaze data fusion for wheelchair navigation enhancement: case of distracted users. Robotica. https://doi.org/10.1017/S0263574718000991

  • Lazar NA, Luna B, Sweeney JA, Eddy WF (2002) Combining brains: a survey of methods for statistical pooling of information. NeuroImage 16(2):538–550

    Article  PubMed  Google Scholar 

  • Li X-O, Zhang X-W, Feng H (2003) Real time extraction of visual evoked potentials. In: International conference on neural networks and signal processing, 2003. Proceedings of the 2003, vol 2, pp 1342–1345

  • Liberati G, Federici S, Pasqualotto E (2015) Extracting neurophysiological signals reflecting users’ emotional and affective responses to bci use: a systematic literature review. NeuroRehabilitation 37:10

    Article  Google Scholar 

  • Luck SJ (2005) An introduction to the event-related potential technique (cognitive neuroscience). A Bradford Book, 1 edn

  • Mandel C, Luth T, Laue T, Rofer T, Graser A, Krieg-Bruckner B (2009) Navigating a smart wheelchair with a brain–computer interface interpreting steady-state visual evoked potentials. In: IEEE/RSJ international conference on intelligent robots and systems, 2009. IROS 2009, pp 1118–1125

  • Meng L-F, Lu C-P, Chen B-W, Chen C-H (2006) Fatigue-induced reversed hemispheric plasticity during motor repetitions: a brain electrophysiological study. In: Proceedings of the 13 international conference on neural information processing—volume part I, ICONIP’06. Springer, Berlin, pp 65–71. ISBN 3-540-46479-4, 978-3-540-46479-2

  • Molina GG, Tsoneva T, Nijholt A (2009) Emotional brain–computer interfaces. In: 3rd international conference on affective computing and intelligent interaction and workshops, 2009. ACII 2009, pp 87–108

  • Murata A, Uetake A, Takasawa Y (2005) Evaluation of mental fatigue using feature parameter extracted from event-related potential. Int J Ind Ergon 35(8):761–770

    Article  Google Scholar 

  • Myers LJ, Hunter DJ (2009) Parsimonious identification of physiological indices for monitoring cognitive fatigue. In: Proceedings of the 5th international conference on foundations of augmented cognition. Neuroergonomics and Operational Neuroscience: Held As Part of HCI International 2009, FAC ’09. Springer, Berlin, pp 495–503. ISBN 978-3-642-02811-3

  • O’Donnell RD, Eggemeier FT (1986) Workload assessment methodology. Cognit Process Perform 2:1–49

    Google Scholar 

  • Perez JLM, Cruz AB (2007) Linear discriminant analysis on brain computer interface. In: IEEE International Symposium on Intelligent Signal Processing, 2007. WISP 2007, pp 1–6

  • Perlstein WM, Cole MA, Larson M, Kelly K, Seignourel P, Keil A (2003) Steady-state visual evoked potentials reveal frontally-mediated working memory activity in humans. Neurosci Lett 342(3):191–195

    Article  CAS  PubMed  Google Scholar 

  • Pimenta A, Carneiro D, Novais P, Neves J (2013) Monitoring mental fatigue through the analysis of keyboard and mouse interaction patterns. In: Pan J-S, Polycarpou MM, Woźniak M, de Carvalho ACPLF, Quintián H, Corchado E (eds) Hybrid artificial intelligent systems. Springer, Berlin, pp 222–231

    Chapter  Google Scholar 

  • Puanhvuan D, Khemmachotikun S, Wechakarn P, Wijarn B, Wongsawat Y (2017) Navigation-synchronized multimodal control wheelchair from brain to alternative assistive technologies for persons with severe disabilities. Cognit Neurodyn 11:117–134

    Article  Google Scholar 

  • Queteschiner D (2012) Reality factory. http://www.realityfactory.info/cms/tutorials.html

  • Regan D (1966) Some characteristics of average steady-state and transient responses evoked by modulated light. Electroencephalogr Clin Neurophysiol 20(3):238–248

    Article  CAS  PubMed  Google Scholar 

  • Roy RN, Bonnet S, Charbonnier S, Campagne A (2013) Mental fatigue and working memory load estimation: interaction and implications for EEG-based passive BCI. In: 2013 35th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 6607–6610

  • Ryu K, Myung R (2005) Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. Int J Ind Ergon 35(11):991–1009

    Article  Google Scholar 

  • Schlesinger M, Amso D, Johnson SP (2007) The neural basis for visual selective attention in young infants: a computational account. Adapt Behav 15(2):135–148

    Article  Google Scholar 

  • Valbuena D, Cyriacks M, Friman O, Volosyak I, Graser A (2007) Brain–computer interface for high-level control of rehabilitation robotic systems. In: IEEE 10th international conference on rehabilitation robotics, 2007. ICORR 2007, pp 619–625

  • Vialatte F-B, Maurice M, Dauwels J, Cichocki A (2010) Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol 90(4):418–438

    Article  PubMed  Google Scholar 

  • Wang Y, Wang R, Gao X, Hong B, Gao S (2006) A practical VEP-based brain–computer interface. IEEE Trans Neural Syst Rehabil Eng 14(2):234–240

    Article  CAS  PubMed  Google Scholar 

  • Wu Z, Lai Y, Xia Y, Wu D, Yao D (2008) Stimulator selection in SSVEP-based BCI. Med Eng Phys 30(8):1079–1088. Special Issue (part): Bioengineering in Taiwan

  • Zhang Y, Guo D, Peng X, Zhang Y, Yao D (2016) Robust frequency recognition for ssvep-based bci with temporally local multivariate synchronization index. Cognit Neurodyn 10(6):505–511

    Article  Google Scholar 

  • Zhang C, Zheng C, Pei X, Yu X (2007) Estimating mental fatigue based on multichannel linear descriptors analysis. In: Proceedings of the 1st international conference on digital human modeling, ICDHM’07. Springer, Berlin, pp 521–529. ISBN 978-3-540-73318-8. http://dl.acm.org/citation.cfm?id=1784074.1784136

Download references

Author information

Authors and Affiliations

  1. COnception de Systemes Mecaniques et Robotiques (COSMER) Laboratory, University of Toulon, Toulon, France

    Hachem A. Lamti & Vincent Hugel

  2. Impact de l’Activite Physique sur la Sante (IAPS) Laboratory, University of Toulon, Toulon, France

    Mohamed Moncef Ben Khelifa

Authors
  1. Hachem A. Lamti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Moncef Ben Khelifa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vincent Hugel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hachem A. Lamti.

Ethics declarations

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lamti, H.A., Ben Khelifa, M.M. & Hugel, V. Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment. Cogn Neurodyn 13, 271–285 (2019). https://doi.org/10.1007/s11571-019-09523-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11571-019-09523-2

Keywords

Search

Navigation