Skip to main content
SpringerLink
Log in

Fusion of forehead EEG with machine vision for real-time fatigue detection in an automatic processing pipeline

  • S.I.: AI based Techniques and Applications for Intelligent IoT Systems
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Driving fatigue is a leading contributor to traffic accidents and fatalities. For automatic detection of fatigue, multimodal data fusion is a potential key technique, especially the merging of electroencephalogram (EEG) and eye movement. Although EEG can produce objective measures of fatigue with very high temporal resolution, an unfriendly multichannel system limits its practical application while portable wearables and machine vision receive much attention since they are pervasive and user friendly. Hence, this study aims to construct a novel pipeline by using machine vision technique to improve the quality of driver fatigue detection based on forehead EEG. By coupling the Karolinska Sleepiness Scale (KSS) and percentage of eyelid closure (PERCLOS), the precise and reliable dataset for reflecting drivers’ fatigue levels were obtained. Moreover, major artifact contamination related to blink activity for frontal-channel EEG was removed by a synchronous video-based eyeblink event marker. In addition, the scale-invariant feature transform (SIFT) features of eyelid keypoints was applied to fuse with the EEG-driven features. Sixteen subjects participated in a realistic driving simulation experiment. Both machine and deep learning methods were used to implement the intra-subject and inter-subject cross validation. It demonstrated that our proposed method can achieve a significant performance. The present work showed the usefulness of forehead EEG and eye images that was originally merged for detecting fatigue. It provides a new strategy of using forehead EEG combined with machine vision to design a potential and automatic pipeline for driver fatigue detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Klauer SG, Dingus TA, Neale VL, Sudweeks JD, Ramsey DJ (2006) The impact of driver inattention on near-crash/crash risk: an analysis using the 100-car naturalistic driving study data. Department of Transportation, NHTSA, Washington

  2. Connor J, Norton R, Ameratunga S, Robinson E, Civil I, Dunn R et al (2002) Driver sleepiness and risk of serious injury to car occupants: population based case control study. BMJ 1125

  3. Sikander G, Anwar S (2018) Driver fatigue detection systems: a review. IEEE T Intell Transp 20:2339–2352

    Article  Google Scholar 

  4. Urigüen JA, Garcia-Zapirain B (2015) EEG artifact removal—state-of-the-art and guidelines. J Neural Eng 031001

  5. Ko L-W, Komarov O, Lai W-K, Liang W-G, Jung T-P (2020) Eyeblink recognition improves fatigue prediction from single-channel forehead EEG in a realistic sustained attention task. J Neural Eng 036015

  6. Khatun S, Mahajan R, Morshed BI (2016) Comparative study of wavelet-based unsupervised ocular artifact removal techniques for single-channel EEG data. IEEE J Transl Eng He 4:1–8

    Google Scholar 

  7. Luo H, Qiu T, Liu C, Huang P (2019) Research on fatigue driving detection using forehead EEG based on adaptive multi-scale entropy. Biomed Signal Proces 51:50–58

    Article  Google Scholar 

  8. Zhang X, Wang X, Yang X, Xu C, Zhu X, Wei J (2020) Driver drowsiness detection using mixed-effect ordered logit model considering time cumulative effect. Anal Methods Accid R 100114

  9. Mandal B, Li L, Wang GS, Lin J (2016) Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE T Intell Transp 18:545–557

    Article  Google Scholar 

  10. Ma Y, Fu R (2015) Research and development of drivers visual behavior and driving safety. China J Highway Transp 28:82–94

    Google Scholar 

  11. Ramzan M, Khan HU, Awan SM, Ismail A, Ilyas M, Mahmood A (2019) A survey on state-of-the-art drowsiness detection techniques. IEEE Access 7:61904–61919

    Article  Google Scholar 

  12. Bakker B, Zabłocki B, Baker A, Riethmeister V, Marx B, Iyer G et al (2021) A multi-stage, multi-feature machine learning approach to detect driver sleepiness in naturalistic road driving conditions. IEEE T Intell Transp 99:1–10

    Google Scholar 

  13. Quddus A, Zandi AS, Prest L, Comeau FJ (2021) Using long short term memory and convolutional neural networks for driver drowsiness detection. Accident Anal Prev 106107

  14. Kim K-J, Lim K-T, Woon Baek J, Shin M (2021) Low-cost real-time driver drowsiness detection based on convergence of IR images and EEG signals. In: 2021 international conference on artificial intelligence in information and communication (ICAIIC), pp 438–443

  15. Dkhil MB, Wali A, Alimi AM (2018) A hierarchical fuzzy system for an advanced driving assistance system. arXiv preprint arXiv:1806.04611

  16. Cao L, Li J, Xu Y, Zhu H, Jiang C (2016) A hybrid vigilance monitoring study for mental fatigue and its neural activities. Cogn Comput 8:228–236

    Article  Google Scholar 

  17. Zheng W-L, Lu B-L (2017) A multimodal approach to estimating vigilance using EEG and forehead EOG. J Neural Eng 026017

  18. Gou C, Wu Y, Wang K, Wang K, Wang F-Y, Ji Q (2017) A joint cascaded framework for simultaneous eye detection and eye state estimation. Pattern Recogn 67:23–31

    Article  Google Scholar 

  19. Wu Z, Huang NE (2009) Ensemble empirical mode decomposition: a noise-assisted data analysis method. Adv Adap Data Anal 1:1–41

    Article  Google Scholar 

  20. Chen X, Chen Q, Zhang Y, Wang ZJ (2018) A novel EEMD-CCA approach to removing muscle artifacts for pervasive EEG. IEEE Sens J 19:8420–8431

    Article  Google Scholar 

  21. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60:91–110

    Article  Google Scholar 

  22. Zhang Q, Chen Y, Zhang Y, Xu Y (2008) SIFT implementation and optimization for multi-core systems. In: 2008 IEEE international symposium on parallel and distributed processing, pp 1–8

  23. Staudemeyer RC, Morris ER (2019) Understanding LSTM—a tutorial into long short-term memory recurrent neural networks. arXiv preprint arXiv:1909.09586

  24. Debie E, Rojas RF, Fidock J, Barlow M, Kasmarik K, Anavatti S et al (2019) Multimodal fusion for objective assessment of cognitive workload: a review. IEEE T Cybern 51(3):1542–1555

    Article  Google Scholar 

  25. Chen X, Xu X, Liu A, Lee S, Chen X, Zhang X et al (2019) Removal of muscle artifacts from the EEG: a review and recommendations. IEEE Sens J 19:5353–5368

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Research and Development Program of China (2020YFB1600400), the Shenzhen Science and Technology Program (Grant No.RCBS20200714114920272) and the Key Research and Development Program of Guangzhou (202007050002).

Funding

National Research and Development Program of China, 2020YFB1600400, Shenzhen Science and Technology Program, RCBS20200714114920272, Key Research and Development Program of Guangzhou, 202007050002.

Author information

Authors and Affiliations

  1. School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-Sen University, Shenzhen, 518107, China

    Jianliang Min, Ming Cai, Chao Gou & Chen Xiong

  2. Guangdong Province Key Laboratory of Intelligent Transportation System, Guangzhou, China

    Ming Cai, Chao Gou & Chen Xiong

  3. School of Biomedical Engineering, Shenzhen Campus of Sun Yat-Sen University, Shenzhen, 518107, China

    Jianliang Min

  4. Research Institute for Road Safety of MPS, Beijing, 100051, China

    Xuejiao Yao

Authors
  1. Jianliang Min
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chao Gou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chen Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xuejiao Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chao Gou.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Min, J., Cai, M., Gou, C. et al. Fusion of forehead EEG with machine vision for real-time fatigue detection in an automatic processing pipeline. Neural Comput & Applic 35, 8859–8872 (2023). https://doi.org/10.1007/s00521-022-07466-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07466-0

Keywords

Search

Navigation