Skip to main content
SpringerLink
Log in

A novel deep-learning model based on τ-shaped convolutional network (τNet) with long short-term memory (LSTM) for physiological fatigue detection from EEG and EOG signals

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

In recent years, fatigue driving has become the main cause of traffic accidents, leading to increased attention towards fatigue detection systems. However, the pooling and strided convolutional operations in fatigue detection algorithm based on traditional deep learning methods may led to the loss of some useful information. This paper proposed a novel \(\uptau\)-shaped convolutional network (\(\mathrm{\tau Net}\)) aiming to address this issue. Unlike traditional network structures, \(\mathrm{\tau Net}\) incorporates the operations of upsampling features and concatenating high- and low-level features, enabling full utilization of useful information. Moreover, considering that the fatigue state is a mental state involving temporal evolution, we proposed the novel long short-term memory (LSTM)–\(\uptau\)-shaped convolutional network (LSTM-\(\mathrm{\tau Net}\)), a parallel structure composed of LSTM and \(\mathrm{\tau Net}\) for fatigue detection, where \(\mathrm{\tau Net}\) extracts time-invariant features with location information, and LSTM extracts long temporal dependencies. We compared LSTM-\(\mathrm{\tau Net}\) with six competing methods based on two datasets. Results showed that the proposed algorithm achieved higher classification accuracy than the other methods, with 94.25% on EEG data (binary classification) and 82.19% on EOG data (triple classification). Additionally, the proposed algorithm exhibits low computational cost, good training stability, and robustness against insufficient training. Therefore, it is promising for further implementation of fatigue online detection systems.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Vanlaar W, Simpson H, Mayhew D, Robertson R (2008) Fatigued and drowsy driving: a survey of attitudes, opinions and behaviors. J Safety Res 39:303–309. https://doi.org/10.1016/j.jsr.2007.12.007

    Article  PubMed  Google Scholar 

  2. Gevins A, Leong H, Du R et al (1995) Towards measurement of brain function in operational environments. Biol Psychol 40:169–186. https://doi.org/10.1016/0301-0511(95)05105-8

    Article  CAS  PubMed  Google Scholar 

  3. Khunpisuth O, Chotchinasri T, Koschakosai V, Hnoohom N (2017) Driver drowsiness detection using eye-closeness detection. 12th Int Conf Signal Image Technol Internet-Based Syst SITIS 2016, Naples, ITALY, Nov 28-Dec 01,2016, Proceedings. pp 661–668. https://doi.org/10.1109/SITIS.2016.110

    Chapter  Google Scholar 

  4. Hashemi Nazari SS, Moradi A, Rahmani K (2017) A systematic review of the effect of various interventions on reducing fatigue and sleepiness while driving. Chin J Traumatol - English Ed 20:249–258. https://doi.org/10.1016/j.cjtee.2017.03.005

    Article  Google Scholar 

  5. Du G, Zhang L, Su K et al (2022) A multimodal fusion fatigue driving detection method based on heart rate and PERCLOS. IEEE Trans Intell Transp Syst 23:21810–21820. https://doi.org/10.1109/TITS.2022.3176973

    Article  Google Scholar 

  6. Ansari S, Naghdy F, Du H, Pahnwar YN (2022) Driver mental fatigue detection based on head posture using new modified reLU-BiLSTM deep neural network. IEEE Trans Intell Transp Syst 23:10957–10969. https://doi.org/10.1109/TITS.2021.3098309

    Article  Google Scholar 

  7. Li Z, Chen L, Nie L, Yang SX (2022) A novel learning model of driver fatigue features representation for steering wheel angle. IEEE Trans Veh Technol 71:269–281. https://doi.org/10.1109/TVT.2021.3130152

    Article  Google Scholar 

  8. Li R, Chen YV, Zhang L (2021) A method for fatigue detection based on driver’s steering wheel grip. Int J Ind Ergon 82:103083. https://doi.org/10.1016/j.ergon.2021.103083

    Article  Google Scholar 

  9. Karchani M, Mazloumi A, Saraji GN et al (2015) Presenting a model for dynamic facial expression changes in detecting drivers’ drowsiness. Electron Physician 7:1073–1077. https://doi.org/10.14661/2015.1073-1077

    Article  PubMed  PubMed Central  Google Scholar 

  10. Liu Q, Liu Y, Chen K et al (2021) Research on channel selection and multi-feature fusion of EEG signals for mental fatigue detection. Entropy 23:457. https://doi.org/10.3390/e23040457

    Article  PubMed  PubMed Central  Google Scholar 

  11. Zheng WL, Lu BL (2017) A multimodal approach to estimating vigilance using EEG and forehead EOG. J Neural Eng 14:026017. https://doi.org/10.1088/1741-2552/aa5a98

    Article  PubMed  Google Scholar 

  12. Cao Z, Chuang CH, King JK, Lin CT (2019) Multi-channel EEG recordings during a sustained-attention driving task. Sci Data 6:19. https://doi.org/10.1038/s41597-019-0027-4

    Article  PubMed  PubMed Central  Google Scholar 

  13. Liu CC, Hosking SG, Lenné MG (2009) Predicting driver drowsiness using vehicle measures: recent insights and future challenges. J Safety Res 40:239–245. https://doi.org/10.1016/j.jsr.2009.04.005

    Article  PubMed  Google Scholar 

  14. Horne JA, Baulk SD (2004) Awareness of sleepiness when driving. Psychophysiology 41:161–165. https://doi.org/10.1046/j.1469-8986.2003.00130.x

    Article  PubMed  Google Scholar 

  15. Tong W, Chen W, Han W et al (2020) Channel-attention-based densenet network for remote sensing image scene classification. IEEE J Sel Top Appl Earth Obs Remote Sens 13:4121–4132. https://doi.org/10.1109/JSTARS.2020.3009352

    Article  Google Scholar 

  16. Gao Z, Wang X, Yang Y et al (2019) EEG-based spatio-temporal convolutional neural network for driver fatigue evaluation. IEEE Trans Neural Netw Learn Syst 30:2755–2763. https://doi.org/10.1109/TNNLS.2018.2886414

    Article  PubMed  Google Scholar 

  17. Yeo MVM, Li X, Shen K, Wilder-Smith EPV (2009) Can SVM be used for automatic EEG detection of drowsiness during car driving? Saf Sci 47:115–124. https://doi.org/10.1016/j.ssci.2008.01.007

    Article  Google Scholar 

  18. Dong N, Li Y, Gao Z et al (2019) A WPCA-based method for detecting fatigue driving from EEG-based internet of vehicles system. IEEE Access 7:124702–124711. https://doi.org/10.1109/ACCESS.2019.2937914

    Article  Google Scholar 

  19. Peng Y, Wong CM, Wang Z et al (2019) Fatigue evaluation using multi-scale entropy of EEG in SSVEP-based BCI. IEEE Access 7:108200–108210. https://doi.org/10.1109/ACCESS.2019.2932503

    Article  Google Scholar 

  20. Papadelis C, Chen Z, Kourtidou-Papadeli C et al (2007) Monitoring sleepiness with on-board electrophysiological recordings for preventing sleep-deprived traffic accidents. Clin Neurophysiol 118:1906–1922. https://doi.org/10.1016/j.clinph.2007.04.031

    Article  PubMed  Google Scholar 

  21. Zheng WL, Gao K, Li G et al (2020) Vigilance estimation using a wearable EOG device in real driving environment. IEEE Trans Intell Transp Syst 21:170–184. https://doi.org/10.1109/TITS.2018.2889962

    Article  Google Scholar 

  22. Jammes B, Sharabty H, Esteve D (2008) Automatic EOG analysis: a first step toward automatic drowsiness scoring during wake-sleep transitions. Somnologie 12:227–232. https://doi.org/10.1007/s11818-008-0351-y

    Article  Google Scholar 

  23. Das AK, Kumar P, Halder S (2022) Experimentation on detection and analysis of drowsiness and fatigue based on permutation entropy and hurst exponent. Condition Assessment Techniques in Electrical Systems: IEEE 6th International Conference, IEEE CATCON 2022, Natl Inst Technol, Dept Elect Engn, Durgapur, INDIA, DEC 17-19, 2022, Proceedings. IEEE, pp 239–243. https://doi.org/10.1109/CATCON56237.2022.10077702

    Chapter  Google Scholar 

  24. Lee DH, Jeong JH, Kim K et al (2020) Continuous EEG decoding of pilots’ mental states using multiple feature block-based convolutional neural network. IEEE Access 8:121929–121941. https://doi.org/10.1109/ACCESS.2020.3006907

    Article  Google Scholar 

  25. Wang H, Xu L, Bezerianos A et al (2021) Linking attention-based multiscale CNN with dynamical GCN for driving fatigue detection. IEEE Trans Instrum Meas 70:2504811. https://doi.org/10.1109/TIM.2020.3047502

    Article  Google Scholar 

  26. Falk T, Mai D, Bensch R et al (2019) U-Net: deep learning for cell counting, detection, and morphometry. Nat Methods 16:67–70. https://doi.org/10.1038/s41592-018-0261-2

    Article  CAS  PubMed  Google Scholar 

  27. Mai Y, Chen Z, Yu B et al (2022) Non-contact heartbeat detection based on ballistocardiogram using UNet and bidirectional long short-term memory. IEEE J Biomed Heal Informatics 26:3720–3730. https://doi.org/10.1109/JBHI.2022.3162396

    Article  Google Scholar 

  28. Hersek S, Semiz B, Shandhi MMH et al (2020) A globalized model for mapping wearable seismocardiogram signals to whole-body ballistocardiogram signals based on deep learning. IEEE J Biomed Heal Informatics 24:1296–1309. https://doi.org/10.1109/JBHI.2019.2931872

    Article  Google Scholar 

  29. Perslev M, Darkner S, Kempfner L et al (2021) U-sleep: resilient high-frequency sleep staging. npj Digit Med 4:1–12. https://doi.org/10.1038/s41746-021-00440-5

    Article  Google Scholar 

  30. Khessiba S, Blaiech AG, Ben Khalifa K et al (2021) Innovative deep learning models for EEG-based vigilance detection. Neural Comput Appl 33:6921–6937. https://doi.org/10.1007/s00521-020-05467-5

    Article  Google Scholar 

  31. Jiao Y, Deng Y, Luo Y, Lu BL (2020) Driver sleepiness detection from EEG and EOG signals using GAN and LSTM networks. Neurocomputing 408:100–111. https://doi.org/10.1016/j.neucom.2019.05.108

    Article  Google Scholar 

  32. Karim F, Majumdar S, Darabi H, Chen S (2017) LSTM fully convolutional networks for time series classification. IEEE Access 6:1662–1669. https://doi.org/10.1109/ACCESS.2017.2779939

    Article  Google Scholar 

  33. Teimouri N, Dyrmann M, Jørgensen RN (2019) A novel spatio-temporal FCN-LSTM network for recognizing various crop types using multi-temporal radar images. Remote Sens 11:1–18. https://doi.org/10.3390/rs11080893

    Article  Google Scholar 

  34. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  CAS  PubMed  Google Scholar 

  35. Shahid A, Wilkinson K, Marcu S, Shapiro CM (2012) STOP, THAT and one hundred other sleep scales. STOP, THAT One Hundred Other Sleep Scales 1–406. https://doi.org/10.1007/978-1-4419-9893-4

  36. Navab N, Hornegger J, Wells WM, Frangi AF (2015) Medical image computing and computer-assisted intervention - MICCAI 2015: 18th International Conference Munich, Germany, October 5–9, 2015 proceedings, part III. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 9351:12–20. https://doi.org/10.1007/978-3-319-24574-4

    Article  Google Scholar 

  37. Wang Z, Yan W, Oates T (2017) Time series classification from scratch with deep neural networks: a strong baseline. Proceedings of the IEEE International Joint Conference on Neural Networks. pp 1578–1585. https://doi.org/10.1109/IJCNN.2017.7966039

    Chapter  Google Scholar 

  38. Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. Proceedings of the 27th International Conference on International Conference on Machine Learning. pp 807–814

    Google Scholar 

  39. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. 32nd Int Conf Mach Learn ICML 1:448–456

    Google Scholar 

  40. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7132–7141

    Google Scholar 

  41. Lin M, Chen Q, Yan S (2014) Network in network. arXiv:1312.4400

    Google Scholar 

  42. Srivastava N, Hinton G, Krizhevsky A et al (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    Google Scholar 

  43. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision. pp 1026–1034. https://doi.org/10.1109/ICCV.2015.123

    Chapter  Google Scholar 

  44. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. arXiv:1412.6980

    Google Scholar 

  45. Chollet F  et al. Keras [Online]. Available: https://github.com/fchollet/keras

  46. Cui J, Lan Z, Sourina O, Muller-Wittig W (2022) EEG-based cross-subject driver drowsiness recognition with an interpretable convolutional neural network. IEEE Trans Neural Networks Learn Syst 34:7921–7933. https://doi.org/10.1109/TNNLS.2022.3147208

    Article  Google Scholar 

  47. Karim F, Majumdar S, Darabi H, Harford S (2019) Multivariate LSTM-FCNs for time series classification. Neural Netw 116:237–245. https://doi.org/10.1016/j.neunet.2019.04.014

    Article  PubMed  Google Scholar 

  48. Shi S (2021) Visualizing data using GTSNE. arXiv:2108.01301

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the Prof. Bao-Liang Lu and his research team for providing the public SEED-VIG dataset. The authors also thank Dr. Fazle Karim for his demo code.

Author information

Authors and Affiliations

  1. State Key Laboratory of Power Transmission Equipment Technology, School of Electrical Engineering, Chongqing University, Chongqing, 400044, People’s Republic of China

    Le He, Li Zhang, Xiangtian Lin & Yunfeng Qin

Authors
  1. Le He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiangtian Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunfeng Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Zhang.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, L., Zhang, L., Lin, X. et al. A novel deep-learning model based on τ-shaped convolutional network (τNet) with long short-term memory (LSTM) for physiological fatigue detection from EEG and EOG signals. Med Biol Eng Comput 62, 1781–1793 (2024). https://doi.org/10.1007/s11517-024-03033-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-024-03033-y

Keywords

Search

Navigation