Skip to main content
SpringerLink
Log in

Computer vision-based driver fatigue detection framework with personalization threshold and multi-feature fusion

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

A novel non-intrusive and computer vision-based framework for driver fatigue detection from video is proposed in this paper. To improve the judging accuracy of the driver’s facial expressions, the personalized threshold is proposed based on the driver’s eye aspect ratio and mouth aspect ratio instead of the traditional average threshold, as individual drivers have different eye and mouth sizes. In order to alleviate the impact of the lack of relevant public data on model training, transfer learning is employed to train the eye and mouth state classifier. Furthermore, to address the low universality and accuracy of driver fatigue detection caused by using only one type of facial features, multiple features, including appearance-based features and deep learning-based features, are utilized. The experiment results indicate that our method achieves a 92.21% F1 score and 29 fps, outperforming traditional methods on the public NTHU-DDD dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability statement

NTHU-DDD can be downloaded from https://cv.cs.nthu.edu.tw/php/callforpaper/datasets/DDD/

References

  1. Amodio, A., Ermidoro, M., Maggi, D., et al.: Automatic detection of driver impairment based on pupillary light re-flex. IEEE Trans. Intell. Transp. Syst. 20(8), 3038–3048 (2019)

    Article  Google Scholar 

  2. Kamti, M.K., Iqbal, R.: Evolution of driver fatigue detection techniques-a review from 2007 to 2021. Transp. Res. Rec. 2676(12), 485–507 (2022)

    Article  Google Scholar 

  3. Zheng, W.L., Gao, K.P., Li, G., et al.: Vigilance estimation using a wearable EOG device in real driving environment. IEEE Trans. Intell. Transp. Syst. 21(1), 170–184 (2020)

    Article  Google Scholar 

  4. Zhu, M., Chen, J., Li, H., et al.: Vehicle driver drowsiness detection method using wearable EEG based on convolution neural network. Neural Comput. Appl. 33, 13965–13980 (2021)

    Article  Google Scholar 

  5. Arefnezhad, S., Samiee, S., Eichberger, A., et al.: Ap-plying deep neural networks for multi-level classification of driver drowsiness using vehicle-based measures. Expert Syst. Appl. 162, 113778 (2020)

    Article  Google Scholar 

  6. Jeon, Y., Kim, B., Baek, Y.: Ensemble CNN to detect drowsy driving with in-vehicle sensor data. Sensors 21, 2372 (2021)

    Article  Google Scholar 

  7. Ling, Y., Luo, R., Dong, X., et al.: Driver eye location and state estimation based on a robust model and data augmentation. IEEE Access 9, 67219–67231 (2021)

    Article  Google Scholar 

  8. Zhuang, Q.Y., Zhang, K.H., Wang, J.Y., et al.: Driver fatigue detection method based on eye states with pupil and iris segmentation. IEEE Access 8, 173440–173449 (2020)

    Article  Google Scholar 

  9. Yanyan, Y.A.N.G., Leixiao, L.I., Hao, L.I.N.: A review of research on fatigue driving detection based on driver facial features. J. Front. Comput. Sci. Technol. 17(6), 1249 (2023)

    Google Scholar 

  10. Ji, Y.Y., Wang, S.G., Zhao, Y., et al.: Fatigue state detection based on multi-index fusion and state recognition network. IEEE Access 7, 64136–64147 (2019)

    Article  Google Scholar 

  11. Huang, R., Wang, Y., Guo, L.: P-FDCN based eye state analysis for fatigue detection, In: 2018 IEEE 18th International Conference on Communication Technology (ICCT), 1174-1178 (2018)

  12. Huang, R., Wang, Y., Li, Z.J., et al.: RF-DCM: multigranularity deep convolutional model based on feature re-calibration and fusion for driver fatigue detection. IEEE Trans. Intell. Transp. Syst. 23(1), 630–640 (2022)

    Article  Google Scholar 

  13. Du, G.L., Li, T., Li, C.Q., et al.: Vision-based fatigue driving recognition method integrating heart rate and facial features. IEEE Trans. Intell. Transp. Syst. 22(5), 3089–3100 (2021)

    Article  Google Scholar 

  14. Yang, H., Liu, I., Min, W.D.: Driver yawning detection based on subtle facial action recognition. IEEE Trans. Multimed. 23, 572–583 (2021)

    Article  Google Scholar 

  15. Li, K., Gong, Y., Ren, Z.: A fatigue driving detection algorithm based on facial multi-feature fusion. IEEE Access 8, 101244–101259 (2020)

    Article  Google Scholar 

  16. Kong, X.J., Xia, F., Li, J.X., et al.: A shared bus profiling scheme for smart cities based on heterogeneous mobile crowdsourced Data. IEEE Trans. Ind. Inform. 16(2), 1436–1444 (2020)

    Article  Google Scholar 

  17. Ning, Z.L., Huang, J., Wang, X.J., et al.: Mobile edge computing enabled internet of vehicles: toward energy-efficient scheduling. IEEE Netw. 33(5), 198–205 (2019)

    Article  Google Scholar 

  18. Feng, Y., Li, X.L., Gong, Y.B., et al.: A real-time driving drowsiness detection algorithm with individual differences consideration. IEEE Access 7, 179396–179408 (2019)

    Article  Google Scholar 

  19. Ji, Y.Y., Wang, S.G., Lu, Y., et al.: Eye and mouth state detection algorithm based on contour feature extraction. J. Electron. Imag. 27(5), 1–8 (2018)

    Article  Google Scholar 

  20. Zhang, C., Wu, X.P., Zheng, X., et al.: Driver drowsiness detection using multi-channel second order blind identifications. IEEE Access 7, 11829–11843 (2019)

    Article  Google Scholar 

  21. Al-Libawy, H., Al-Ataby, A., Al-Nuaimy, W., Al-Taee, M.A.: Modular design of fatigue detection in naturalistic driving environments. Accid. Anal. Prev. 120, 188–194 (2018)

  22. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) 2014, 1867–1874 (2014)

  23. Mehmood, I., Li, H., Umer, W., Arsalan, A., Shakeel, M.S., Anwer, S.: Validity of facial features? Geometric measurements for real-time assessment of mental fatigue in construction equipment operators. Adv. Eng. Inform. 54, 101777 (2022)

    Article  Google Scholar 

  24. Weng, C.H., Lai, Y.H., Lai, S.H.: Driver drowsiness detection via a hierarchical temporal deep belief network, In: asian conference on computer vision workshop on driver drowsiness detection from video, Taiwan: Taipei, 2016

  25. Howard, A., Sandler, M., Chu, G., et al.: Searching for mobilenetv3, In: Proceedings of the IEEE/CVF international conference on computer vision. 1314-1324 (2019)

  26. Chen, P., et al.: A multi-domain fusion human motion recognition method based on lightweight network. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2022)

    Google Scholar 

  27. Ao, B.Q., Yang, S., Linghu, J.Q., et al.: Design of fatigue driving detection system based on cascaded neural network. J. Syst. Simul. 34(2), 323–333 (2022)

    Google Scholar 

  28. Fu, R.R., Wang, H., Zhao, W.B.: Dynamic driver fatigue detection using hidden markov model in real driving condition. Expert Syst. Appl. 63, 397–411 (2016)

    Article  Google Scholar 

  29. Yu, J., Park, S., Lee, S., Jeon, M. Representation Learning, Scene Understanding, and Feature Fusion for Drowsiness Detection, Chen, CS., Lu, J., Ma, KK. (eds) Computer Vision, ACCV 2016 Workshops. ACCV 2016. In: Lecture Notes in Computer Science, 2017, 10118. Springer, Cham

  30. Guo, J.M., Markoni, H.: Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimed. Tools Appl. 78, 29059–29087 (2019)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Inner Mongolia Autonomous Region College Youth Science and Technology Talent Support Program Project (NJYT22084), Inner Mongolia Autonomous Region Key R and D and Achievement Transformation Program Project (2022YFSJ0013), Science and Technology Plan Projects of Inner Mongolia Autonomous Region (2020GG0104), and Natural Science Foundation of Inner Mongolia (2023MS06008).

Author information

Authors and Affiliations

  1. College of Data Science and Application, Inner Mongolia University of Technology, Hohhot, 010080, China

    Xiaolei Li & Yanyan Yang

  2. School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, 300384, China

    Hao Lin & Jinze Du

Authors
  1. Xiaolei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinze Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yanyan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LL: Investigation, Supervision, Writing review. HL: Conceptualization, Data curation, Writing original draft, Writing review editing. JD: Writing review and editing. YY: Methodology, Visualization, Writing original draft.

Corresponding author

Correspondence to Hao Lin.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

See Tables 4, 5, 6 and 7.

Table 5 Time-consuming of each module
Full size table
Table 6 Comparison of slow blink detection with different indicators using personalized threshold
Full size table
Table 7 Comparison of yawn detection with different indicators using personalized threshold
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Lin, H., Du, J. et al. Computer vision-based driver fatigue detection framework with personalization threshold and multi-feature fusion. SIViP 18, 505–514 (2024). https://doi.org/10.1007/s11760-023-02733-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02733-6

Keywords

Search

Navigation