Skip to main content
SpringerLink
Log in

Hierarchical deep neural networks to detect driver drowsiness

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Driver drowsiness is one of the main reasons for deadly accidents, especially on suburban roads. Researchers have used many methods for analyzing videos and detecting drowsiness, and the most up-to-date methods among them are using deep learning. This paper proposes a hierarchical framework comprising deep networks with split spatial and temporal phases referred to as hierarchical deep drowsiness detection (HDDD) network. The proposed method uses ResNet to detect the driver’s face, lighting condition, and whether the driver is wearing glasses or not. This phase also causes a significant increase in eyes and mouth detection percentage in the next stage. Afterward, the LSTM network is used to take advantage of temporal information between the frames. The average accuracy of the drowsiness detection system is reached 87.19 percent.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Alioua N, Amine A, Rziza M (2014) Driver’s fatigue detection based on yawning extraction. Int J Veh Technol, vol 2014

  2. Bergasa LM, Nuevo J, Sotelo MA, Barea R, Lopez ME (2006) Real-time system for monitoring driver vigilance. IEEE Trans Intell Transp Syst 7(1):63–77

    Article  Google Scholar 

  3. Bhandari G, Durge A, Bidwai A, Aware U (2014) Driver drowsiness monitoring. Int J Sci Eng Technol 3(2):203–206

    Google Scholar 

  4. Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions. arXiv:1705.01842

  5. Choi IH, Kim YG (2014) Head pose and gaze direction tracking for detecting a drowsy driver. In: 2014 International conference on big data and smart computing (BIGCOMP). IEEE, pp 241–244

  6. Craye C, Karray F (2015) Driver distraction detection and recognition using rgb-d sensor. arXiv:1502.00250

  7. Diaz-Chito K, Hernández-Sabaté A., López A. M. (2016) A reduced feature set for driver head pose estimation. Appl Soft Comput 45:98–107

    Article  Google Scholar 

  8. Feng R, Zhang G, Cheng B (2009) An on-board system for detecting driver drowsiness based on multi-sensor data fusion using dempster-shafer theory. In: 2009 International conference on networking, sensing and control. IEEE, pp 897–902

  9. González-Ortega D, Díaz-Pernas F, Antón-Rodríguez M, Martínez-Zarzuela M, Díez-Higuera J (2013) Real-time vision-based eye state detection for driver alertness monitoring. Pattern Anal Appl 16(3):285–306

    Article  MathSciNet  Google Scholar 

  10. Gritzman AD, Rubin DM, Pantanowitz A (2015) Comparison of colour transforms used in lip segmentation algorithms. SIViP 9(4):947–957

    Article  Google Scholar 

  11. Guo JM, Markoni H (2019) Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimed Tools Appl 78(20):29,059–29,087

    Article  Google Scholar 

  12. Hachisuka S (2013) Human and vehicle-driver drowsiness detection by facial expression. In: 2013 International conference on biometrics and Kansei engineering. IEEE, pp 320–326

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  14. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  15. Ibrahim MM, Soraghan JJ, Petropoulakis L, Di Caterina G (2015) Yawn analysis with mouth occlusion detection. Biomed Sig Process Control 18:360–369

    Article  Google Scholar 

  16. Ingre M, Åkerstedt T, Peters B, Anund A, Kecklund G (2006) Subjective sleepiness, simulated driving performance and blink duration: examining individual differences. J Sleep Res 15(1):47–53

    Article  Google Scholar 

  17. Ishii Y, Ogitsu T, Takemura H, Mizoguchi H (2014) Real-time eyelid open/closed state recognition based on hlac towards driver drowsiness detection. In: 2014 IEEE international conference on robotics and biomimetics (ROBIO 2014). IEEE, pp 2449–2454

  18. Jones M, Viola P (2003) Fast multi-view face detection. Mitsubishi Electr Res Lab TR-20003-96 3(14):2

    Google Scholar 

  19. Kalbkhani H, Amirani MC (2012) An efficient algorithm for lip segmentation in color face images based on local information. J World’s Electr Eng Technol 1(1):12–16

    Google Scholar 

  20. Kaplan S, Guvensan MA, Yavuz AG, Karalurt Y (2015) Driver behavior analysis for safe driving: A survey. IEEE Trans Intell Transp Syst 16 (6):3017–3032

    Article  Google Scholar 

  21. Li G, Lee BL, Chung WY (2015) Smartwatch-based wearable eeg system for driver drowsiness detection. IEEE Sensors J 15(12):7169–7180

    Article  Google Scholar 

  22. Li Z, Li SE, Li R, Cheng B, Shi J (2017) Online detection of driver fatigue using steering wheel angles for real driving conditions. Sensors 17(3):495

    Article  Google Scholar 

  23. Li H, Lin Z, Shen X, Brandt J, Hua G (2015) A convolutional neural network cascade for face detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 5325–5334

  24. Lin CT, Chang CJ, Lin BS, Hung SH, Chao CF, Wang IJ (2010) A real-time wireless brain–computer interface system for drowsiness detection. IEEE Trans Biomed Circ Syst 4(4):214–222

    Article  Google Scholar 

  25. Lu X, Wang W, Shen J, Tai YW, Crandall DJ, Hoi SC (2020) Learning video object segmentation from unlabeled videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8960–8970

  26. Mandal B, Li L, Wang GS, Lin J (2016) Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Trans Intell Transp Syst 18(3):545–557

    Article  Google Scholar 

  27. Mbouna RO, Kong SG, Chun MG (2013) Visual analysis of eye state and head pose for driver alertness monitoring. IEEE Trans Intell Transp Syst 14(3):1462–1469

    Article  Google Scholar 

  28. Mehta S, Dadhich S, Gumber S, Jadhav Bhatt A (2019) Real-time driver drowsiness detection system using eye aspect ratio and eye closure ratio. In: Proceedings of international conference on sustainable computing in science, technology and Management (SUSCOM), Amity University Rajasthan, Jaipur-India

  29. Omidyeganeh M, Shirmohammadi S, Abtahi S, Khurshid A, Farhan M, Scharcanski J, Hariri B, Laroche D, Martel L (2016) Yawning detection using embedded smart cameras. IEEE Trans Instrum Meas 65(3):570–582

    Article  Google Scholar 

  30. Park S, Pan F, Kang S, Yoo CD (2016) Driver drowsiness detection system based on feature representation learning using various deep networks. In: Asian conference on computer vision. Springer, pp. 154–164

  31. Pratama BG, Ardiyanto I, Adji TB (2017) A review on driver drowsiness based on image, bio-signal, and driver behavior. In: 2017 3rd international conference on science and technology-computer (ICST). IEEE, pp 70–75

  32. Saini V, Saini R (2014) Driver drowsiness detection system and techniques: a review. Int J Comput Sci Inf Technol 5(3):4245–4249

    Google Scholar 

  33. Shih TH, Hsu CT (2016) Mstn: Multistage spatial-temporal network for driver drowsiness detection. In: Asian conference on computer vision. Springer, pp 146–153

  34. Tansakul W, Tangamchit P (2016) Fatigue driver detection system using a combination of blinking rate and driving inactivity. J Autom Control Eng, vol 4(1)

  35. Tu Y, Zeng C, Yeh C, Huang S, Cheng T, Ouhyoung M (2011) Real-time head pose estimation using depth map for avatar control. In: Proceedings of IPPR conference on computer vision, graphics, and image processing

  36. Weng CH, Lai YH, Lai SH (2016) Driver drowsiness detection via a hierarchical temporal deep belief network. In: Asian conference on computer vision. Springer, pp 117–133

  37. Wood R, Olszewska JI (2012) Lighting-variable adaboost based-on system for robust face detection. In: Proceedings of the 5th international conference on bio-inspired systems and signal processing. SciTePress digital library, pp. 494–497

  38. Yu J, Park S, Lee S, Jeon M (2016) Representation learning, scene understanding, and feature fusion for drowsiness detection. In: Asian conference on computer vision. Springer, pp 165–177

  39. Zhang L, Liu F, Tang J (2015) Real-time system for driver fatigue detection by rgb-d camera. ACM Trans Intell Syst Technol (TIST) 6(2):1–17

    Google Scholar 

  40. Zhang Z, Luo P, Loy CC, Tang X (2014) Facial landmark detection by deep multi-task learning. In: European conference on computer vision. Springer, pp 94–108

  41. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig Process Lett 23(10):1499–1503

    Article  Google Scholar 

  42. Zhao L, Wang Z, Wang X, Qi Y, Liu Q, Zhang G (2016) Human fatigue expression recognition through image-based dynamic multi-information and bimodal deep learning. J Electron Imaging 25(5):053,024

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Engineering, Alzahra University, Tehran, Iran

    Samaneh Jamshidi, Reza Azmi & Mehran Sharghi

  2. School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran

    Mohsen Soryani

Authors
  1. Samaneh Jamshidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reza Azmi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mehran Sharghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohsen Soryani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samaneh Jamshidi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jamshidi, S., Azmi, R., Sharghi, M. et al. Hierarchical deep neural networks to detect driver drowsiness. Multimed Tools Appl 80, 16045–16058 (2021). https://doi.org/10.1007/s11042-021-10542-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10542-7

Keywords

Search

Navigation