Abstract
Drowsiness of drivers is a critical problem and has recently attracted a lot of attention from both academia and industry. A real-time driver’s drowsiness detection system is often considered as a crucial component of an Advanced Driver Assistance System (ADAS). Although, there are a number of physical parameters associated with drowsiness like blink frequency, eye closure duration, pose, gaze, etc., yawing can also be used as an indicator of drowsiness. This work presents a novel deep learning-based framework for driver’s drowsiness prediction based on yawn detection in a video stream. The proposed approach uses a combination of a convolutional neural network (CNN), 1D-CNN, and bi-directional LSTM (Bi-LSTM). In the first step, the pipeline extracts the mouth region from each frame of the video using a combination of face and landmark detector. In the subsequent step, spatial information from the mouth region is extracted using a pre-trained deep convolutional neural network (DCNN). Finally, temporal information which models the evaluation of yawn using the extracted mouth feature is learned using a blend of 1D-CNN and bi-directional LSTM (Bi-LSTM). Experiments were performed on manually extracted and annotated video clips obtained from two publically available drowsiness detection dataset namely YawDD and NTHU-DDD. Experimental results show the effectiveness of the proposed approach both in terms of recognition accuracy and computational efficiency. Thus, the proposed pipeline is a good candidate for real-time implementation of yawn detection system for driver’s drowsiness prediction on an embedded device.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
https://timesofindia.indiatimes.com/india/over-1-51-lakh-died-in-road-accidents-last-year-up-tops-among-states/articleshow/72078508.cms. Accessed 18 Nov 2019
National Center for Statistics and Analysis: 2016 fatal motor vehicle crashes: Overview. (Traffic Safety Facts Research Note. Report No. DOT HS 812 456) (October 2017). National Highway Traffic Safety Administration, Washington, DC
Sikander, G., Anwar, S.: Driver fatigue detection systems: a review. IEEE Trans. Intell. Transp. Syst. 20(6), 2339–2352 (2018)
Ji, Q., Zhu, Z., Lan, P.: Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans. Veh. Technol. 53(4), 1052–1068 (2004)
Wang, T., Shi, P.: Yawning detection for determining driver drowsiness. In: Proceedings of IEEE International Workshop on VLSI Design and Video Technology, May 2005, pp. 373–376 (2005)
Rongben, W., Lie, G., Bingliang, T., Lisheng, J.: Monitoring mouth movement for driver fatigue or distraction with one camera. In: Proceedings of the 7th IEEE International Conference on Intelligent Transportation Systems, October 2004, pp. 314–319 (2004)
Lu, Y., Wang, Z.: Detecting driver yawning in successive images. In: 1st International Conference on Bioinformatics and Biomedical Engineering, July 2007, pp. 581–583 (2007)
Fan, X., Yin, B.C., Sun, Y.F.: Yawning detection for monitoring driver fatigue. In: IEEE International Conference on Machine Learning and Cybernetics, August 2007, vol. 2, pp. 664–668 (2007)
Medeiros, R.S., Scharcanski, J., Wong, A.: Multi-scale stochastic color texture models for skin region segmentation and gesture detection. In: IEEE International Conference on Multimedia and Expo Workshops (ICMEW), July 2013, pp. 1–4 (2013)
Li, L., Chen, Y., Li, Z.: Yawning detection for monitoring driver fatigue based on two cameras. In: 12th IEEE International Conference on Intelligent Transportation Systems, October 2009, pp. 1–6 (2009)
Bouvier, C., Benoit, A., Caplier, A., Coulon, P.-Y.: Open or closed mouth state detection: static supervised classification based on log-polar signature. In: Blanc-Talon, J., Bourennane, S., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2008. LNCS, vol. 5259, pp. 1093–1102. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88458-3_99
Alioua, N., Amine, A., Rziza, M.: Driver’s fatigue detection based on yawning extraction. Int. J. Veh. Technol. 2014 (2014)
Omidyeganeh, M., et al.: Yawning detection using embedded smart cameras. IEEE Trans. Instrum. Meas. 65(3), 570–582 (2016)
Jie, Z., Mahmoud, M., Stafford-Fraser, Q., Robinson, P., Dias, E., Skrypchuk, L.: Analysis of yawning behaviour in spontaneous expressions of drowsy drivers. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018, May 2018, pp. 571–576 (2018)
Zhang, W., Su, J.: Driver yawning detection based on long short-term memory networks. In: IEEE Symposium Series on Computational Intelligence (SSCI), November 2017, pp. 1–5 (2017)
King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)
Barsoum, E., Zhang, C., Ferrer, C.C., Zhang, Z.: Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, October 2016, pp. 279–283. ACM (2016)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: continual prediction with LSTM (1999)
Abtahi, S., Omidyeganeh, M., Shirmohammadi, S., Hariri, B.: YawDD: a yawning detection dataset. In: Proceedings of the 5th ACM Multimedia Systems Conference, March 2014, pp. 24–28. ACM (2014)
Weng, C.-H., Lai, Y.-H., Lai, S.-H.: Driver drowsiness detection via a hierarchical temporal deep belief network. In: Chen, C.-S., Lu, J., Ma, K.-K. (eds.) ACCV 2016. LNCS, vol. 10118, pp. 117–133. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54526-4_9
Chollet, F.: Keras (2017). https://github.com/fchollet/keras
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Saurav, S., Mathur, S., Sang, I., Prasad, S.S., Singh, S. (2020). Yawn Detection for Driver’s Drowsiness Prediction Using Bi-Directional LSTM with CNN Features. In: Tiwary, U., Chaudhury, S. (eds) Intelligent Human Computer Interaction. IHCI 2019. Lecture Notes in Computer Science(), vol 11886. Springer, Cham. https://doi.org/10.1007/978-3-030-44689-5_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-44689-5_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-44688-8
Online ISBN: 978-3-030-44689-5
eBook Packages: Computer ScienceComputer Science (R0)