Abstract
Drowsiness and fatigue of the drivers are amongst the significant causes of the car accidents. Every year the number of deaths and fatalities are tremendously increasing due to multifaceted issues and henceforth requires an intelligent processing system for accident avoidance. In relevant with this, an effective driver drowsiness detection system is proposed. The main challenges are robustness of the algorithm towards variation of the human face and real-time processing capability. The first challenge pertaining to the facial variation has been handled well using conventional image processing and hand-craft features of computer vision algorithms. Yet, variations such as facial expression, lighting condition, intra-class variation, and pose variation are additional issues that conventional method failed to address. Deep learning is an alternative solution which provides a better performance by learning features automatically. Thus, this paper proposed a new concept for handling the real-time driver drowsiness detection using the hybrid of convolutional neural network (CNN) and long short-term memory (LSTM). The performance of the system has been tested using the public drowsy driver dataset from ACCV 2016 competition. The results show that it can outperform the former schemes in the literature.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akopyan M, Khashba E (2017) Large-Scale YouTube-8M Video Understanding with Deep Neural Networks. arXiv preprint arXiv:170604488
Campos V, Jou B, Giró-i-Nieto X, Torres J, Chang S-F (2017) Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks. arXiv preprint arXiv:170806834
Gonzalez RC, Woods RE (2002) Digital image processing. Prentice Hall, New Jersey
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press
He K, Sun J (2015) Convolutional neural networks at constrained time cost. 5353–5360
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. 770–778
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift: 448–456
Ji S, Xu W, Yang M, Yu K (2013) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231
Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video classification with convolutional neural networks: 1725-1732
Khushaba RN, Kodagoda S, Lal S, Dissanayake G (2011) Driver drowsiness classification using fuzzy wavelet-packet-based feature-extraction algorithm. IEEE Trans Biomed Eng 58(1):121–131
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. : 1097–1105
Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint arXiv:13124400
Ma C-Y, Chen M-H, Kira Z, AlRegib G (2017) TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition. arXiv preprint arXiv:170310667
McCann MT, Jin KH, Unser M (2017) A review of convolutional neural networks for inverse problems in imaging. arXiv preprint arXiv:171004011
Miao Y, Li J, Wang Y, Zhang S-X, Gong Y (2016) Simplifying long short-term memory acoustic models for fast training and decoding. Acoust Speech Signal Process (ICASSP) 2016 IEEE Int Conf. IEEE: 2284–2288
Park S, Pan F, Kang S, Yoo CD (2016) Driver drowsiness detection system based on feature representation learning using various deep networks. Springer: 154–164
Sahayadhas A, Sundaraj K, Murugappan M (2012) Detecting driver drowsiness based on sensors: a review. Sensors 12(12):16937–16953
Shih T-H, Hsu C-T (2016) MSTN: Multistage spatial-temporal network for driver drowsiness detection. Springer, 146–153
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556
Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions: 1–9
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision: 2818–2826
Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. IEEE: I–I
Weng C-H, Lai Y-H, Lai S-H (2016) Driver drowsiness detection via a hierarchical temporal deep belief network. Springer: 117–133
Wu D, Lawhern VJ, Gordon S, Lance BJ, Lin C-T (2016) Driver drowsiness estimation from EEG signals using online weighted adaptation regularization for regression (OwARR). IEEE Trans Fuzzy Syst
Yu J, Park S, Lee S, Jeon M (2016) Representation learning, scene understanding, and feature fusion for drowsiness detection. Springer: 165–177
Zhang W, Cheng B, Lin Y (2012) Driver drowsiness recognition based on computer vision technology. Tsinghua Sci Technol 17(3):354–362
Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503
Acknowledgements
The author would like to thank the NTHU Computer Vision Lab for providing driver drowsiness dataset.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Guo, JM., Markoni, H. Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimed Tools Appl 78, 29059–29087 (2019). https://doi.org/10.1007/s11042-018-6378-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-018-6378-6