Abstract
The use of face video information for driver fatigue detection has received extensive attention because of its low cost and non-invasiveness. However, the current vehicle-mounted embedded device has insufficient memory and limited computing power, which cannot complete the real-time detection of driver fatigue based on deep learning. Therefore, this paper designs a lightweight neural network model to solve this problem. The model includes object detection and fatigue detection. First, a lightweight object detection network is designed, which can quickly identify the opening and closing states of the driver’s eyes and mouth in the time series video. Secondly, the EYE-MOUTH (EM) driver fatigue detection model is designed, which encodes the driver’s eye and mouth opening and closing state, and calculates the driver’s PERCLOS (Percentage of Eyelid Closure over the Pupil) and FOM (Frequency of Open Mouth) according to the coding sequence. Finally, the multi-feature fusion judgment algorithm is used to realize the judgment of the driver’s fatigue state. The experimental results show that our method has an accuracy rate of 98.30% for drowsiness and yawning behaviors in a real vehicle environment, and a detection speed of 27FPS, which is better than other advanced methods and meets the requirements of real-time detection.
Similar content being viewed by others
References
Anund A, Fors C, Ahlstrom C (2017) The severity of driver fatigue in terms of line crossing: a pilot study comparing day-and night time driving in simulator. Eur Transp Res Rev 9(31). https://doi.org/10.1007/s12544-017-0248-6
Azim T, Jaffar M A, Mirza A M (2009) Automatic fatigue detection of drivers through pupil detection and yawning analysis, Int Conf Innov Comput, Inf Control (ICICIC), 441-445
Bochkovskiy A, Wang C Y, Liao H Y M (2020) YOLOv4: optimal speed and accuracy of object detection, arXiv preprint arXiv:2004.10934
Bosso A, Conficoni C, Raggini D (2020) A computational-effective field-oriented control strategy for accurate and efficient electric propulsion of unmanned aerial vehicles. IEEE/ASME Trans Mechatron 99:1–1. https://doi.org/10.1109/TMECH.2020.3022379
Cashman D, Patterson G, Mosca A (2018) Rnnbow: visualizing learning via backpropagation gradients in rnns. IEEE Comput Graph Appl 38(6):39–50
Dwivedi K, Biswaranjan K, Sethi A (2014) Drowsy driver detection using representation learning, IEEE Int Adv Comput Conf (IACC), 995-999
Feldman D, Schmidt M, Sohler C (2020) Turning big data into tiny data: constant-size coresets for k-means, PCA, and projective clustering. SIAM J Comput 49(3):601–657. https://doi.org/10.1137/18M1209854
Flores MJ, Armingol JM, de la Escalera A (2010) Real-time warning system for driver drowsiness detection using visual information. J Intell Robot Syst 59(2):103–125
Geng LF, Yuan ZT, Xiao ZT (2018) Driver fatigue detection method based on facial behavior analysis. Comput Eng 44(1):274–279
Girshick R (2015) Fast r-cnn. Proceedings of the IEEE international conference on computer vision (ICCV), pp 1440–1448. https://doi.org/10.1109/ICCV.2015.169
Girshick R, Donahue J, Darrell T (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. IEEE Conf Comput Vis Pattern Recognition:580–587
Gu WH, Zhu Y, Chen XD (2018) Hierarchical CNN-based real-time fatigue detection system by visual-based technologies using MSP model. IET Image Process 12(12):2319–2329
He K (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):1904–1916
Huang J, Lin Z (1651) Multi-feature fatigue driving detection based on computer vision. J Phys Conf Ser 2020(1):012188
Huang R, Pedoeem J, Chen C (2018) YOLO-LITE: a real-time object detection algorithm optimized for non-GPU computers, IEEE international conference on big data. pp 2503-2510. https://doi.org/10.1109/BigData.2018.8621865
Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning 37:338–356
Ivanov Y (2015) Adaptive moving object segmentation algorithms in cluttered environments, The Exp Design Appl CAD Syst Microelectron, 97–99
Khunpisuth O, Chotchinasri T, Koschakosai V (2016) Driver drowsiness detection using eye-closeness detection, international conference on signal-image Technology & Internet-Based Systems, 661-668
Kim S, Wisanggeni I, Ros R (2020) Detecting fatigue driving through PERCLOS: a review. Int J Image Process (IJIP) 14(1):1
Koh S, Cho B R, Lee J (2017) Driver drowsiness detection via PPG biosignals by using multimodal head support, 2017 4th international conference on control, decision and information technologies (CoDIT), pp 0383–0388. https://doi.org/10.1109/CoDIT.2017.8102622
Li K, Gong Y, Ren Z (2020) A fatigue driving detection algorithm based on facial multi-feature fusion. IEEE Access 8:101244–101259. https://doi.org/10.1109/ACCESS.2020.2998363
Liu W, Anguelov D, Erhan D (2016) Ssd: single shot multibox detector. Eur Conf Comput Vis Springer:21–37
LIU Z, LUO P, WANG X (2015) Deep learning face attributes in the wild. IEEE Comput Soc Conf Comput Vis Pattern Recog:3730–3738
Lu X, Ma C, Ni B (2018) Deep regression tracking with shrinkage loss. Proceedings of the European Conference on Computer Vision (ECCV), pp 353–369). https://doi.org/10.1007/978-3-030-01264-9_22
Lv Z, Qiao L (2020) Deep belief network and linear perceptron based cognitive computing for collaborative robots Appl Soft Comput, 92
Lv Z, Qiao L, Li J (2020) Deep learning enabled security issues in the internet of things. IEEE Internet Things J 99:1–1
Lv Z, Zhang S, Xiu W (2020) Solving the security problem of intelligent transportation system with deep learning. IEEE Trans Intell Transp Syst 99:1–10. https://doi.org/10.1109/TITS.2020.2980864
Ma N, Zhang X, Zheng H T (2018) Shufflenet v2: practical guidelines for efficient cnn architecture design, proceedings of the European conference on computer vision (ECCV). https://doi.org/10.1007/978-3-030-01264-9_8
Mao Q, Sun H, Liu Y, Jia R (2019) Mini-YOLOv3: real-time object detector for embedded applications. IEEE Access 7:133529–133538
Navastara DA, Putra WYM, Fatichah C (2020) Drowsiness detection based on facial landmark and uniform local binary pattern. J Phys Conf Ser 1529(5):052015. https://doi.org/10.1088/1742-6596/1529/5/052015
Parekh V, Shah D, Shah M (2020) Fatigue detection using artificial intelligence framework. Augmented Hum Res 5(1):1–17. https://doi.org/10.1007/s41133-019-0023-4
Peleshko D, Ivanov Y, Sharov B, Izonin I, Borzov Y (2016) Design and implementation of visitors queue density analysis and registration method for retail video surveillance purposes. IEEE First Int Conf Data Stream Min Process (DSMP) 2016:159–162
Ravi A, Phanigna T R, Lenina Y (2020) Real time driver fatigue detection and smart rescue system, international conference on electronics and sustainable communication systems (ICESC), pp 434-439. https://doi.org/10.1109/ICESC48915.2020.9156021
Redmon J, Divvala S, Girshick R (2016) You only look once: unified, real-time object detection. IEEE Conf Comput Vis Pattern Recognition (CVPR), pp 779–788. https://doi.org/10.1109/CVPR.2016.91
Redmon J, Farhadi A (2017) YOLO9000: better, faster, stronger, IEEE Conf Comput Vis Pattern Recog, 7263–7271
Redmon J, Farhadi A (2018) Yolov3: An incremental improvement, 1-6. [online]. Available: https://pjreddie.com/media/files/papers/YOLOv3.pdf
Ren S, He K, Girshick R (2016) Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149
Sandler M, Howard A, Zhu M (2018) Mobilenetv2: inverted residuals and linear bottlenecks, IEEE Conf Comput Vis Pattern Recog, 4510-4520
Taigman Y, Yang M, Ranzato M A (2014) Deepface: closing the gap to human-level performance in face verification, IEEE Conf Comput Vis Pattern Recog, 1701-1708
Tkachenko R, Tkachenko P, Izonin I (2018) Learning-based image scaling using neural-like structure of geometric transformation paradigm. Adv Soft Comput Mach Learn Image Process, 537–565. https://doi.org/10.1007/978-3-319-63754-9_25
Wong A, Famuori M, Shafiee M J (2019) YOLO Nano: a Highly Compact You Only Look Once Convolutional Neural Network for Object Detection, arXiv preprint arXiv:1910.01271, 2019
Xu H, Zhou X, Xue C (2020) Fatigue measurement of task: based on multiple eye-tracking parameters and task performance, international conference on intelligent human systems integration, pp 1263-1269. https://doi.org/10.1007/978-3-030-39512-4_193
Yadav N, Banerjee K, Bali V (2020) A survey on fatigue detection of workers using machine learning. Int J E-Health Med Commun (IJEHMC) 11(3):1–8
Zhang H, Li Y, Lv Z, Sangaiah AK, Huang T (2020) A real-time and ubiquitous network attack detection based on deep belief network and support vector machine. IEEE/CAA J Autom Sin 7(3):790–799. https://doi.org/10.1109/JAS.2020.1003099
Zhang W, Murphey Y L, Wang T (2015) Driver yawning detection based on deep convolutional neural learning and robust nose tracking, Int Joint Conf Neural Netw (IJCNN), 1-8
Zhang P, Zhong Y, Li X (2019) SlimYOLOv3: narrower, faster and better for real-time UAV applications. IEEE international conference on computer vision (ICCV). https://doi.org/10.1109/ICCVW.2019.00011
Zhou Z, Cai Y, Ke R, Yang J (2017) A collision avoidance model for twopedestrian groups: considering random avoidance patterns. Physica A: Stat Mech Appl 475:142–154. https://doi.org/10.1016/j.physa.2016.12.041
Zhou Z, Zhou Y, Pu Z (2019) Simulation of pedestrian behavior during the flashing green signal using a modified social force model. Transportmetrica A: Transport Sci 15(2):1019–1040. https://doi.org/10.1080/23249935.2018.1559895
Acknowledgements
The authors are grateful for collaborative funding support from the Natural Science Foundation of Shandong Province, China (ZR2018MEE008), the Key Research and Development Project of Shandong Province, China (2019JZZY020326).
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Cui, Z., Sun, HM., Yin, RN. et al. Real-time detection method of driver fatigue state based on deep learning of face video. Multimed Tools Appl 80, 25495–25515 (2021). https://doi.org/10.1007/s11042-021-10930-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-021-10930-z