Skip to main content

Advertisement

SpringerLink
Log in

A novel approach for driver fatigue detection based on visual characteristics analysis

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Driver drowsiness is the major cause of many traffic accidents. A study by the National Institute of Sleep and Vigilance showed that the majority of the accidents took place on fast tracks. 34% of these accidents are mainly due to lack of sleep. Indeed, the drowsiness of drivers appears mainly by a fall of vigilance with the appearance of various behaviors that represent clues for the decrease of reflexes like the presence of yawning, heaviness of the eyelids and the difficulty to keep the head in frontal position compared to the vision field. In this context, we are interested particularly in the proposal of new solutions allowing the automatic control of the driver drowsiness states. These solutions need to be non-invasive while being based only on the analysis of the visual indices. These indices are calculated starting from the spatiotemporal information of the contents of a video stream. The main goal of our work is to recognize driver abnormal behavior by analyzing the characteristics of the face. In fact, we have proposed a fusion system based on detection of yawn, detection of somnolence and the 3D head pose estimation. This fusion system is evaluated by the three databases and shows many success of our suggested approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34

Similar content being viewed by others

References

  • Abtahi S, Hariri B, Shirmohammadi S (2011) Driver drowsiness monitoring based on yawning detection. In: Instrumentation and Measurement Technology Conference (I2MTC), 2011 IEEE, pp 1–4. https://doi.org/10.1109/IMTC.2011.5944101

  • Abtahi S, Omidyeganeh M, Shirmohammadi S, Hariri B (2014) Yawdd: a yawning detection dataset. In: Proceedings of the 5th ACM Multimedia Systems Conference, ACM, New York, NY, USA, MMSys ’14, pp 24–28. https://doi.org/10.1145/2557642.2563678

  • Adrian G, Mathis J, Herrmann U, Hess C (2007) The functional relationship between yawning and vigilance. Behavioural Brain Research 179(1):159–166. https://doi.org/10.1016/j.bbr.2007.01.027. URL http://www.sciencedirect.com/science/article/pii/S0166432807000745

  • Akrout B, Mahdi W (2013a) A blinking measurement method for driver drowsiness detection. In: The 8 International Conference on Computer Recognition Systems (CORES 2013), Milkow, Poland

  • Akrout B, Mahdi W (2013b) Hypovigilance detection based on eyelids behavior study. iJES 1(1):39–45. http://online-journals.org/index.php/i-jes/article/view/2927

  • Akrout B, Mahdi W (2013c) Vision based approach for driver drowsiness detection based on 3d head orientation. In: The 7th FTRA International Conference on Multimedia and Ubiquitous Engineering (MUE 2013), Seoul, Korea

  • Akrout B, Mahdi W (2013d) A visual based approach for drowsiness detection. In: Intelligent Vehicles Symposium (IV), 2013 IEEE, pp 1324–1329. https://doi.org/10.1109/IVS.2013.6629650

  • Akrout B, Mahdi W (2014) Hypovigilance detection by calculating and analyzing a spatio-temporal features of the face components based on 3d head orientation. In: Advanced Technologies for Signal and Image Processing (ATSIP), 2014 1st International Conference on, pp 137–142. https://doi.org/10.1109/ATSIP.2014.6834593

  • Akrout B, Mahdi W (2015) Spatio-temporal features for the automatic control of driver drowsiness state and lack of concentration. Mach Vis Appl 26(1):1–13. https://doi.org/10.1007/s00138-014-0644-z

    Article  Google Scholar 

  • Akrout B, Mahdi W, Hamadou A (2013) Drowsiness detection based on video analysis approach. In: The 8th International Conference on Computer Vision Theory and Applications (VISAPP 2013), Bacelona, Spain

  • Allen J, Xu R, Jin J (2004) Object tracking using camshift algorithm and multiple quantized feature spaces. In: Proceedings of the Pan-Sydney area workshop on Visual information processing, Australian Computer Society, Inc., Darlinghurst, Australia, Australia, VIP ’05, pp 3–7. http://dl.acm.org/citation.cfm?id=1082121.1082122

  • Antoine P, Sylvie C, Caplier A, Ngoc-Son V (2012) Using retina modelling to characterize blinking: comparison between eog and video analysis. Mach Vis Appl 23(6):1195–1208

    Article  Google Scholar 

  • Arnin J, Anopas D, Horapong M, Triponyuwasi P, Yamsa-ard T, Iampetch S, Wongsawat Y (2013) Wireless-based portable eeg-eog monitoring for real time drowsiness detection. In: Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE, pp 4977–4980. https://doi.org/10.1109/EMBC.2013.6610665

  • Awais M, Badruddin N, Drieberg M (2017) Eeg brain connectivity analysis to detect driver drowsiness using coherence. In: 2017 International Conference on Frontiers of Information Technology (FIT), pp 110–114. https://doi.org/10.1109/FIT.2017.00027

  • Bauer N, Pathirana P, Hodgson P (2006) Robust optical flow with combined lucas-kanade/horn-schunck and automatic neighborhood selection. In: Information and Automation, 2006. ICIA 2006. International Conference on, pp 378–383. https://doi.org/10.1109/ICINFA.2006.374108

  • Benoit A, Caplier A (2005) Hypovigilence analysis: open or closed eye or mouth? blinking or yawning frequency? In: IEEE Conference on Advanced Video and Signal Based Surveillance, 2005., pp 207–212. https://doi.org/10.1109/AVSS.2005.1577268

  • Bidgoli NM, Raie AA, Naraghi M (2015) Probabilistic principal component analysis for texture modelling of adaptive active appearance models and its application for head pose estimation. IET Comput Vision 9(1):51–62. https://doi.org/10.1049/iet-cvi.2013.0317

    Article  Google Scholar 

  • Bottou L, Cortes C, Denker J (1994) Comparison of classifier methods: A case study in handwritten digit recognition. In: Proceedings of the 12th International Conference on Pattern Recognition, Los Alamitos, CA., vol 2, pp 77–82

  • Chen Y, Fu M, Yang Y, Song W (2014) A method of head pose estimation based on active shape model and stereo vision. In: Control Conference (CCC), 2014 33rd Chinese, pp 8277–8282. https://doi.org/10.1109/ChiCC.2014.6896387

  • Chieh T, Mustafa M, Hussain A, Hendi S, Majlis B (2005) Development of vehicle driver drowsiness detection system using electrooculogram (eog). In: Computers, Communications, Signal Processing with Special Track on Biomedical Engineering, 2005. CCSP 2005. 1st International Conference on, pp 165–168. https://doi.org/10.1109/CCSP.2005.4977181

  • Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24(5):603–619. https://doi.org/10.1109/34.1000236

    Article  Google Scholar 

  • Diz F, Buron A, Solana J (2007) Haar wavelet based processor scheme for image coding with low circuit complexity. Pergamon Press Inc., Tarrytown, NY, USA, vol 33, pp 109–126. https://doi.org/10.1016/j.compeleceng.2006.08.001

  • Fatima B, Shahid AR, Ziauddin S, Safi AA, Ramzan H (2020) Driver fatigue detection using viola jones and principal component analysis. Appl Artif Intell 34(6):456–483. https://doi.org/10.1080/08839514.2020.1723875

    Article  Google Scholar 

  • Hiroshi U, Kaneda M, Tsukino M (1994) Development of drowsiness detection system. In: Vehicle Navigation and Information Systems Conference, 1994. Proceedings., 1994, pp 15–20. https://doi.org/10.1109/VNIS.1994.396873

  • Hongbiao M, Zehong Y, Yixu S, Peifa J (2008) A fast method for monitoring driver fatigue using monocular camera. In: Proceedings of the 11th Joint Conference on Information Sciences, Atlantis Press

  • Horn B, Schunck B (1981) Determining optical flow. Artificial Intelligence 17(1–3):185–203. https://doi.org/10.1016/0004-3702(81)90024-2. http://www.sciencedirect.com/science/article/pii/0004370281900242

  • Horng W, Chih-Yuan C, Chang Y, Chun-Hai F (2004) Driver fatigue detection based on eye tracking and dynamk template matching. In: Networking, Sensing and Control, 2004 IEEE International Conference on, pp 7 – 12

  • Horprasert T, Yacoob Y, Davis L (1996) Computing 3-d head orientation from a monocular image sequence. In: Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition (FG ’96), IEEE Computer Society, Washington, DC, USA, FG ’96, pp 242. http://dl.acm.org/citation.cfm?id=524467.796020

  • Huan X, Constantine C, Shie M (2009) Robustness and regularization of support vector machines. J Mach Learn Res 10:1485–1510

    MathSciNet  MATH  Google Scholar 

  • Huang N, Shen Z, Long S, Wu M, Shih H, Zheng Q, Yen N, Tung C, Liu H (1998) The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis. The Royal Society, vol 454, pp 903–995. http://rspa.royalsocietypublishing.org/cgi/doi/10.1098/rspa.1998.0193

  • Iole Z, Fiorenza G, VALERIA U (2008) Yawning and subjective sleepiness in the elderly. J Sleep Res 17(3):303–308. https://doi.org/10.1111/j.1365-2869.2008.00666.x

    Article  Google Scholar 

  • Jean C, Fabrice L (2009) Key figures for road safety. In: National Inter-Departmental Observatory on Road Safety, France

  • Junli X, Jianliang M, Jianfeng H (2018) Real-time eye tracking for the assessment of driver fatigue. Healthcare Technology Letters 5(2):54–58. https://doi.org/10.1049/htl.2017.0020

    Article  Google Scholar 

  • Kecman V (2001) Learning and soft computing: Support vector machines, neural networks, and fuzzy logic models

  • Koelstra S, Muhl C, Soleymani M, Jong-Seok L, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) Deap: A database for emotion analysis; using physiological signals. Affective Computing, IEEE Transactions on 3(1):18–31. https://doi.org/10.1109/T-AFFC.2011.15

    Article  Google Scholar 

  • Krinidis M, Nikolaidis N, Pitas I (2007) 3d head pose estimation using support vector machines and physics-based deformable surfaces. In: Signal Processing and Its Applications, 2007. ISSPA 2007. 9th International Symposium on, pp 1–4. https://doi.org/10.1109/ISSPA.2007.4555497

  • Li Y, Gong S, Sherrah J, Liddell H (2000) Multi-view face detection using support vector machines and eigenspace modelling. In: Knowledge-Based Intelligent Engineering Systems and Allied Technologies, 2000. Proceedings. Fourth International Conference on, vol 1, pp 241–244. https://doi.org/10.1109/KES.2000.885802

  • Maryam H, Alireza M, Aliasghar S (2020) Driver safety development: real-time driver drowsiness detection system based on convolutional neural network. SN Comput Sci 1:1–10. https://doi.org/10.1007/s42979-020-00306-9

    Article  Google Scholar 

  • Murray J, Andrew T, Robert C, Kate C, Natalie M (2007) Monitoring eye and eyelid movements by infrared reflectance oculography to measure drowsiness in drivers. In: Somnologie - Schlafforschung und Schlafmedizin, Steinkopff-Verlag, pp 234–242

  • Muzet A, Pebayle T, Langrognet J, Otmani S (2003) Awake pilot study no.2 : Testing steering grip sensor measures. Technical Report IST-2000-28062

  • Narayanan A, Kaimal RM, Bijlani K (2016) Estimation of driver head yaw angle using a generic geometric model. IEEE Trans Intell Transp Syst PP(99):1–15. https://doi.org/10.1109/TITS.2016.2551298

  • Qiang J, Yang X (2002) Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real-Time Imaging 8(5):357–377. https://doi.org/10.1006/rtim.2002.0279

    Article  MATH  Google Scholar 

  • Qiang J, Zhiwei Z, Peilin L (2004) Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans Veh Technol 53(4):1052–1068. https://doi.org/10.1109/TVT.2004.830974

    Article  Google Scholar 

  • Ribeiro D, Cardoso A, Teixeira C (2017) Online demonstration of a eeg-based drowsiness detector. In: 2017 4th Experiment@International Conference (exp.at’17), pp 93–94. https://doi.org/10.1109/EXPAT.2017.7984342

  • Robert R (1985) Yawning as a stereotyped action pattern and releasing stimulus. Ethology: Formerly Zeitschrift fur Tierpsychologie 72:109–122. https://doi.org/10.1111/j.1439-0310.1986.tb00611.x

    Article  Google Scholar 

  • Sarbjit S, Papanikolopoulos NP (1999) Monitoring driver fatigue using facial analysis techniques. In: Intelligent Transportation Systems, 1999. Proceedings. 1999 IEEE/IEEJ/JSAI International Conference on, pp 314–318. https://doi.org/10.1109/ITSC.1999.821073

  • Scholkopf B, Smola A (2001) Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT Press, Cambridge, MA, USA

    Google Scholar 

  • Shabnam A, Behnoosh H, Shervin S (2011) Driver drowsiness monitoring based on yawning detection. In: IEEE Instrumentation and Measurement Technology Conference, pp 1–4. https://doi.org/10.1109/IMTC.2011.5944101

  • Smereka M, Dulkeba I (2008) Circular object detection using a modified hough transform. Versita, Warsaw, Poland, Poland 18:85–91. https://doi.org/10.2478/v10006-008-0008-9

    Article  MathSciNet  Google Scholar 

  • Storer M, Urschler M, Bischof H (2009) 3d-mam: 3d morphable appearance model for efficient fine head pose estimation from still images. In: Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12th International Conference on, pp 192–199. https://doi.org/10.1109/ICCVW.2009.5457701

  • Tnkehiro I, Shinji M, Kduo K, Tomoaki N, Shin Y (2002) Driver blink measurement by the motion picture processing and its application to drowsiness detection. In: The IEEE 5th International Conference on Intelligent Transportation Systems, IEEE Conference Publications. http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=01041208

  • Vicente F, Huang Z, Xiong X, la Torre FD, Zhang W, Levi D (2015) Driver gaze tracking and eyes off the road detection system. IEEE Trans Intell Transp Syst 16(4):2014–2027. https://doi.org/10.1109/TITS.2015.2396031

    Article  Google Scholar 

  • Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: CVPR (1), pp 511–518

  • Wang K, Ji Q (2017) Real time eye gaze tracking with 3d deformable eye-face model. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 1003–1011. https://doi.org/10.1109/ICCV.2017.114

  • Xiao F, Bao-Cai Y, Yan-Feng S (2007) Yawning detection for monitoring driver fatigue. In: 2007 International Conference on Machine Learning and Cybernetics, vol 2, pp 664–668. https://doi.org/10.1109/ICMLC.2007.4370228

  • Yin L, Basu A (1997) Mpeg4 face modeling using fiducial points. In: ICIP (1), pp 109–112

  • Yong D, Peijun M, Xiaohong S, Yingjun Z (2008) Driver fatigue detection based on eye state analysis. In: Proceedings of the 11th Joint Conference on Information Sciences, Atlantis Press

  • Zeng S, Li J, Jiang L, Jiang J (2017) A driving assistant safety method based on human eye fatigue detection. In: 2017 29th Chinese Control And Decision Conference (CCDC), pp 6370–6377. https://doi.org/10.1109/CCDC.2017.7978318

  • Zhang C, Wang H, Fu R (2014) Automated detection of driver fatigue based on entropy and complexity measures. IEEE Trans Intell Transp Syst 15(1):168–177. https://doi.org/10.1109/TITS.2013.2275192

    Article  Google Scholar 

  • Zhang C, Wu X, Zheng X, Yu S (2019) Driver drowsiness detection using multi-channel second order blind identifications. IEEE Access 7:11829–11843. https://doi.org/10.1109/ACCESS.2019.2891971

    Article  Google Scholar 

  • Zhao Z, Zhou N, Zhang L, Yan H, Xu Y, Zhang Z (2020) Driver fatigue detection based on convolutional neural networks using em-cnn. Comput Intell Neurosci 2020:1–11. https://doi.org/10.1155/2020/7251280

    Article  Google Scholar 

  • Zhiwei Z, Qiang J, Peilin L (2004) Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans Veh Technol 53(4):1052–1068. https://doi.org/10.1109/TVT.2004.830974

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, College of Computer Engineering and Sciences, Prince Sattam Bin Abdulaziz University, Alkharj, 11942, Saudi Arabia

    Belhassen Akrout

  2. Higher Institute of Computer Science and Multimedia, University of Sfax, Sfax, Tunisia

    Walid Mahdi

  3. MIRACL Laboratory ISIMS Pole Technologique de Sfax BP 242-3021, Sfax university, Sakiet Ezzit Sfax, Tunisia

    Belhassen Akrout & Walid Mahdi

Authors
  1. Belhassen Akrout
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Walid Mahdi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Belhassen Akrout.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Akrout, B., Mahdi, W. A novel approach for driver fatigue detection based on visual characteristics analysis. J Ambient Intell Human Comput 14, 527–552 (2023). https://doi.org/10.1007/s12652-021-03311-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-021-03311-9

Keywords

Search

Navigation