Driver Drowsiness Detection via a Hierarchical Temporal Deep Belief Network

  • Ching-Hua Weng
  • Ying-Hsiu Lai
  • Shang-Hong LaiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10118)


Drowsy driver alert systems have been developed to minimize and prevent car accidents. Existing vision-based systems are usually restricted to using visual cues, depend on tedious parameter tuning, or cannot work under general conditions. One additional crucial issue is the lack of public datasets that can be used to evaluate the performance of different methods. In this paper, we introduce a novel hierarchical temporal Deep Belief Network (HTDBN) method for drowsy detection. Our scheme first extracts high-level facial and head feature representations and then use them to recognize drowsiness-related symptoms. Two continuous-hidden Markov models are constructed on top of the DBNs. These are used to model and capture the interactive relations between eyes, mouth and head motions. We also collect a large comprehensive dataset containing various ethnicities, genders, lighting conditions and driving scenarios in pursuit of wide variations of driver videos. Experimental results demonstrate the feasibility of the proposed HTDBN framework in detecting drowsiness based on different visual cues.


Support Vector Machine False Alarm Rate Dynamic Bayesian Network Restrict Boltzmann Machine Facial Landmark 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors would like to thank Qualcomm Technologies Inc. for supporting this research work.


  1. 1.
    Bergasa, L., Nuevo, J., Sotelo, M., Barea, R., Lopez, M.: Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Transp. Syst. 7, 63–77 (2006)CrossRefGoogle Scholar
  2. 2.
    World Health Organization: Global status report on road safety 2013: supporting a decade of action: summary. World Health Organization (2013)Google Scholar
  3. 3.
    Wheaton, G., Shults, R.: Drowsy driving and risk behaviors 10 states and Puerto Rico. Online article (2014)Google Scholar
  4. 4.
    National Sleep Foundation: Drowsy driving reduction act of 2015 (2014)Google Scholar
  5. 5.
    Colic, A., Marques, O., Furht, B.: Driver Drowsiness Detection: Systems and Solutions. Springer, Heidelberg (2014)Google Scholar
  6. 6.
    Mercedes-Benz: Attention assist: drowsiness-detection system warns drivers to prevent them falling asleep momentarily. Online article (2008)Google Scholar
  7. 7.
    Teyeb, I., Jemai, O., Zaied, M., Ben Amar, C.: A drowsy driver detection system based on a new method of head posture estimation. In: Corchado, E., Lozano, J.A., Quintián, H., Yin, H. (eds.) IDEAL 2014. LNCS, vol. 8669, pp. 362–369. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10840-7_44 Google Scholar
  8. 8.
    Qiang, J., Lan, P., Looney, C.: A probabilistic framework for modeling and real-time monitoring human fatigue. IEEE Trans. Syst. Man Cybern. Part A: Syst. Hum. 36, 862–875 (2006)CrossRefGoogle Scholar
  9. 9.
    Lucey, P., Cohn, J., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (2010)Google Scholar
  10. 10.
    Wu, D., Shao, L.: Leveraging hierarchical parametric networks for skeletal joints based action segmentation and recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 724–731 (2014)Google Scholar
  11. 11.
    Yang, G., Lin, Y., Bhattacharya, P.: A driver fatigue recognition model based on information fusion and dynamic Bayesian network. Inf. Sci. 180, 1942–1954 (2010)CrossRefGoogle Scholar
  12. 12.
    Ji, Q., Zhu, Z., Lan, P.: Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans. Veh. Technol. 53, 1052–1068 (2004)CrossRefGoogle Scholar
  13. 13.
    Mohamed, A., Dahl, G., Hinton, G.: Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Process. 20, 14–22 (2012)CrossRefGoogle Scholar
  14. 14.
    Dasgupta, A., George, A., Happy, S., Routray, A.: A vision-based system for monitoring the loss of attention in automotive drivers. IEEE Trans. Intell. Transp. Syst. 14, 1825–1838 (2013)CrossRefGoogle Scholar
  15. 15.
    Alioua, N., Amine, A., Rziza, M.: Drivers fatigue detection based on yawning extraction. Int. J. Veh. Technol. 2014 (2014)Google Scholar
  16. 16.
    Rezaei, M., Klette, R.: Look at the driver, look at the road: no distraction! No accident! In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 129–136 (2014)Google Scholar
  17. 17.
    Smith, P., Shah, M., da Vitoria Lobo, N.: Determining driver visual attention with one camera. IEEE Trans. Intell. Transp. Syst. 4, 205–218 (2003)CrossRefGoogle Scholar
  18. 18.
    Eskandarian, A., Sayed, R.: Analysis of driver impairment, fatigue, and drowsiness and an unobtrusive vehicle-based detection scheme. In: Proceeding of International Conference on Traffic Accidents (2005)Google Scholar
  19. 19.
    Taylor, G., Hinton, G., Roweis, S.: Modeling human motion using binary latent variables. In: Neural Information Processing Systems, pp. 1345–1352 (2006)Google Scholar
  20. 20.
    Hinton, G., Osindero, S.: A fast learning algorithm for deep belief nets. Neural Comput. 18 (2006)Google Scholar
  21. 21.
    Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human-level performance in face verification. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)Google Scholar
  22. 22.
    Xiong, X., de la Torre, F.: Supervised descent method and its application to face alignment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539 (2013)Google Scholar
  23. 23.
    DeMenthon, F., Davis, L.: Model-based object pose in 25 lines of code. Int. J. Comput. Vis. 15, 123–141 (1995)CrossRefGoogle Scholar
  24. 24.
    Heo, J., Savvides, M.: Gender and ethnicity specific generic elastic models from a single 2D image for novel 2D pose face synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2341–2350 (2012)CrossRefGoogle Scholar
  25. 25.
    Yang, X., Tian, Y.: Eigenjoints-based action recognition using naive-bayes-nearest-neighbor. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 14–19 (2012)Google Scholar
  26. 26.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates Inc, Red Hook (2012)Google Scholar
  27. 27.
    Hinton, G.E.: A practical guide to training restricted Boltzmann machines. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, 2nd edn, pp. 599–619. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-35289-8_32 CrossRefGoogle Scholar
  28. 28.
    Freund, Y., Haussler, D.: Unsupervised learning of distributions on binary vectors using two layer networks. Technical report, University of California at Santa Cruz, Santa Cruz, CA, USA (1994)Google Scholar
  29. 29.
    Yang, L., Widjaja, B., Prasad, R.: Application of hidden Markov models for signature verification. Pattern Recogn. 28, 161–170 (1995)CrossRefGoogle Scholar
  30. 30.
    Devijver, P.A.: Baum’s forward-backward algorithm revisited. Pattern Recogn. Lett. 3, 369–373 (1985)CrossRefzbMATHGoogle Scholar
  31. 31.
    Abtahi, S., Omidyeganeh, M., Shirmohammadi, S., Hariri, B.: YawDD: a yawning detection dataset. In: Proceedings of the 5th ACM Multimedia Systems Conference, pp. 24–28. ACM (2014)Google Scholar
  32. 32.
    Omidyeganeh, M., Shirmohammadi, S., Abtahi, S., Khurshid, A., Farhan, M., Scharcanski, J., Hariri, B., Laroche, D., Martel, L.: Yawning detection using embedded smart cameras. IEEE Trans. Instrum. Meas. 65, 570–582 (2016)CrossRefGoogle Scholar
  33. 33.
    Zhang, W., Murphey, Y.L., Wang, T., Xu, Q.: Driver yawning detection based on deep convolutional neural learning and robust nose tracking. In: 2015 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceNational Tsing Hua UniversityHsinchuTaiwan

Personalised recommendations