Skip to main content

Advertisement

SpringerLink
Log in

Driver fatigue detection and human-machine cooperative decision-making for road scenarios

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Driver fatigue detection and human-machine cooperative decision-making are key issues in intelligent transportation systems. The proposed method consists of two main stages.In the first stage, we propose an improved DenseNet-based method for driver fatigue detection. We propose a model representation enhancement module to improve the model’s adaptability to multi-scale features. Additionally, we propose an improved channel attention mechanism to enhance channel dependencies and spatial encoding capabilities. In the second stage, we propose a human-machine collaborative decision-making method based on guided policy search (GPS). Our method utilizes a reinforcement learning algorithm based on actor-critic and design an improved actor network based on the iLQR-based GPS. Finally, we propose an adaptive module that enables the vehicle to make decisions based on the driver’s fatigue state. The effectiveness of the proposed method is demonstrated through experimental results and comparisons.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

Data Availibility

The yawning video data that support the findings of this study are openly available in YaWDD at https://doi.org/10.21227/e1qm-hb90. The public real-life video data that support the findings of this study are openly available from Vision-Learning-Mining Lab, University of Texas at Arlington at sites.google.com/view/utarldd/home. The multimodality drowsiness data that support the findings of this study are openly available in DROZY at http://www.drozy.ulg.ac.be.

References

  1. Alioua N, Amine A, Rziza M (2014) Driver’s fatigue detection based on yawning extraction. International Journal of Vehicular Technology 2014:23–30

    Article  Google Scholar 

  2. Alioua N, Amine A, Rziza M (2014) Driver’s fatigue detection based on yawning extraction. International Journal of Vehicular Technology 2014:23–30

    Article  Google Scholar 

  3. Chen J, Wang Z, Tomizuka M (2018) Deep hierarchical reinforcement learning for autonomous driving with distinct behaviors. In: 2018 IEEE intelligent vehicles symposium (IV). IEEE, pp 1239–1244

  4. Choi I-H, Kim Y-G (2014) Head pose and gaze direction tracking for detecting a drowsy driver. In: 2014 international conference on big data and smart computing (BIGCOMP). IEEE, pp 241–244

  5. Choi I-H, Kim Y-G (2014) Head pose and gaze direction tracking for detecting a drowsy driver. In: 2014 international conference on big data and smart computing (BIGCOMP). IEEE, pp 241–244

  6. Ghourabi A, Ghazouani H, Barhoumi W (2020) Driver drowsiness detection based on joint monitoring of yawning, blinking and nodding. In: 2020 IEEE 16th international conference on intelligent computer communication and processing (ICCP). IEEE, pp 407–414

  7. Guo J-M, Markoni H (2019) Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimedia Tools and Applications 78(20):29059–29087

    Article  Google Scholar 

  8. Guo W, Di C, Long L (2021) Research on fatigue detection method of equipment operators based on multi-source physiological signals. Journal of Physics: Conference Series 1982(1):012067

    Google Scholar 

  9. Kendall A, Hawke J, Janz D, Mazur P, Reda D, Allen J-M, Lam V-D, Bewley A, Shah A (2019) Learning to drive in a day. In: 2019 International conference on robotics and automation (ICRA). pp 8248–8254

  10. Kendall A, Hawke J, Janz D, Mazur P, Reda D, Allen J-M, Lam V-D, Bewley A, Shah A (2019)Learning to drive in a day. In: 2019 international conference on robotics and automation (ICRA), pp 8248–8254

  11. Li G, Chung W-Y (2013) Detection of driver drowsiness using wavelet analysis of heart rate variability and a support vector machine classifier. Sensors 13(12):16494–16511

    Article  Google Scholar 

  12. Li Z, Li SE, Li R, Cheng B, Shi J (2017) Online detection of driver fatigue using steering wheel angles for real driving conditions. Sensors 17(3):495

    Article  Google Scholar 

  13. Li Z, Li SE, Li R, Cheng B, Shi J (2017) Online detection of driver fatigue using steering wheel angles for real driving conditions. Sensors 17(3):495

    Article  Google Scholar 

  14. Li Y, Tee KP, Yan R, Ge SS (2019) Reinforcement learning for human-robot shared control. Assembly Automation

  15. Mandal B, Li L, Wang GS, Lin J (2016) Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Transactions on Intelligent Transportation Systems 18(3):545–557

    Article  Google Scholar 

  16. Mandal B, Li L, Wang GS, Lin J (2016) Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Transactions on Intelligent Transportation Systems 18(3):545–557

    Article  Google Scholar 

  17. Mania H, Guy A, Recht B (2018) Simple random search of static linear policies is competitive for reinforcement learning. Advances in Neural Information Processing Systems 31:2746–2754

    Google Scholar 

  18. Mania H, Guy A, Recht B (2018) Simple random search of static linear policies is competitive for reinforcement learning. Advances in Neural Information Processing Systems 31:2746–2754

    Google Scholar 

  19. Omidyeganeh M, Shirmohammadi S, Abtahi S, Khurshid A, Farhan M, Scharcanski J, Hariri B, Laroche D, Martel L (2016) Yawning detection using embedded smart cameras. IEEE Transactions on Instrumentation and Measurement 65(3):570–582

    Article  Google Scholar 

  20. Peng K, Fei J, Yang K (2022) MASS: Multi-attentional semantic segmentation of LiDAR data for dense top-view understanding. In: 2022 IEEE transactions on intelligent transportation systems. IEEE, pp 15824–15840

  21. Pomerleau DA (1988) Alvinn: An autonomous land vehicle in a neural network. Advances in Neural Information Processing Systems 1:305–313

    Google Scholar 

  22. Pomerleau DA (1988) Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems 1:305–313

    Google Scholar 

  23. Shah S, Dey D, Lovett C, Kapoor A (2018) Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In: Field and service robotics. Springer, pp 621–635

  24. Todorov E, Erez T, Tassa Y (2012) Mujoco: A physics engine for model-based control. In: 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 5026–5033

  25. Todorov E, Erez T, Tassa Y (2012) Mujoco: A physics engine for model-based control. In: 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 5026–5033

  26. Vu TH, Dang A, Wang J-C (2019) A deep neural network for real-time driver drowsiness detection. IEICE Transactions on Information and Systems 102(12):2637–2641

    Article  Google Scholar 

  27. Wolf P, Hubschneider C, Weber M, Bauer A, Härtl J, Dürr F, Zöllner JM (2017) Learning how to drive in a real world simulation with deep q-networks. In: 2017 IEEE intelligent vehicles symposium (IV). pp 244–250

  28. Wu J, Huang Z, Lv C (2021) Uncertainty-aware model-based reinforcement learning with application to autonomous driving. arXiv:2106.12194

  29. Xie Y, Chen K, Murphey YL (2018) Real-time and robust driver yawning detection with deep neural networks. In: 2018 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, pp 532–538

  30. Yang H, Liu L, Min W, Yang X, Xiong X (2020) Driver yawning detection based on subtle facial action recognition. IEEE Transactions on Multimedia 23:572–583

    Article  Google Scholar 

  31. Zhao L, Wang Z, Zhang G, Gao H (2020) Driver drowsiness recognition via transferred deep 3d convolutional network and state probability vector. Multimedia Tools and Applications 79(35):26683–26701

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by Key Research and Development Plan of Shaanxi Province (China) under grant no. 2022GY-080.

Author information

Authors and Affiliations

  1. School of Software Engineering, Xi’an Jiaotong University, Xi’an, China

    Anna Li, Xinnan Ma, Jiaxin Guo, Jingyue Zhang, Jing Wang, Kai Zhao & Yaochen Li

Authors
  1. Anna Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinnan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiaxin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jingyue Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kai Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yaochen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yaochen Li.

Ethics declarations

Conflicts of interest

No conflict of interest exits in the submission of this manuscript, and the manuscript is approved by all the authors for publication. The work described is our original research that has not been published previously, and not under consideration for publication elsewhere.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, A., Ma, X., Guo, J. et al. Driver fatigue detection and human-machine cooperative decision-making for road scenarios. Multimed Tools Appl 83, 12487–12518 (2024). https://doi.org/10.1007/s11042-023-15994-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15994-7

Keywords

Search

Navigation