Skip to main content
SpringerLink
Log in

Training Deep Learning Sequence Models to Understand Driver Behavior

  • Chapter
  • First Online:
Deep Learning Applications, Volume 2

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1232))

  • 1033 Accesses

Abstract

Driver distraction is one of the leading causes of fatal car accidents in the U.S. Analyzing driver behavior using machine learning and deep learning models is an emerging solution to detect abnormal behavior and alarm the driver. Models with memory such as LSTM networks outperform memoryless models in car safety applications since driving is a continuous task and considering information in the sequence of driving data can increase the model’s performance. In this work, we used time-sequenced driving data that we collected in eight driving contexts to measure the driver distraction. Our model is also capable of detecting the type of behavior that caused distraction. We used the driver interaction with the car infotainment system as the distracting activity. A multilayer neural network (MLP) was used as the baseline and two types of LSTM networks including the LSTM model with attention network and the encoder–decoder model with attention were built and trained to analyze the effect of memory and attention on the computational expense and performance of the model. We compare the performance of these two complex networks to that of the MLP in estimating driver behavior. We show that our encoder–decoder with attention model outperforms the LSTM attention while using LSTM networks with attention enhanced training process of the MLP network.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. B. Darrow, Distracted driving is now an epidemic in the U.S., Fortune (2016). http://fortune.com/2016/09/14/distracted-driving-epidemic/

  2. National Center for Statistics and Analysis, Distracted driving in fatal crashes, 2017, (Traffic Safety Facts Research Note, Report No. DOT HS 812 700) (Washington, DC, National Highway Traffic Safety Administration, 2019)

    Google Scholar 

  3. N. Chaudhary, J. Connolly, J. Tison, M. Solomon, K. Elliott, Evaluation of the NHTSA distracted driving high-visibility enforcement demonstration projects in California and Delaware. (Report No. DOT HS 812 108) (Washington, DC, National Highway Traffic Safety Administration, 2015)

    Google Scholar 

  4. National Center for Statistics and Analysis, Distracted driving in fatal crashes, 2017, (Traffic safety facts research Note, Report No. DOT HS 812 700), (Washington, DC: National Highway Traffic Safety Administration, 2019)

    Google Scholar 

  5. S. Monjezi Kouchak, A. Gaffar, Driver distraction detection using deep neural network, in The Fifth International Conference on Machine Learning, Optimization, and Data Science (Siena, Tuscany, Italy, 2019)

    Google Scholar 

  6. J. Lee, Dynamics of driver distraction: the process of engaging and disengaging. Assoc. Adv. Autom. Med. 58, 24–35 (2014)

    Google Scholar 

  7. T. Hirayama, K. Mase, K. Takeda, Analysis of temporal relationships between eye gaze and peripheral vehicle behavior for detecting driver distraction. Hindawi Publ. Corp. Int. J. Veh. Technol. 2013, 8 (2013)

    Google Scholar 

  8. National Highway Traffic Safety Administration. Blueprint for Ending Distracted Driving. Washington, DC: U.S. Department of Transportation. National Highway Traffic Safety Administration, DOT HS 811 629 (2012)

    Google Scholar 

  9. T.B. Sheridan, R. Parasuraman, Human-automation interaction. reviews of human factors and ergonomics, vol. 1, pp. 89–129 (2015). https://doi.org/10.1518/155723405783703082

  10. U. Hamid, F. Zakuan, K. Zulkepli, M. ZulfaqarAzmi, H. Zamzuri, M. Rahman, M. Zakaria, Autonomous Emergency Braking System with Potential Field Risk Assessment for Frontal Collision Mitigation (IEEE ICSPC, Malaysia, 2017)

    Book  Google Scholar 

  11. L. Li, D. Wen, N. Zheng, L. Shen, Cognitive cars: a new frontier for ADAS research. IEEE Trans. Intell. Transp. Syst. 13 (2012)

    Google Scholar 

  12. S. Monjezi Kouchak, A. Gaffar, Estimating the driver status using long short term memory, in Machine Learning and Knowledge Extraction, Third IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2019 (2019). https://doi.org/10.1007/978-3-030-29726-8_5

  13. P. Koopman, M. Wagner, Autonomous vehicle safety: an interdisciplinary challenge. IEEE Intell. Transp. Syst. Mag. 9, 90–96 (2017)

    Article  Google Scholar 

  14. M. Benmimoun, A. Pütz, A. Zlocki, L. Eckstein, euroFOT: field operational test and impact assessment of advanced driver assistance systems: final results, in SAE-China, FISITA (eds) Proceedings of the FISITA 2012 World Automotive Congress. Lecture Notes in Electrical Engineering, vol. 197 (Springer, Berlin, Heidelberg, 2013)

    Google Scholar 

  15. S. Monjezi Kouchak, A. Gaffar, Determinism in future cars: why autonomous trucks are easier to design, in IEEE Advanced and Trusted Computing (ATC 2017) (San Francisco Bay Area, USA, 2017)

    Google Scholar 

  16. S. Kaplan, M.A. Guvensan, A.G. Yavuz, Y. Karalurt, Driver behavior analysis for safe driving: a survey, IEEE Trans. Intell. Transp. Syst. 16, 3017–3032 (2015)

    Google Scholar 

  17. A. Aksjonov, P. Nedoma, V. Vodovozov, E. Petlenkov, M. Herrmann, Detection and evaluation of driver distraction using machine learning and fuzzy logic. IEEE Trans. Intell. Transp. Syst. 1–12 (2018). https://doi.org/10.1109/tits.2018.2857222

  18. R. Harb, X. Yan, E. Radwan, X. Su, Exploring precrash maneuvers using classification trees and random forests, Accid. Anal. Prev. 41, 98–107 (2009)

    Google Scholar 

  19. A. Alvarez, F. Garcia, J. Naranjo, J. Anaya, F. Jimenez, Modeling the driving behavior of electric vehicles using smartphones and neural networks. IEEE Intell. Transp. Syst. Mag. 6, 44–53 (2014)

    Article  Google Scholar 

  20. J. Morton, T. Wheeler, M. Kochenderfer, Analysis of recurrent neural networks for probabilistic modeling of driver behavior. IEEE Trans. Intell. Transp. Syst. 18, 1289–1298 (2017)

    Article  Google Scholar 

  21. A. Sathyanarayana, P. Boyraz, J. Hansen, Driver behavior analysis and route recognition by Hidden Markov models, IEEE International Conference on Vehicular Electronics and Safety (2008)

    Google Scholar 

  22. J. Li, X. Mei, D. Prokhorov, D. Tao, Deep neural network for structural prediction and lane detection in traffic scene. IEEE Trans. Neural Netw. Learn. Syst. 28, 14 (2017)

    Google Scholar 

  23. S. Monjezi Kouchak, A. Gaffar, Non-intrusive distraction pattern detection using behavior triangulation method, in 4th Annual Conference on Computational Science and Computational Intelligence CSCI-ISAI (USA, 2017)

    Google Scholar 

  24. S. Su, B. Nugraha, Fahmizal, Towards self-driving car using convolutional neural network and road lane detector, in 2017 2nd International Conference on Automation, Cognitive Science, Optics, Micro Electro-Mechanical System, and Information Technology (ICACOMIT) (Jakarta, Indonesia, 2017), p. 5

    Google Scholar 

  25. S. Hung, I. Choi, Y. Kim, Real-time categorization of driver’s gaze zone using the deep learning techniques, in 2016 International Conference on Big Data and Smart Computing (BigComp) (2016), pp. 143–148

    Google Scholar 

  26. A. Koesdwiady, S. Bedavi, C. Ou, F. Karray, End-to-end deep learning for driver distraction recognition. Springer International Publishing AG 2017 (2017), p. 8

    Google Scholar 

  27. I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (The MIT press, 2016), ISBN: 9780262035613

    Google Scholar 

  28. J. Schmidhuber, Deep learning in neural networks: an overview, vol. 61, (Elsevier, 2015), pp. 85–117

    Google Scholar 

  29. M. Wöllmer, C. Blaschke, T. Schindl, B. Schuller, B. Färber, S. Mayer, B. Trefflich, Online driver distraction detection using long short-term memory. IEEE Trans. Intell. Transp. Syst. 2(2), 574–582 (2011)

    Article  Google Scholar 

  30. K. Xu, J.L. Bay, R. Kirosy, K. Cho, A. Courville, R. Salakhutdinovy, R.S. Zemely, Y. Bengio, Show, attend and tell: neural image caption generation with visual attention, in 32 nd International Conference on Machine Learning (Lille, France, 2015)

    Google Scholar 

  31. T. Xiao, Y. Xu, K. Yang, J. Zhang, Y. Peng, Z. Zhang, The application of two-level attention models in deep convolutional neural network for fine-grained image classification, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015), pp. 842–850

    Google Scholar 

  32. P. Huang, F. Liu, S. Shiang, J. Oh, C. Dyer, Attention-based multimodal neural machine translation, in Proceedings of the First Conference on Machine Translation, Shared Task Papers, vol. 2, (Berlin, Germany, 2016), pp. 639–645

    Google Scholar 

  33. Y. Lv, Y. Duan, W. Kang, Z. Li, F. Wang, Traffic flow prediction with big data: a deep learning approach. IEEE Trans. Intell. Transp. Syst. 16, 865–873 (2015)

    Google Scholar 

  34. K. Saleh, M. Hossny, S. Nahavandi, Driving behavior classification based on sensor data fusion using LSTM recurrent neural networks, in IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) (2017)

    Google Scholar 

  35. A. Gaffar, S. Monjezi Kouchak, Minimalist design: an optimized solution for intelligent interactive infotainment systems, in IEEE IntelliSys, the International Conference on Intelligent Systems and Artificial Intelligence (London, UK, 2017)

    Google Scholar 

  36. C. Bishop, Pattern Recognition and Machine Learning (Springer). ISBN-13: 978-0387310732

    Google Scholar 

  37. M. Magic, Action recognition using Python and recurrent neural network, First edn. (2019). ISBN: 978-1798429044

    Google Scholar 

  38. D. Mandic, J. Chambers, Recurrent neural networks for prediction: learning algorithms, architectures and stability, First edn. (Wiley, 2001). ISBN: 978-0471495178

    Google Scholar 

  39. J. Rogerson, Theory, Concepts and Methods of Recurrent Neural Networks and Soft Computing (2015). ISBN-13: 978-1632404930

    Google Scholar 

  40. M.A. Wani, F.A. Bhat, S. Afzal, A.I. Khan, Advances in Deep Learning (Springer, 2020)

    Google Scholar 

  41. A. Gaffar, E. M. Darwish, A. Tridane, Structuring heterogeneous big data for scalability and accuracy. Int. J. Digit. Inf. Wirel. Commun. 4, 10–23 (2014)

    Google Scholar 

  42. A. Gaffar, H. Javahery, A. Seffah, D. Sinnig, A pattern framework for eliciting and delivering UCD knowledge and practices, in Proceedings of the Tenth International Conference on Human-Computer Interaction (2003), pp. 108–112

    Google Scholar 

  43. A. Gaffar, Enumerating mobile enterprise complexity 21 complexity factors to enhance the design process, in Proceedings of the 2009 Conference of the Center for Advanced Studies on Collaborative Research (2009), pp. 270–282

    Google Scholar 

  44. A. Gaffar, The 7C’s: an iterative process for generating pattern components, in 11th International Conference on Human-Computer Interaction (2005)

    Google Scholar 

  45. J. Bermudez, Cognitive Science: An Introduction to the Science of the Mind, 2nd edn. (2014). 978-1107653351

    Google Scholar 

  46. B. Garrett, G. Hough, Brain & Behavior: an Introduction to Behavioral Neuroscience, 5th edn. (SAGE). ISBN: 978-1506349206

    Google Scholar 

  47. Y. Wang, M. Huang, L. Zhao, X. Zhu, Attention-based LSTM for aspect-level sentiment classification, in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016), pp. 606–615

    Google Scholar 

  48. I. Sutskever, O. Vinyals, Q. Le, Sequence to sequence learning with neural networks, in Advances in Neural Information Processing Systems 27 (NIPS 2014) (2014)

    Google Scholar 

  49. S. Frintrop, E. Rome, H. Christenson, Computational visual attention systems and their cognitive foundations: a survey, ACM Trans. Appl. Percept. (TAP) 7 (2010). https://doi.org/10.1145/1658349.1658355

  50. Z. Yang, D. Yang, C. Dyer, X. He, A Smola, E. Hovy, Hierarchical attention networks for document classification, in NAACL-HLT 2016 (San Diego, California, 2016), pp. 1480–1489

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Arizona State University, Tempe, USA

    Shokoufeh Monjezi Kouchak & Ashraf Gaffar

Authors
  1. Shokoufeh Monjezi Kouchak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ashraf Gaffar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ashraf Gaffar .

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of Kashmir, Srinagar, India

    M. Arif Wani

  2. Computer and Electrical Engineering, Florida Atlantic University, Boca Raton, FL, USA

    Taghi M. Khoshgoftaar

  3. Faculty of Engineering and Computing, Coventry University, Coventry, UK

    Vasile Palade

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Kouchak, S.M., Gaffar, A. (2021). Training Deep Learning Sequence Models to Understand Driver Behavior. In: Wani, M.A., Khoshgoftaar, T.M., Palade, V. (eds) Deep Learning Applications, Volume 2. Advances in Intelligent Systems and Computing, vol 1232. Springer, Singapore. https://doi.org/10.1007/978-981-15-6759-9_6

Download citation

Publish with us

Policies and ethics

Search

Navigation