Advances in Multimodal Tracking of Driver Distraction

  • Carlos BussoEmail author
  • Jinesh Jain


This chapter discusses research efforts focused on tracking driver distraction using multimodal features. A car equipped with various sensors is used to collect a database with real driving conditions. During the recording, the drivers were asked to perform common secondary tasks such as operating a cell phone, talking to another passenger, and changing the radio stations. We analyzed the differences observed across multimodal features when the driver was engaged in these secondary tasks. The study considers features extracted from the controller area network bus (CAN-bus), a frontal camera facing the driver, and a microphone. These features are used to predict the distraction level of the drivers. The output of the proposed regression model has high correlation with human subjective evaluations (ρ = 0.728), which validates our approach.


Attention CAN-bus data Distraction Driver behavior Driver distraction Head pose estimation Multimodal feature analysis Real traffic driving recording Secondary task Subjective evaluation of distraction 


  1. 1.
    TA Ranney, WR Garrott, MJ Goodman (2001) NHTSA driver distraction research: past, present, and future. Technical Report Paper No. 2001-06-0177, National Highway Traffic Safety Administration, June 2001Google Scholar
  2. 2.
    V Neale, T Dingus, S Klauer, J Sudweeks, M Goodman (2005) An overview of the 100-car naturalistic study and findings. Technical Report Paper No. 05-0400, National Highway Traffic Safety Administration, June 2005Google Scholar
  3. 3.
    TA Ranney (2008) Driver distraction: a review of the current state-of-knowledge. Technical Report DOT HS 810 787, National Highway Traffic Safety Administration, April 2008Google Scholar
  4. 4.
    I Trezise, EG Stoney, B Bishop, J Eren, A Harkness, C Langdon, T Mulder (2006) Inquiry into driver distraction: report of the road safety committee on the inquiry into driver distraction. Technical Report No. 209 Session 2003–2006, Melbourne: Road Safety Committee, Parliament of Victoria, August 2006Google Scholar
  5. 5.
    J Jain, C Busso (2011) Analysis of driver behaviors during common tasks using frontal video camera and CAN-Bus information. In: IEEE international conference on multimedia and expo (ICME 2011), Barcelona, July 2011Google Scholar
  6. 6.
    JJ Jain, C Busso (2011) Assessment of driver’s distraction using perceptual evaluations, self assessments and multimodal feature analysis. In: 5th Biennial workshop on DSP for in-vehicle systems, Kiel, September 2011Google Scholar
  7. 7.
    P Angkititrakul, D Kwak, S Choi, J Kim, A Phucphan, A Sathyanarayana, JHL Hansen (2007) Getting start with UTDrive: driver-behavior modeling and assessment of distraction for in-vehicle speech systems. In: Interspeech 2007, Antwerp, August 2007, pp 1334–1337Google Scholar
  8. 8.
    Lin C-T, Wu R-C, Liang S-F, Chao W-H, Chen Y-J, Jung T-P (2005) EEG-based drowsiness estimation for safety driving using independent component analysis. IEEE Trans Circ Syst I: Regular Papers 52(12):2726–2738CrossRefGoogle Scholar
  9. 9.
    T Rahman, S Mariooryad, S Keshavamurthy, G Liu, JHL Hansen, C Busso (2011) Detecting sleepiness by fusing classifiers trained with novel acoustic features. In: 12th annual conference of the international speech communication association (Interspeech’2011), Florence, August 2011Google Scholar
  10. 10.
    P Angkititrakul, M Petracca, A Sathyanarayana, JHL Hansen (2007) UTDrive: driver behavior and speech interactive systems for in-vehicle environments. In: IEEE intelligent vehicles symposium, Istanbul, June 2007, pp 566–569Google Scholar
  11. 11.
    M Kutila, M Jokela, G Markkula, MR Rue (2007) Driver distraction detection with a camera vision system. In: IEEE international conference on image processing (ICIP 2007), vol 6. San Antonio, September 2007, pp 201–204Google Scholar
  12. 12.
    Y Dong, Z Hu, K Uchimura, N Murayama (2009) Driver inattention monitoring system for intelligent vehicles: a review. In: IEEE intelligent vehicles symposium, Xi’an, June 2009, pp 875–880Google Scholar
  13. 13.
    Liang Y, Reyes ML, Lee JD (2007) Real-time detection of driver cognitive distraction using support vector machines. IEEE Trans Intell Transport Syst 8(2):340–350CrossRefGoogle Scholar
  14. 14.
    MC Su, CY Hsiung, DY Huang (2006) A simple approach to implementing a system for monitoring driver inattention. In: IEEE international conference on systems, man and cybernetics (SMC 2006), vol 1. Taipei, October 2006, pp 429–433Google Scholar
  15. 15.
    Damousis IG, Tzovaras D (2008) Fuzzy fusion of eyelid activity indicators for hypovigilance-related accident prediction. IEEE Trans Intell Transportation Syst 9(3):491–500CrossRefGoogle Scholar
  16. 16.
    F Putze, J-P Jarvis, T Schultz (2010) Multimodal recognition of cognitive workload for multitasking in the car. In: International conference on pattern recognition (ICPR 2010), Istanbul, August 2010Google Scholar
  17. 17.
    Bergasa LM, Nuevo J, Sotelo MA, Barea R, Lopez ME (2006) Real-time system for monitoring driver vigilance. IEEE Trans Intell Transport Syst 7(1):63–77CrossRefGoogle Scholar
  18. 18.
    Ersal T, Fuller HJA, Tsimhoni O, Stein JL, Fathy HK (2010) Model-based analysis and classification of driver distraction under secondary tasks. IEEE Trans Intell Transport Syst 11(3):692–701CrossRefGoogle Scholar
  19. 19.
    Tango F, Botta M (2009) Evaluation of distraction in a driver-vehicle-environment framework: an application of different data-mining techniques. In: Perner P (ed) Advances in data mining. Applications and theoretical aspects, volume 5633 of lecture notes in computer science. Springer, Berlin/Heidelberg, pp 176–190CrossRefGoogle Scholar
  20. 20.
    KM Bach, MG Jaeger, MB Skov, NG Thomassen (2009) Interacting with in-vehicle systems: understanding, measuring, and evaluating attention. In: Proceedings of the 23rd British HCI Group annual conference on people and computers: celebrating people and technology, Cambridge, UK, September 2009Google Scholar
  21. 21.
    A Sathyanarayana, P Boyraz, Z Purohit, R Lubag, JHL Hansen (2010) Driver adaptive and context aware active safety systems using CAN-bus signals. In: IEEE intelligent vehicles symposium (IV 2010), San Diego, June 2010Google Scholar
  22. 22.
    A Sathyanarayana, S Nageswaren, H Ghasemzadeh, R Jafari, JHL Hansen (2008) Body sensor networks for driver distraction identification. In: IEEE international conference on vehicular electronics and safety (ICVES 2008), Columbus, September 2008Google Scholar
  23. 23.
    J Yang, TN Chang, E Hou (2010) Driver distraction detection for vehicular monitoring. In: Annual conference of the IEEE industrial electronics society (IECON 2010), Glendale, November 2010Google Scholar
  24. 24.
    Murphy-Chutorian E, Trivedi MM (2010) Head pose estimation and augmented reality tracking: an integrated system and evaluation for monitoring driver awareness. IEEE Trans Intell Transport Syst 11(2):300–311CrossRefGoogle Scholar
  25. 25.
    Harbluk JL, Noy YI, Trbovich PL, Eizenman M (2007) An on-road assessment of cognitive distraction: impacts on drivers’ visual behavior and braking performance. Accid Anal Prev 39(2):372–379CrossRefGoogle Scholar
  26. 26.
    Perez A, Garcia MI, Nieto M, Pedraza JL, Rodriguez S, Zamorano J (2010) Argos: an advanced in-vehicle data recorder on a massively sensorized vehicle for car driver behavior experimentation. IEEE Trans Intell Transport Syst 11(2):463–473CrossRefGoogle Scholar
  27. 27.
    Abut H, Erdoğan H, Ercil A, Curuklu B, Koman HC, Taş F, Argunşah AO, Coşar S, Akan B, Karabalkan H et al (2009) Real-world data collection with “UYANIK”. In: Takeda K, Erdoğan H, Hansen JHL, Abut H (eds) In-vehicle corpus and signal processing for driver behavior. Springer, New York, pp 23–43CrossRefGoogle Scholar
  28. 28.
    P Boyraz, X Yang, JHL Hansen (2009) Computer vision applications for context-aware intelligent vehicles. In: 4th Biennial workshop on DSP for in-vehicle systems and safety, Dallas, June 2009Google Scholar
  29. 29.
    DL Strayer, JM Cooper, FA Drews (2004) What do drivers fail to see when conversing on a cell phone? In: Proceedings of human factors and ergonomics society annual meeting, vol 48. New Orleans, September 2004Google Scholar
  30. 30.
    MS Bartlett, GC Littlewort, MG Frank, C Lainscsek, I Fasel, JR Movellan (2006) Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 6(1):22–35, September 2006Google Scholar
  31. 31.
    Mendenhall W, Sincich T (2006) Statistics for engineering and the sciences. Prentice-Hall, Upper Saddle RiverGoogle Scholar
  32. 32.
    Berka C, Levendowski DJ, Lumicao MN, Yau A, Davis G, Zivkovic VT, Olmstead RE, Tremoulet PD, Craven PL (2007) EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviat Space Environ Med 78(5):231–244Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.The University of Texas at DallasRichardsonUSA

Personalised recommendations