Vision for Driver Assistance: Looking at People in a Vehicle

  • Cuong TranEmail author
  • Mohan Manubhai Trivedi


An important real-life application domain of computer vision techniques looking at people is in developing Intelligent Driver Assistance Systems (IDAS’s). By analyzing information from both looking in and looking out of the vehicle, such systems can actively prevent vehicular accidents, improve driver safety as well as driver experience. Towards such goals, developing systems looking people in a vehicle (i.e. driver and passengers) to understand their intent, behavior, and states is needed. This is a challenging task which typically requires high reliability, accuracy, and efficient performance. Challenges also come from the dynamic background and varying lighting condition in driving scenes. However, looking at people in a vehicle also has its own characteristics which could be exploited to simplify the problem such as people typically sitting in a fixed position and their activities being highly related to the driving context. In this chapter, we give a concise overview of various related research studies to see how their approaches were developed to fit the specific requirements and characteristics of looking at people in a vehicle. From a historical point of view, we first discuss studies looking at head, eyes, and facial landmarks and then studies looking at body, hands, and feet. Despite lots of active research and published papers, developing accurate, reliable, and efficient approaches for looking at people in real-world driving scenarios is still an open problem. To this end, we will discuss some remaining issues for the future development in the area.


Particle Filter Controller Area Network Facial Landmark Computer Vision Technique National Highway Traffic Safety Administration 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We thank the sponsorships of U.C. Discovery Program, National Science Foundation as well as industry sponsors including Nissan, Volkswagen Electronic Research Laboratory, and Mercedes. We also thank former and current colleagues from our Laboratory for Intelligent and Safe Automobiles (LISA) for their cooperation, assistance, and contributions: Dr. Kohsia Huang, Dr. Joel McCall, Dr. Tarak Gandhi, Dr. Sangho Park, Dr. Shinko Cheng, Dr. Steve Krotosky, Dr. Junwen Wu, Dr. Erik Murphy-Chutorian, Dr. Brendan Morris, Dr. Anup Doshi, Mr. Sayanan Sivaraman, Mr. Ashish Tawari, and Mr. Ofer Achlertheir.


  1. 1.
    Adell, E., Várhelyi, A.: Development of HMI components for a driver assistance system for safe speed and safe distance. In: The 13th World Congress and Exhibition on Intelligent Transport Systems and Services ExCel London, United Kingdom (2006) Google Scholar
  2. 2.
    Artaud, P., Planque, S., Lavergne, C., Cara, H., de Lepine, P., Tarriere, C., Gueguen, B.: An on-board system for detecting lapses of alertness in car driving. In: The 14th Int. Conf. Enhanced Safety of Vehicles (1994) Google Scholar
  3. 3.
    Bergasa, L.M., Nuevo, J., Sotelo, M.A., Barea, R., Lopez, M.E.: Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Transp. Syst. 7(1), 63–77 (2006) CrossRefGoogle Scholar
  4. 4.
    Burnham, G.O., Seo, J., Bekey, G.A.: Identification of human drivers models in car following. IEEE Trans. Autom. Control 19(6), 911–915 (1974) CrossRefGoogle Scholar
  5. 5.
    Cheng, S.Y., Park, S., Trivedi, M.M.: Multiperspective and multimodal video arrays for 3d body tracking and activity analysis. Comput. Vis. Image Underst. (Special Issue on Advances in Vision Algorithms and Systems Beyond the Visible Spectrum) 106(2–3), 245–257 (2007) CrossRefGoogle Scholar
  6. 6.
    Cheng, S.Y., Trivedi, M.M.: Turn-intent analysis using body pose for intelligent driver assistance. IEEE Pervasive Comput. 5(4), 28–37 (2006) CrossRefGoogle Scholar
  7. 7.
    Cheng, S.Y., Trivedi, M.M.: Vision-based infotainment user determination by hand recognition for driver assistance. IEEE Trans. Intell. Transp. Syst. 11(3), 759–764 (2010) CrossRefGoogle Scholar
  8. 8.
    Datta, A., Sheikh, Y., Kanade, T.: Linear motion estimation for systems of articulated planes. In: IEEE Conference on Computer Vision and Pattern Recognition (2008) Google Scholar
  9. 9.
    Dinges, D., Mallis, M.: Managing fatigue by drowsiness detection: Can technological promises be realized? In: Hartley, L. (ed.) Managing Fatigue in Transportation, Elsevier, Oxford (1998) Google Scholar
  10. 10.
    Doshi, A., Trivedi, M.M.: Investigating the relationships between gaze patterns, dynamic vehicle surround analysis, and driver intentions. In: IEEE Intelligent Vehicles Symposium (2009) Google Scholar
  11. 11.
    Doshi, A., Trivedi, M.M.: On the roles of eye gaze and head pose in predicting driver’s intent to change lanes. IEEE Trans. Intell. Transp. Syst. 10(3), 453–462 (2009) CrossRefGoogle Scholar
  12. 12.
    Doshi, A., Trivedi, M.M.: Examining the impact of driving style on the predictability and responsiveness of the driver: Real-world and simulator analysis. In: IEEE Intelligent Vehicles Symposium (2010) Google Scholar
  13. 13.
    Fletchera, L., Loyb, G., Barnesc, N., Zelinsky, A.: Correlating driver gaze with the road scene for driver assistance systems. Robot. Auton. Syst. 52(1), 71–84 (2005) CrossRefGoogle Scholar
  14. 14.
    Grace, R., Byrne, V.E., Bierman, D.M., Legrand, J.M., Davis, R.K., Staszewski, J.J., Carnahan, B.: A drowsy driver detection system for heavy vehicles. In: Digital Avionics Systems Conference, Proceedings, The 17th DASC, The AIAA/IEEE/SAE (1998) Google Scholar
  15. 15.
    Hammoud, R., Wilhelm, A., Malawey, P., Witt, G.: Efficient realtime algorithms for eye state and head pose tracking in advanced driver support systems. In: IEEE Conference on Computer Vision and Pattern Recognition (2005) Google Scholar
  16. 16.
    Healey, J.R., Carty, S.S.: Driver error found in some Toyota acceleration cases. In: USA Today (2010) Google Scholar
  17. 17.
    Huang, K.S., Trivedi, M.M., Gandhi, T.: Driver’s view and vehicle surround estimation using omnidirectional video stream. In: IEEE Intelligent Vehicles Symposium (2003) Google Scholar
  18. 18.
    Ishikawa, T., Baker, S., Matthews, I., Kanade, T.: Passive driver gaze tracking with active appearance models. In: The 11th World Congress on Intelligent Transportation Systems (2004) Google Scholar
  19. 19.
    Ito, T., Kanade, T.: Predicting driver operations inside vehicles. In: IEEE International Conference on Automatic Face and Gesture Recognition (2008) Google Scholar
  20. 20.
    Ji, Q., Zhu, Z., Lan, P.: Real time non-intrusive monitoring and prediction of driver fatigue. IEEE Trans. Veh. Technol. 53(4), 1052–1068 (2004) CrossRefGoogle Scholar
  21. 21.
    McCall, J., Wipf, D., Trivedi, M.M., Rao, B.: Lane change intent analysis using robust operators and sparse Bayesian learning. IEEE Trans. Intell. Transp. Syst. 8(3), 431–440 (2007) CrossRefGoogle Scholar
  22. 22.
    McCall, J.C., Trivedi, M.M.: Driver behavior and situation aware brake assistance for intelligent vehicles. Proc. IEEE 95(2), 374–387 (2007) CrossRefGoogle Scholar
  23. 23.
    Mulder, M., Pauwelussen, J.J.A., van Paassen, M.M., Mulder, M., Abbink, D.A.: Active deceleration support in car following. IEEE Trans. Syst. Man Cybern., Part A, Syst. Hum. 40(6), 1271–1284 (2010) CrossRefGoogle Scholar
  24. 24.
    Murphey, Y.L., Milton, R., Kiliaris, L.: Driver’s style classification using jerk analysis. In: IEEE Workshop on Computational Intelligence in Vehicles and Vehicular Systems (2009) Google Scholar
  25. 25.
    Murphy-Chutorian, E., Trivedi, M.M.: Head pose estimation in computer vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 31(4), 607–626 (2009) CrossRefGoogle Scholar
  26. 26.
    Murphy-Chutorian, E., Trivedi, M.M.: Head pose estimation and augmented reality tracking: An integrated system and evaluation for monitoring driver awareness. IEEE Trans. Intell. Transp. Syst. 11(2), 300–311 (2010) CrossRefGoogle Scholar
  27. 27.
    NHTSA: Traffic safety facts 2006 – a compilation of motor vehicle crash data from the fatality analysis reporting system and the general estimates system. In: Washington, DC: Nat. Center Stat. Anal., US Dept. Transp. (2006) Google Scholar
  28. 28.
    Peden, M., Scurfield, R., Sleet, D., Mohan, D., Hyder, A.A., Jarawan, E., Mathers, C.: World report on road traffic injury prevention: Summary. In: World Health Organization, Geneva, Switzerland (2004) Google Scholar
  29. 29.
    Pugeault, N., Bowden, R.: Learning pre-attentive driving behaviour from holistic visual features. In: The 11th European Conference on Computer Vision (2010) Google Scholar
  30. 30.
    Smith, P., Shah, M., Lobo, N.V.: Determining driver visual attention with one camera. IEEE Trans. Intell. Transp. Syst. 4(4), 205–218 (2003) CrossRefGoogle Scholar
  31. 31.
    Tran, C., Trivedi, M.M.: Driver assistance for keeping hands on the wheel and eyes on the road. In: IEEE International Conference on Vehicular Electronics and Safety (2009) Google Scholar
  32. 32.
    Tran, C., Trivedi, M.M.: Introducing ‘XMOB’: Extremity movement observation framework for upper body pose tracking in 3d. In: IEEE International Symposium on Multimedia (2009) Google Scholar
  33. 33.
    Tran, C., Trivedi, M.M.: Towards a vision-based system exploring 3d driver posture dynamics for driver assistance: Issues and possibilities. In: IEEE Intelligent Vehicles Symposium (2010) Google Scholar
  34. 34.
    Trivedi, M.M., Cheng, S.Y.: Holistic sensing and active displays for intelligent driver support systems. IEEE Comput. 40(5), 60–68 (2007) Google Scholar
  35. 35.
    Trivedi, M.M., Cheng, S.Y., Childers, E., Krotosky, S.: Occupant posture analysis with stereo and thermal infrared video: Algorithms and experimental evaluation. IEEE Trans. Veh. Technol. (Special Issue on In-Vehicle Vision Systems) 53(6), 1698–1712 (2004) CrossRefGoogle Scholar
  36. 36.
    Trivedi, M.M., Gandhi, T., McCall, J.: Looking-in and looking-out of a vehicle: Computer-vision-based enhanced vehicle safety. IEEE Trans. Intell. Transp. Syst. 8(1), 108–120 (2007) CrossRefGoogle Scholar
  37. 37.
    Veeraraghavan, H., Atev, S., Bird, N., Schrater, P., Papanikolopoulos, N.: Driver activity monitoring through supervised and unsupervised learning. In: IEEE Conference on Intelligent Transportation Systems (2005) Google Scholar
  38. 38.
    Wu, J., Trivedi, M.M.: An eye localization, tracking and blink pattern recognition system: Algorithm and evaluation. ACM Trans. Multimedia Comput. Commun. Appl. 6(2) (2010) Google Scholar
  39. 39.
    Yammamoto, K., Higuchi, S.: Development of a drowsiness warning system. J. Soc. Automot. Eng. Jpn. (1992) Google Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.Laboratory for Intelligent and Safe Automobiles (LISA)University of California at San DiegoSan DiegoUSA

Personalised recommendations