Awareness of Road Scene Participants for Autonomous Driving

  • Anna PetrovskayaEmail author
  • Mathias Perrollaz
  • Luciano Oliveira
  • Luciano Spinello
  • Rudolph Triebel
  • Alexandros Makris
  • John-David Yoder
  • Christian Laugier
  • Urbano Nunes
  • Pierre Bessiere
Reference work entry


This chapter describes detection and tracking of moving objects (DATMO) for purposes of autonomous driving. DATMO provides awareness of road scene participants, which is important in order to make safe driving decisions and abide by the rules of the road. Three main classes of DATMO approaches are identified and discussed. First is the traditional approach, which includes data segmentation, data association, and filtering using primarily Kalman filters. Recent work within this class of approaches has focused on pattern recognition techniques. The second class is the model-based approach, which performs inference directly on the sensor data without segmentation and association steps. This approach utilizes geometric object models and relies on non-parametric filters for inference. Finally, the third class is the grid-based approach, which starts by constructing a low level grid representation of the dynamic environment. The resulting representation is immediately useful for determining free navigable space within the dynamic environment. Grid construction can be followed by segmentation, association, and filtering steps to provide object level representation of the scene. The chapter introduces main concepts, reviews relevant sensor technologies, and provides extensive references to recent work in the field. The chapter also provides a taxonomy of DATMO applications based on road scene environment and outlines requirements for each application.


Markov Chain Monte Carlo Data Association Conditional Random Field Sensor Fusion Occupancy Grid 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Andrieu C, De Freitas N, Doucet A, Jordan M (2003) An introduction to MCMC for machine learning. Mach Learn 50(1):5–43zbMATHCrossRefGoogle Scholar
  2. Arras K, Mozos O, Burgard W (2007) Using boosted features for the detection of people in 2d range data. In: Proceedings of IEEE international conference on robotics and automation (ICRA), Rome, pp 3402–3407Google Scholar
  3. Arulampalam S, Maskell S, Gordon N, Clapp T (2002) A tutorial on particle filters for on-line nonlinear/non-gaussian bayesian tracking. IEEE Trans Signal Process 50:174–188CrossRefGoogle Scholar
  4. Badino H, Franke U, Mester R (2007) Free space computation using stochastic occupancy grids and dynamic programming. In: IEEE international conference on computer vision, workshop on dynamical vision, Rio de JaneiroGoogle Scholar
  5. Bar-Shalom Y, Jaffer A (1972) Adaptive nonlinear filtering for tracking with measurements of uncertain origin. In: Conference on decision and control and 11th symposium on adaptive processes, New Orleans. Institute of Electrical and Electronics Engineers, New York, pp 243–247CrossRefGoogle Scholar
  6. Bertozzi M, Broggi A, Castelluccio S (1997) A real-time oriented system for vehicle detection. J Syst Archit 43:317–325CrossRefGoogle Scholar
  7. Bertozzi M, Broggi A, Fascioli A (2000) Vision-based intelligent vehicles: state of the art and perspectives. Robot Auton Syst 32:1–16CrossRefGoogle Scholar
  8. Blackman S, Co R, El Segundo C (2004) Multiple hypothesis tracking for multiple target tracking. IEEE Aerosp Electron Syst Mag 19(1 Part 2):5–18CrossRefGoogle Scholar
  9. Blackwell D (1947) Conditional expectation and unbiased sequential estimation. Ann Math Stat 18(1):105–110MathSciNetzbMATHCrossRefGoogle Scholar
  10. Borgefors G (1986) Distance transformations in digital images. Comput Vision Graph Image Process 34:344–371CrossRefGoogle Scholar
  11. Braillon C, Pradalier C, Usher K, Crowley J, Laugier C (2006) Occupancy grids from stereo and optical flow data. In: Proceedings of international symposium on experimental robotics, Rio de JaneiroGoogle Scholar
  12. Broggi A, Cerri P, Ghidoni S, Grisleri P, Gi J (2008) Localization and analysis of critical areas in urban scenarios. In: Proceedings of IEEE international symposium on intelligent vehicles, Eindhoven, pp 1074–1079Google Scholar
  13. Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24:603–619CrossRefGoogle Scholar
  14. Coué C, Pradalier C, Laugier C, Fraichard T, Bessiére P (2006) Bayesian occupancy filtering for multitarget tracking: an automotive application. Int J Robot Res 25(1):19–30CrossRefGoogle Scholar
  15. Cox I (1993) A review of statistical data association techniques for motion correspondence. Int J Comput Vis 10(1):53–66CrossRefGoogle Scholar
  16. Cucchiara R, Piccardi M (1999) Vehicle detection under day and night illumination. In: Proceedings of the 3rd international ICSC symposium on intelligent industrial automation, GenovaGoogle Scholar
  17. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), vol 1, San Diego, pp 886–893Google Scholar
  18. Dasarathy B (1997) Sensor fusion potential exploitation – innovative and illustrative applications. In: Proceedings of the IEEE special issue on sensor fusion, vol 85, pp 24–38. IEEEGoogle Scholar
  19. Diard J, Bessiere P, Mazer E (2003) A survey of probabilistic models using the bayesian programming methodology as a unifying framework. In: The second international conference on computational intelligence, robotics and autonomous systems (CIRAS 2003), SingaporeGoogle Scholar
  20. Dickmanns E, Behringer R, Dickmanns D, Hilde-brandt T, Maurer M, Thomanek F, Schiehlen J (1994) The seeing passenger car “vamors-p”. In: Proceedings of the intelligent vehicles 1994 symposium, Paris, France, pp 68–73Google Scholar
  21. Dollar P, Wojek C, Schiele B, Perona P (2009) Pedestrian detection: a benchmark. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), Miami, USA, pp 304–311Google Scholar
  22. Douillard B, Fox D, Ramos F (2007) A spatio-temporal probabilistic model for multi-sensor object recognition. In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems, San Diego, CA, USA, pp 2402–2408Google Scholar
  23. Douillard B, Fox D, Ramos F (2008) Laser and vision based outdoor object mapping. In: Proceedings of robotics: science and systems (RSS), ZurichGoogle Scholar
  24. Enzweiler M, Gavrila D (2009) Monocular pedestrian detection: survey and experiments. IEEE Trans Pattern Anal Mach Intell 31(12):2179–2195CrossRefGoogle Scholar
  25. Ess A, Leibe B, Schindler K, Van Gool L (2009) Moving obstacle detection in highly dynamic scenes. In: Proceedings of IEEE international conference on Robotics and Automation (ICRA), Kobe, JapanGoogle Scholar
  26. Felzenszwalb P, McAllester D, Ramanan D (2008) A discriminatively trained, multiscale, deformable part model. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), Anchorage, Alaska, USA, pp 1–8Google Scholar
  27. Fod A, Howard A, Mataric M (2002) Laser-based people tracking. In: Proceedings of IEEE international conference on robotics and automation (ICRA), vol 3, Washington, DC, pp 3024–3029Google Scholar
  28. Fortmann T, Bar-Shalom Y, Scheffe M (1983) Sonar tracking of multiple targets using joint probabilistic data association. IEEE J Ocean Eng 8(3):173–184CrossRefGoogle Scholar
  29. Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139MathSciNetzbMATHCrossRefGoogle Scholar
  30. Gavrila D (2000) Pedestrian detection from a moving vehicle. In: European conference on computer vision (ECCV), Dublin, pp 37–49. IEEE Computer SocietyGoogle Scholar
  31. Gordon N (1993) Bayesian methods for tracking. PhD thesis, University of LondonGoogle Scholar
  32. Hastings WK (1970) Monte Carlo sampling methods using markov chains and their applications. Biometrika 57(1):97–109zbMATHCrossRefGoogle Scholar
  33. Kleinhagenbrock M, Lang S, Fritsch J, Lömker F, Fink G, Sagerer G (2002) Person tracking with a mobile robot based on multi-modal anchoring. In: IEEE international workshop on robot and human interactive communication (ROMAN), Berlin, GermanyGoogle Scholar
  34. Kuehnle A (1991) Symmetry-based recognition of vehicle rears. Pattern Recognit Lett 12:249–258CrossRefGoogle Scholar
  35. Kuncheva L (2004) Combining pattern classifiers: methods and algorithms. Wiley-Interscience, HobokenzbMATHCrossRefGoogle Scholar
  36. Labayrade R, Aubert D, Tarel J (2002) Real time obstacles detection on non flat road geometry through v-disparity representation. In: Proceedings of IEEE Intelligent Vehicle Symposium (IV), Versailles, FranceGoogle Scholar
  37. Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: probabilistic models for segmentation and labeling sequence data. In: International conference on machine learning (ICML), Williamstown, pp 282–289Google Scholar
  38. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2279–2324CrossRefGoogle Scholar
  39. Leibe B, Seemann E, Schiele B (2005) Pedestrian detection in crowded scenes. In: Proceedings of IEEE international conference on computer vision and pattern recognition (CVPR), vol 1, San Diego, pp 878–885Google Scholar
  40. Leibe B, Cornelis N, Cornelis K, Van Gool L (2007) Dynamic 3D scene analysis from a moving vehicle. In: Proceedings of IEEE international conference on computer vision and pattern recognition (CVPR), vol 1, Minneapolis, pp 1–8Google Scholar
  41. Leonard J, How J, Teller S, Berger M, Campbell S, Fiore G, Fletcher L, Frazzoli E, Huang A, Kara-man S et al (2008) A perception-driven autonomous urban vehicle. J Field Robot 25(10):727–774CrossRefGoogle Scholar
  42. Lundquist C, Schon T (2008) Road geometry estimation and vehicle tracking using a single track model. In: Intelligent vehicles symposium, 2008 IEEE, pp 144–149. IEEE, Eindhoven, The NetherlandsGoogle Scholar
  43. MacKay DJC (1998) Introduction to Monte Carlo methods. In: Jordan MI (ed) Learning in graphical models, NATO science series. Kluwer Academic, Dordrecht, pp 175–204Google Scholar
  44. Mahlisch RS, Ritter W, Dietmayer K (2006) Sen-sorfusion using spatio-temporal aligned video and lidar for improved vehicle detection. In: Proceedings of IEEE international symposium on intelligent vehicles, Tokyo, Japan, pp 424–429Google Scholar
  45. Matthies L, Elfes A (1988) Integration of sonar and stereo range data using a grid-based representation. In: Proceedings of IEEE international conference on robotics and automation, PhiladelphiaGoogle Scholar
  46. Mekhnacha K, Mao Y, Raulo D, Laugier C (2008) Bayesian occupancy filter based Fast Clustering-Tracking algorithm. In: Proceedings of IEEE/RSJ international conference on intelligent robot and systems, NiceGoogle Scholar
  47. Metropolis N, Rosenbluth A, Rosenbluth M, Teller A, Teller E et al (1953) Equation of state calculations by fast computing machines. J Chem Phys 21(6):1087CrossRefGoogle Scholar
  48. Mikolajczyk K, Schmid C, Zisserman A (2004) Human detection based on a probabilistic assembly of robust part detectors. In: European conference on computer vision (ECCV), Prague, Czech Republic, pp 69–82Google Scholar
  49. Mohan A, Papageorgiou C, Poggio T (2001) Example-based object detection in images by components. IEEE Trans Pattern Anal Mach Intell 23(4):349–361CrossRefGoogle Scholar
  50. Montemerlo M (2003) FastSLAM: a factored solution to the simultaneous localization and mapping problem with unknown data association. PhD thesis, Robotics Institute, Carnegie Mellon UniversityGoogle Scholar
  51. Montemerlo M, Becker J, Bhat S, Dahlkamp H, Dolgov D, Ettinger S, Hähnel D, Hilden T, Hoffmann G, Huhnke B, Johnston D, Klumpp S, Langer D, Levandowski A, Levinson J, Marcil J, Orenstein D, Paefgen J, Penny I, Petrovskaya A, Pflueger M, Stanek G, Stavens D, Vogt A, Thrun S (2008) Junior: the stanford entry in the urban challenge. J Field Robot 25(9):569–597CrossRefGoogle Scholar
  52. Moravec H (1988) Sensor fusion in certainty grids for mobile robots. AI Mag 9(2)Google Scholar
  53. Munder S, Gavrila D (2006) An experimental study on pedestrian classification. IEEE Trans Pattern Anal Mach Intell 28:1863–1868CrossRefGoogle Scholar
  54. Murphy K, Russell S (2001) Rao-blackwellized particle filtering for dynamic Bayesian networks. Springer, HeidelbergGoogle Scholar
  55. Murray D, Little J (2000) Using real-time stereo vision for mobile robot navigation. Auton Robot 8(2):161–171CrossRefGoogle Scholar
  56. Oliveira L, Nunes U (2010) Context-aware pedestrian detection using lidar. In: Proceedings of IEEE international symposium on intelligent vehicles, San Diego, CA, USAGoogle Scholar
  57. Oliveira L, Nunes U, Peixoto P (2010a) On exploration of classifier ensemble synergism in pedestrian detection. IEEE Trans Intell Transp Syst 11:16–27CrossRefGoogle Scholar
  58. Oliveira L, Nunes U, Peixoto P, Silva M, Moita F (2010b) Semantic fusion of laser and vision in pedestrian detection. Pattern Recognit 43:3648–3659zbMATHCrossRefGoogle Scholar
  59. Papageorgiou C, Poggio T (2000) A trainable system for object detection. Int J Comput Vis 38(1):15–33zbMATHCrossRefGoogle Scholar
  60. Perrollaz M, Spalanzani A, Aubert D (2010a) Probabilistic representation of the uncertainty of stereo-vision and application to obstacle detection. In: Proceedings of IEEE international symposium on intelligent vehicles, San Diego, CA, USA, pp 313–318Google Scholar
  61. Perrollaz M, Yoder J-D, Laugier C (2010b) Using obstacle and road pixels in the disparity space computation of stereovision based occupancy grids. In: Proceedings of IEEE international conference on intelligent transportation systems, Madeira, PortugalGoogle Scholar
  62. Petrovskaya A (2011) Towards dependable robotic perception. Ph D thesis, Stanford University, StanfordGoogle Scholar
  63. Petrovskaya A, Khatib O (2011) Global localization of objects via touch. IEEE Trans Robot 27(3):569–585CrossRefGoogle Scholar
  64. Petrovskaya A, Thrun S (2009) Model based vehicle detection and tracking for autonomous urban driving. Auton Robot 26(2):123–139CrossRefGoogle Scholar
  65. Premebida C, Monteiro G, Nunes U, Peixoto P (2007) A lidar and vision-based approach for pedestrian and vehicle detection and tracking. In: Proceedings of IEEE international conference on intelligent transportation systems, Seattle, USA, pp 1044–1049Google Scholar
  66. Prisacariu VA, Reid I (2009) fasthog- a realtime gpu implementation of hog technical report no. 2310/09Google Scholar
  67. Rao C (1945) Information and accuracy obtainable in one estimation of a statistical parameter. Bull Calcutta Math Soc 37:81–91MathSciNetzbMATHGoogle Scholar
  68. Reid DB (1979) An algorithm for tracking multiple targets. IEEE Trans Autom Control 24:843–854CrossRefGoogle Scholar
  69. Reisman P, Mano O, Avidan S, Shashua A (2004) Crowd detection in video sequences. In: Proceedings of IEEE international symposium on intelligent vehicles, Parma, Italy, pp 66–71. IEEEGoogle Scholar
  70. Richardson M, Domingos P (2006) Markov logic networks. Mach Learn 62:107–136CrossRefGoogle Scholar
  71. Richter E, Schubert R, Wanielik G (2008) Radar and vision based data fusion-advanced filtering techniques for a multi object vehicle tracking system. In: Proceedings of intelligent vehicles symposium, 2008 IEEE, Eindhoven, The Netherlands pp 120–125. IEEEGoogle Scholar
  72. Scheunert U, Mattern N, Lindner P, Wanielik G (2008) Generalized grid framework for multi sensor data fusion. J Inf Fusion, 814–820Google Scholar
  73. Schulz D, Burgard W, Fox D, Cremers AB (2003) People tracking with mobile robots using sample-based joint probabilistic data association filters. Int J Robot Res (IJRR) 22(2):99–116CrossRefGoogle Scholar
  74. Sittler R (1964) An optimal data association problem in surveillance theory. IEEE Trans Militar Electron 8(2):125–139CrossRefGoogle Scholar
  75. Spinello L, Macho A, Triebel R, Siegwart R (2009a) Detecting pedestrians at very small scales. In: Proceedings of IEEE international conference on intelligent robots and systems (IROS), St. Louis, pp 4313–4318Google Scholar
  76. Spinello L, Triebel R, Siegwart R (2009b) A trained system for multimodal perception in urban environments. In proceedings of the workshop on people detection and tracking of IEEE ICRA 2009, Kobe, JapanGoogle Scholar
  77. Spinello L, Triebel R, Siegwart R (2010) Multi-class multimodal detection and tracking in urban environments. Int J Robot Res (IJRR) 29(12):1498–1515CrossRefGoogle Scholar
  78. Sun Z, Bebis G, Miller R (2006) On-road vehicle detection: a review. IEEE Trans Pattern Anal Mach Intell 28:694–711CrossRefGoogle Scholar
  79. Szarvas M, Sakai U, Ogata J (2006) Real-time pedestrian detection using lidar and convolutional neural networks. In: Proceedings of IEEE international symposium on intelligent vehicles, Tokyo, Japan, pp 213–218Google Scholar
  80. Thrun S, Burgard W, Fox D (2005) Probabilistic robotics. MIT Press, Cambridge, MAzbMATHGoogle Scholar
  81. Tuzel O, Porikli F, Meer P (2007) Human detection via classification on riemannian manifolds. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), Rio de Janeiro, Brazil, pp 1–8Google Scholar
  82. Vermaak J, Godsill S, Perez P (2005) Monte Carlo filtering for multi target tracking and data association. IEEE Trans Aerosp Electron Syst 41(1):309–332CrossRefGoogle Scholar
  83. Viola P, Jones MJ, Snow D (2003) Detecting pedestrians using patterns of motion and appearance. In: Proceedings of IEEE international conference on computer vision (ICCV), vol 2, Nice, pp 734–741Google Scholar
  84. Vu T (2009) Vehicle perception: localization, mapping with detection, classification and tracking of moving objects. PhD thesis, Institut National Polytechnique De GrenobleGoogle Scholar
  85. Vu T, Aycard O (2009) Laser-based detection and tracking moving objects using data-driven markov chain Monte Carlo. In: Proceedings of IEEE international conference on robotics and automation (ICRA 2009), Kobe, Japan, pp 3800–3806. IEEEGoogle Scholar
  86. Wang X, Han TX, Yan S (2009) An HOG-LBP human detector with partial occlusion handling. In: Proceedings of 2009 IEEE 12th international conference on computer vision, Kyoto, Japan, pp 32–39Google Scholar
  87. Wu B, Nevatia R (2005) Detection of multiple, partially occluded humans in a single image by Bayesian combination of edgelet part detectors. In: Proceedings of IEEE international conference on computer vision (ICCV), Beijing, ChinaGoogle Scholar
  88. Xavier J, Pacheco M, Castro D, Ruano A, Nunes U (2005) Fast line, arc/circle and leg detection from laser scan data in a player driver. In: Proceedings of IEEE international conference on robotics and automation (ICRA), Barcelona, pp 3930–3935Google Scholar
  89. Zhang Q, Pless R (2004) Extrinsic calibration of a camera and laser range finder (improves camera calibration). In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems, Sendai, Japan, vol 3, pp 2301–2306Google Scholar
  90. Zhu Q, Yeh MC, Cheng KT, Avidan S (2006) Fast human detection using a cascade of histograms of oriented gradients. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), vol 2, New York, pp 1491–1498Google Scholar

Copyright information

© Springer-Verlag London Ltd. 2012

Authors and Affiliations

  • Anna Petrovskaya
    • 1
    Email author
  • Mathias Perrollaz
    • 2
  • Luciano Oliveira
    • 3
  • Luciano Spinello
    • 4
    • 8
  • Rudolph Triebel
    • 5
    • 8
  • Alexandros Makris
    • 9
  • John-David Yoder
    • 6
  • Christian Laugier
    • 2
  • Urbano Nunes
    • 3
  • Pierre Bessiere
    • 7
  1. 1.Artificial Intelligence LaboratoryStanford UniversityStanfordUSA
  2. 2.e-Motion Project-TeamINRIA Grenoble Rhône-AlpesSaint Ismier CedexFrance
  3. 3.Faculty of Science and TechnologyCoimbra University, Pólo IICoimbraPortugal
  4. 4.University of FreiburgFreiburgGermany
  5. 5.University of OxfordOxfordUK
  6. 6.Mechanical Engineering DepartmentOhio Northern UniversityAdaUSA
  7. 7.INRIA and Collège de FranceSaint Ismier CedexFrance
  8. 8.Inst.f. Robotik u. Intelligente SystemeETH ZurichZurichSwitzerland
  9. 9.INRIASaint Ismier CedexFrance

Personalised recommendations