Skip to main content

An Evaluation Framework for Stereo-Based Driver Assistance

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNIP,volume 7474)

Abstract

The accuracy of stereo algorithms or optical flow methods is commonly assessed by comparing the results against the Middlebury database. However, equivalent data for automotive or robotics applications rarely exist as they are difficult to obtain. As our main contribution, we introduce an evaluation framework tailored for stereo-based driver assistance able to deliver excellent performance measures while circumventing manual label effort. Within this framework one can combine several ways of ground-truthing, different comparison metrics, and use large image databases.

Using our framework we show examples on several types of ground-truthing techniques: implicit ground truthing (e.g. sequence recorded without a crash occurred), robotic vehicles with high precision sensors, and to a small extent, manual labeling. To show the effectiveness of our evaluation framework we compare three different stereo algorithms on pixel and object level. In more detail we evaluate an intermediate representation called the Stixel World. Besides evaluating the accuracy of the Stixels, we investigate the completeness (equivalent to the detection rate) of the Stixel World vs. the number of phantom Stixels. Among many findings, using this framework enables us to reduce the number of phantom Stixels by a factor of three compared to the base parametrization. This base parametrization has already been optimized by test driving vehicles for distances exceeding 10000 km.

Keywords

  • Ground Truth
  • Evaluation Framework
  • Velocity Error
  • Ground Truth Data
  • Stereo Match

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Badino, H.: A Robust Approach for Ego-Motion Estimation Using a Mobile Stereo Platform. In: Jähne, B., Mester, R., Barth, E., Scharr, H. (eds.) IWCM 2004. LNCS, vol. 3417, pp. 198–208. Springer, Heidelberg (2007)

    CrossRef  Google Scholar 

  2. Badino, H., Franke, U., Mester, R.: Free space computation using stochastic occupancy grids and dynamic programming. In: Workshop on Dynamical Vision, ICCV, Rio de Janeiro, Brazil (October 2007)

    Google Scholar 

  3. Badino, H., Franke, U., Pfeiffer, D.: The Stixel World - A Compact Medium Level Representation of the 3D-World. In: Denzler, J., Notni, G., Süße, H. (eds.) DAGM 2009. LNCS, vol. 5748, pp. 51–60. Springer, Heidelberg (2009)

    CrossRef  Google Scholar 

  4. Barth, A.: Vehicle Tracking and Motion Estimation Based on Stereo Vision Sequences. PhD thesis, Friedrich-Wilhelms-Universitaet zu Bonn (September 2010)

    Google Scholar 

  5. Barth, A., Franke, U.: Where will the oncoming vehicle be the next second? In: IEEE Intelligent Vehicles Symposium (IV), Eindhoven, Netherlands, pp. 1068–1073 (April 2008)

    Google Scholar 

  6. Barth, A., Siegemund, J., Franke, U., Förstner, W.: Simultaneous Estimation of Pose and Motion at Highly Dynamic Turn Maneuvers. In: Denzler, J., Notni, G., Süße, H. (eds.) DAGM 2009. LNCS, vol. 5748, pp. 262–271. Springer, Heidelberg (2009)

    CrossRef  Google Scholar 

  7. Brox, T., Weickert, J.: Nonlinear Matrix Diffusion for Optic Flow Estimation. In: Van Gool, L. (ed.) DAGM 2002. LNCS, vol. 2449, pp. 446–453. Springer, Heidelberg (2002)

    CrossRef  Google Scholar 

  8. Collins, R., Tsin, Y., Miller, J.R., Lipton, A.: Using a dem to determine geospatial object trajectories. In: Proceedings of the 1998 DARPA Image Understanding Workshop, pp. 115–122 (1998)

    Google Scholar 

  9. Courtney, P., Thacker, N., Clark, A.: Algorithmic modeling for performance evaluation. In: IAPR Conference on Machine Vision Applications (MVA), pp. 219–228 (1997)

    Google Scholar 

  10. Dreuw, P., Steingrube, P., Deselaers, T., Ney, H.: Smoothed Disparity Maps for Continuous American Sign Language Recognition. In: Araujo, H., Mendonça, A.M., Pinho, A.J., Torres, M.I. (eds.) IbPRIA 2009. LNCS, vol. 5524, pp. 24–31. Springer, Heidelberg (2009)

    CrossRef  Google Scholar 

  11. Duffy, B.R., Garcia, C., Rooney, C.F.B., O’Hare, G.M.P.: Sensor fusion for social robotics. In: 31st International Symposium on Robotics, pp. 155–170 (2000)

    Google Scholar 

  12. Everingham, M., Zisserman, A., Williams, C.K.I., Van Gool, L.: The 2005 pascal visual object classes challenge. Selected Proceedings of the 1st PASCAL Challenges Workshop, Springer (2006)

    Google Scholar 

  13. Franke, U.: Real-time stereo vision for urban traffic scene understanding. In: IEEE Intelligent Vehicles Symposium, IV (2000)

    Google Scholar 

  14. Franke, U., Rabe, C., Badino, H., Gehrig, S.: 6D-Vision: Fusion of Stereo and Motion for Robust Environment Perception. In: Kropatsch, W.G., Sablatnig, R., Hanbury, A. (eds.) DAGM 2005. LNCS, vol. 3663, pp. 216–223. Springer, Heidelberg (2005)

    CrossRef  Google Scholar 

  15. Fraundorfer, F., Scaramuzza, D., Pollefeys, M.: A constricted bundle adjustment parameterization for relative scale estimation in visual odometry. In: IEEE International Conference on Robotics and Automation (ICRA), Anchorage, Alaska, USA, pp. 1899–1904 (May 2010)

    Google Scholar 

  16. Gehrig, S.K., Eberli, F., Meyer, T.: A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 134–143. Springer, Heidelberg (2009)

    CrossRef  Google Scholar 

  17. Hohm, A., Wojek, C., Bernt, S., Winner, H.: Multi level sensorfusion and computer-vision algorithms within a driver assistance system for avoiding overtaking accidents. In: FISITA World Automotive Congress, pp. 1–14 (2008)

    Google Scholar 

  18. Hong, T., Chang, T., Takeuchi, A., Cheok, G., Scott, H., Shneier, M.: Performance evaluation of sensors on mobile vehicles using a large data repository and ground truth. In: Proceedings PerMIS (2003)

    Google Scholar 

  19. Huang, W., Tan, C.-L., Zhao, J.: Generating Ground Truthed Dataset of Chart Images: Automatic or Semi-automatic? In: Liu, W., Lladós, J., Ogier, J.-M. (eds.) GREC 2007. LNCS, vol. 5046, pp. 266–277. Springer, Heidelberg (2008)

    CrossRef  Google Scholar 

  20. iMAR Navigation. iTraceRT-F200 (August 2011), http://www.imar-navigation.de/

  21. Kitt, B., Geiger, A., Lategahn, H.: Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In: IEEE Intelligent Vehicles Symposium (IV), San Diego, CA, USA, pp. 486–492 (June 2010)

    Google Scholar 

  22. Lemaire, T., Berger, C., Jung, I.-K., Lacroix, S.: Vision-based slam: Stereo and monocular approaches. International Journal of Computer Vision (IJCV) 74(3), 343–364 (2007)

    CrossRef  Google Scholar 

  23. Levin, A., Szeliski, R.: Visual odometry and map correlation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington, DC, USA, pp. 611–618 (June 2004)

    Google Scholar 

  24. Liu, Z., Klette, R.: Approximated Ground Truth for Stereo and Motion Analysis on Real-World Sequences. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 874–885. Springer, Heidelberg (2009)

    CrossRef  Google Scholar 

  25. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the Seventh International Joint Conference on Artifical Intelligence, IJCAI 1981, Vancouver, Canada, pp. 674–679 (1981)

    Google Scholar 

  26. Manohar, V., Soundararajan, P., Raju, H., Goldgof, D., Kasturi, R., Garofolo, J.S.: Performance Evaluation of Object Detection and Tracking in Video. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3852, pp. 151–161. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  27. Mariano, V.Y., Min, J., Park, J.H., Kasturi, R., Mihalcik, D., Li, H., Doermann, D.S., Drayer, T.: Performance evaluation of object detection algorithms. In: International Conference on Pattern Recognition, ICPR (2002)

    Google Scholar 

  28. Morales, S., Vaudrey, T., Klette, R.: Robustness evaluation of stereo algorithms on long stereo sequences. In: IEEE Intelligent Vehicles Symposium (IV), pp. 347–352 (2009)

    Google Scholar 

  29. Müller, T., Rannacher, J., Rabe, C., Franke, U.: Feature and depth-supported modified total variation optical flow for 3d motion field estimation in real scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA, pp. 1193–1200 (June 2011)

    Google Scholar 

  30. Pfeiffer, D., Franke, U.: Efficient representation of traffic scenes by means of dynamic Stixels. In: IEEE Intelligent Vehicles Symposium (IV), San Diego, CA, USA, pp. 217–224 (June 2010)

    Google Scholar 

  31. Pfeiffer, D., Morales, S., Barth, A., Franke, U.: Ground truth evaluation of the Stixel representation using laser scanners. In: IEEE Conference on Intelligent Transportation Systems (ITSC), Maideira Island, Portugal (September 2010)

    Google Scholar 

  32. Scharstein, D., Szeliski, R.: Middlebury online stereo evaluation (2002), http://vision.middlebury.edu/stereo

  33. Schneider, N.: Evaluation of stereo-based scene analysis under real-world conditions. Master’s thesis, Brunel University (July 2011)

    Google Scholar 

  34. Continental Automotive Industrial Sensors. ARS 300 Long Range Radar Sensor 77 GHz (July 2011), http://www.conti-online.com/generator/www/de/en/continental/industrial_sensors/themes/ars_300/ars_300_en.html

  35. Stein, F.: Efficient Computation of Optical Flow Using the Census Transform. In: Rasmussen, C.E., Bülthoff, H.H., Schölkopf, B., Giese, M.A. (eds.) DAGM 2004. LNCS, vol. 3175, pp. 79–86. Springer, Heidelberg (2004)

    CrossRef  Google Scholar 

  36. Steingrube, P., Gehrig, S., Franke, U.: Performance Evaluation of Stereo Algorithms for Automotive Applications. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 285–294. Springer, Heidelberg (2009)

    CrossRef  Google Scholar 

  37. Tech-News. Toyota’ lexus ls 460 employs stereo camera, http://techon.nikkeibp.co.jp/english/NEWS_EN/20060301/113832/ (viewed April 15, 2009)

  38. Tistarelli, M.: Multiple Constraints for Optical Flow. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 800, pp. 61–70. Springer, Heidelberg (1994)

    CrossRef  Google Scholar 

  39. Tomasi, C., Shi, J.: Good features to track. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA, pp. 593–600 (June 1994)

    Google Scholar 

  40. Vaudrey, T., Rabe, C., Klette, R., Milburn, J.: Differences between stereo and motion behaviour on synthetic and real-world stereo sequences. In: 23rd International Conference on Image and Vision Computing New Zealand, IVCNZ 2008, November 26-28, pp. 1–6 (2008)

    Google Scholar 

  41. Yamada, K., Mochizuki, K., Aizawa, K., Saito, T.: Motion Segmentation with Census Transform. In: Shum, H.-Y., Liao, M., Chang, S.-F. (eds.) PCM 2001. LNCS, vol. 2195, pp. 903–908. Springer, Heidelberg (2001)

    CrossRef  Google Scholar 

  42. Yilmaz, A.: Sensor fusion in computer vision. In: EEE GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas (2007)

    Google Scholar 

  43. Zach, C., Pock, T., Bischof, H.: A Duality Based Approach for Realtime TV–L1 Optical Flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007)

    CrossRef  Google Scholar 

  44. Zanibbi, R., Blostein, D., Cordy, J.R.: White-box evaluation of computer vision algorithms through explicit decision-making (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Schneider, N., Gehrig, S., Pfeiffer, D., Banitsas, K. (2012). An Evaluation Framework for Stereo-Based Driver Assistance. In: Dellaert, F., Frahm, JM., Pollefeys, M., Leal-Taixé, L., Rosenhahn, B. (eds) Outdoor and Large-Scale Real-World Scene Analysis. Lecture Notes in Computer Science, vol 7474. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34091-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-34091-8_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-34090-1

  • Online ISBN: 978-3-642-34091-8

  • eBook Packages: Computer ScienceComputer Science (R0)