Real-World Stereo-Analysis Evaluation

  • Sandino Morales
  • Simon Hermann
  • Reinhard Klette
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7474)


Evaluation of stereo-analysis algorithms is usually done by analysing the performance of stereo matchers on data sets with available ground truth. The trade-off between precise results, obtained with this sort of evaluation, and the limited amount (in both, quantity and diversity) of data sets, needs to be considered if the algorithms are required to analyse real-world environments. This chapter discusses a technique to objectively evaluate the performance of stereo-analysis algorithms using real-world image sequences. The lack of ground truth is tackled by incorporating an extra camera into a multi-view stereo camera system. The relatively simple hardware set-up of the proposed technique can easily be reproduced for specific applications.


Ground Truth Stereo Match Virtual Image Virtual View Control Camera 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Badino, H., Franke, U., Mester, R.: Free space computation using stochastic occupancy grids and dynamic programming. In: Proc. Dynamic Vision, ICCV Workshop, pp. 1–12 (2007)Google Scholar
  2. 2.
    Banks, J., Corke, P.: Quantitative evaluation of matching methods and validity measures for stereo vision. Int. J. Robotics Research 20, 512–532 (2001)CrossRefGoogle Scholar
  3. 3.
    Bolles, R., Baker, H., Hannah, M.: The JISCT stereo evaluation. In: ARPA Image Understanding Workshop, pp. 263–274 (1993)Google Scholar
  4. 4.
    CMU/VASC. Stereo image data base, (retrieved 2012)
  5. 5.
    Computer Vision Group, University of Bonn. Stereo images with ground truth disparity and occlusion, (retrieved 2012)
  6. 6.
    DAGM 2011, adverse vision condition challenge, (retrieved 2012)
  7. 7.
    The .enpeda.. project, The University of Auckland. EISATS, Set 2, (retrieved 2012)
  8. 8.
    Faugeras, O., Fua, P., Hotz, B., Ma, R., Robert, L., Thonnat, M., Zhang, Z.: Quantitative and qualitative comparison of some area and feature-based stereo-analysis algorithms. In: Proc. Workshop Robust Computer Vision, pp. 1–26 (1992)Google Scholar
  9. 9.
    Felzenszwalb, P., Huttenlocher, D.: Efficient belief propagation for early vision. Int. J. Computer Vision 70, 41–54 (2006)CrossRefGoogle Scholar
  10. 10.
    Georgescu, B., Meer, P.: Point matching under large image deformations and illumination changes. IEEE Trans. Pattern Anal. Mach. Intel. 26, 674–688 (2004)CrossRefGoogle Scholar
  11. 11.
    Gherardi, R.: Confidence-based cost modulation for stereo matching. In: Proc. ICPR (2008) 978-1-4244-2175-6 Google Scholar
  12. 12.
    Gülch, E.: Results of test on image matching of ISPRS WG III/4. ISPRS J. Photogrammetry Remote Sensing 46, 1–18 (1991)CrossRefGoogle Scholar
  13. 13.
    Haeusler, R., Klette, R.: Benchmarking Stereo Data (Not the Matching Algorithms). In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) DAGM 2010. LNCS, vol. 6376, pp. 383–392. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  14. 14.
    Hermann, S., Vaudrey, T.: The gradient - a powerful and robust cost function for stereo matching. In: Proc. IVCNZ (2010) 978-1-4244-9631-0Google Scholar
  15. 15.
    Hermann, S., Morales, S., Klette, R.: Half-resolution semi-global stereo matching. In: Proc. IEEE Symp. IV, pp. 201–206 (2011)Google Scholar
  16. 16.
    Hermann, S., Klette, R.: Evaluation of a New Coarse-to-Fine Strategy for Fast Semi-Global Stereo Matching. In: Ho, Y.-S. (ed.) PSIVT 2011, Part I. LNCS, vol. 7087, pp. 395–406. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  17. 17.
    Hirschmüller, H.: Accurate and efficient stereo processing by semi-global matching and mutual information. In: Proc. CVPR, vol. 2, pp. 807–814 (2005)Google Scholar
  18. 18.
    JISCT Stereo Images, (retrieved 2012)
  19. 19.
    Klappstein, J., Vaudrey, T., Rabe, C., Wedel, A., Klette, R.: Moving Object Segmentation using Optical Flow and Depth Information. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 611–623. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  20. 20.
    Klaus, A., Sormann, M., Karner, K.: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: Proc. ICPR (2006), 10.1109/ICPR.2006.1033Google Scholar
  21. 21.
    Klette, R., Schlüns, K., Koschan, A.: Computer vision: three-dimensional data from images. Springer, Singapore (1998)zbMATHGoogle Scholar
  22. 22.
    Klette, R., Zamperoni, P.: Handbook of Image Processing Operators. John Wiley & Sons, Inc. (1996)Google Scholar
  23. 23.
    Klette, R., Krüger, N., Vaudrey, T., Pauwels, K., Hulle, M., Morales, S., Kandil, F., Haeusler, R., Pugeault, N., Rabe, C., Leppe, M.: Performance of correspondence algorithms in vision-based driver assistance using an online image sequence database. IEEE Trans. Vehicular Technology 60, 2012–2026 (2011)CrossRefGoogle Scholar
  24. 24.
    Kogler, J., Hemetsberger, H., Alefs, B., Kubinger, W., Travis, W.: Embedded stereo vision system for intelligent autonomous vehicles. In: Proc. IEEE Symp. IV, pp. 64–69 (2006)Google Scholar
  25. 25.
    Kolmogorov, V., Zabih, R.: Graph cut algorithms for binocular stereo with occlusions. In: Math. Models in Computer Vision: The Handbook, pp. 423–437. Springer (2005)Google Scholar
  26. 26.
    Kondermann, D., Meister, S., Lauer, P.: An outdoor stereo camera system for the generation of real-world benchmark datasets with ground truth. Universität Heidelberg HCI, Technical Rep. (2011)Google Scholar
  27. 27.
    Leclerc, Y.G., Luong, Q.-T., Fua, P.: Measuring the Self-Consistency of Stereo Algorithms. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 282–298. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  28. 28.
    Liu, Z., Klette, R.: Approximate Ground Truth for Stereo and Motion Analysis on Real-World Sequences. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 874–885. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  29. 29.
    Mohan, R., Medioni, G., Nevatia, R.: Stereo error detection, correction and evaluation. IEEE Trans. Pattern Analysis Machine Intelligence 11, 113–120 (1989)CrossRefGoogle Scholar
  30. 30.
    Morales, S., Klette, R.: A Third Eye for Performance Evaluation in Stereo Sequence Analysis. In: Jiang, X., Petkov, N. (eds.) CAIP 2009. LNCS, vol. 5702, pp. 1078–1086. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  31. 31.
    Morales, S., Klette, R.: Ground Truth Evaluation of Stereo Algorithms for Real World Applications. In: Koch, R., Huang, F. (eds.) ACCV 2010 Workshops, Part II. LNCS, vol. 6469, pp. 152–162. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  32. 32.
    Mordohai, P.: The self-aware matching measure for stereo. In: Proc. ICCV, pp. 1841–1848 (2009)Google Scholar
  33. 33.
    Satoh, Y., Sakaue, K.: An omnidirectional stereo vision-based smart wheelchair. EURASIP J. Image Video Processing, 1–12 (2007)Google Scholar
  34. 34.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Computer Vision 47, 7–42 (2001)CrossRefGoogle Scholar
  35. 35.
    Schauwecker, K., Morales, S., Hermann, S., Klette, R.: A comparative study of stereo-matching algorithms for road-modelling in the presence of windscreen wipers. In: Proc. IEEE Symp. IV, pp. 7–12 (2011)Google Scholar
  36. 36.
    Steingrube, P., Gehrig, S.K., Franke, U.: Performance Evaluation of Stereo Algorithms for Automotive Applications. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 285–294. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  37. 37.
    Szeliski, R.: Prediction error as a quality metric for motion and stereo. In: Proc. ICCV, pp. 781–788 (1999)Google Scholar
  38. 38.
    van der Mark, W., Gavrila, M.: Real-time dense stereo for intelligent vehicles. IEEE Trans. Intelligent Transportation Systems 7, 38–50 (2006)CrossRefGoogle Scholar
  39. 39.
    Vaudrey, T., Rabe, C., Klette, R., Milburn, J.: Differences between stereo and motion behaviour on synthetic and real-world stereo sequences. In: Proc. IVCNZ, pp. 1–6 (2008)Google Scholar
  40. 40.
    Vaudrey, T., Morales, S., Wedel, A., Klette, R.: Generalized residual images effect on illumination artifact removal for correspondence algorithms. Pattern Recognition 44, 2034–2046 (2011)CrossRefGoogle Scholar
  41. 41.
    Zabih, R., Woodfill, J.: Non-Parametric Local Transforms for Computing Visual Correspondence. In: Eklundh, J.-O. (ed.) ECCV 1994, Part II. LNCS, vol. 801, pp. 151–158. Springer, Heidelberg (1994)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Sandino Morales
    • 1
  • Simon Hermann
    • 1
  • Reinhard Klette
    • 1
  1. 1.The .enpeda.. ProjectThe University of AucklandNew Zealand

Personalised recommendations