Abstract
The accuracy of stereo algorithms or optical flow methods is commonly assessed by comparing the results against the Middlebury database. However, equivalent data for automotive or robotics applications rarely exist as they are difficult to obtain. As our main contribution, we introduce an evaluation framework tailored for stereo-based driver assistance able to deliver excellent performance measures while circumventing manual label effort. Within this framework one can combine several ways of ground-truthing, different comparison metrics, and use large image databases.
Using our framework we show examples on several types of ground-truthing techniques: implicit ground truthing (e.g. sequence recorded without a crash occurred), robotic vehicles with high precision sensors, and to a small extent, manual labeling. To show the effectiveness of our evaluation framework we compare three different stereo algorithms on pixel and object level. In more detail we evaluate an intermediate representation called the Stixel World. Besides evaluating the accuracy of the Stixels, we investigate the completeness (equivalent to the detection rate) of the Stixel World vs. the number of phantom Stixels. Among many findings, using this framework enables us to reduce the number of phantom Stixels by a factor of three compared to the base parametrization. This base parametrization has already been optimized by test driving vehicles for distances exceeding 10000 km.
Keywords
- Ground Truth
- Evaluation Framework
- Velocity Error
- Ground Truth Data
- Stereo Match
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Badino, H.: A Robust Approach for Ego-Motion Estimation Using a Mobile Stereo Platform. In: Jähne, B., Mester, R., Barth, E., Scharr, H. (eds.) IWCM 2004. LNCS, vol. 3417, pp. 198–208. Springer, Heidelberg (2007)
Badino, H., Franke, U., Mester, R.: Free space computation using stochastic occupancy grids and dynamic programming. In: Workshop on Dynamical Vision, ICCV, Rio de Janeiro, Brazil (October 2007)
Badino, H., Franke, U., Pfeiffer, D.: The Stixel World - A Compact Medium Level Representation of the 3D-World. In: Denzler, J., Notni, G., Süße, H. (eds.) DAGM 2009. LNCS, vol. 5748, pp. 51–60. Springer, Heidelberg (2009)
Barth, A.: Vehicle Tracking and Motion Estimation Based on Stereo Vision Sequences. PhD thesis, Friedrich-Wilhelms-Universitaet zu Bonn (September 2010)
Barth, A., Franke, U.: Where will the oncoming vehicle be the next second? In: IEEE Intelligent Vehicles Symposium (IV), Eindhoven, Netherlands, pp. 1068–1073 (April 2008)
Barth, A., Siegemund, J., Franke, U., Förstner, W.: Simultaneous Estimation of Pose and Motion at Highly Dynamic Turn Maneuvers. In: Denzler, J., Notni, G., Süße, H. (eds.) DAGM 2009. LNCS, vol. 5748, pp. 262–271. Springer, Heidelberg (2009)
Brox, T., Weickert, J.: Nonlinear Matrix Diffusion for Optic Flow Estimation. In: Van Gool, L. (ed.) DAGM 2002. LNCS, vol. 2449, pp. 446–453. Springer, Heidelberg (2002)
Collins, R., Tsin, Y., Miller, J.R., Lipton, A.: Using a dem to determine geospatial object trajectories. In: Proceedings of the 1998 DARPA Image Understanding Workshop, pp. 115–122 (1998)
Courtney, P., Thacker, N., Clark, A.: Algorithmic modeling for performance evaluation. In: IAPR Conference on Machine Vision Applications (MVA), pp. 219–228 (1997)
Dreuw, P., Steingrube, P., Deselaers, T., Ney, H.: Smoothed Disparity Maps for Continuous American Sign Language Recognition. In: Araujo, H., Mendonça, A.M., Pinho, A.J., Torres, M.I. (eds.) IbPRIA 2009. LNCS, vol. 5524, pp. 24–31. Springer, Heidelberg (2009)
Duffy, B.R., Garcia, C., Rooney, C.F.B., O’Hare, G.M.P.: Sensor fusion for social robotics. In: 31st International Symposium on Robotics, pp. 155–170 (2000)
Everingham, M., Zisserman, A., Williams, C.K.I., Van Gool, L.: The 2005 pascal visual object classes challenge. Selected Proceedings of the 1st PASCAL Challenges Workshop, Springer (2006)
Franke, U.: Real-time stereo vision for urban traffic scene understanding. In: IEEE Intelligent Vehicles Symposium, IV (2000)
Franke, U., Rabe, C., Badino, H., Gehrig, S.: 6D-Vision: Fusion of Stereo and Motion for Robust Environment Perception. In: Kropatsch, W.G., Sablatnig, R., Hanbury, A. (eds.) DAGM 2005. LNCS, vol. 3663, pp. 216–223. Springer, Heidelberg (2005)
Fraundorfer, F., Scaramuzza, D., Pollefeys, M.: A constricted bundle adjustment parameterization for relative scale estimation in visual odometry. In: IEEE International Conference on Robotics and Automation (ICRA), Anchorage, Alaska, USA, pp. 1899–1904 (May 2010)
Gehrig, S.K., Eberli, F., Meyer, T.: A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 134–143. Springer, Heidelberg (2009)
Hohm, A., Wojek, C., Bernt, S., Winner, H.: Multi level sensorfusion and computer-vision algorithms within a driver assistance system for avoiding overtaking accidents. In: FISITA World Automotive Congress, pp. 1–14 (2008)
Hong, T., Chang, T., Takeuchi, A., Cheok, G., Scott, H., Shneier, M.: Performance evaluation of sensors on mobile vehicles using a large data repository and ground truth. In: Proceedings PerMIS (2003)
Huang, W., Tan, C.-L., Zhao, J.: Generating Ground Truthed Dataset of Chart Images: Automatic or Semi-automatic? In: Liu, W., Lladós, J., Ogier, J.-M. (eds.) GREC 2007. LNCS, vol. 5046, pp. 266–277. Springer, Heidelberg (2008)
iMAR Navigation. iTraceRT-F200 (August 2011), http://www.imar-navigation.de/
Kitt, B., Geiger, A., Lategahn, H.: Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In: IEEE Intelligent Vehicles Symposium (IV), San Diego, CA, USA, pp. 486–492 (June 2010)
Lemaire, T., Berger, C., Jung, I.-K., Lacroix, S.: Vision-based slam: Stereo and monocular approaches. International Journal of Computer Vision (IJCV) 74(3), 343–364 (2007)
Levin, A., Szeliski, R.: Visual odometry and map correlation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington, DC, USA, pp. 611–618 (June 2004)
Liu, Z., Klette, R.: Approximated Ground Truth for Stereo and Motion Analysis on Real-World Sequences. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 874–885. Springer, Heidelberg (2009)
Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the Seventh International Joint Conference on Artifical Intelligence, IJCAI 1981, Vancouver, Canada, pp. 674–679 (1981)
Manohar, V., Soundararajan, P., Raju, H., Goldgof, D., Kasturi, R., Garofolo, J.S.: Performance Evaluation of Object Detection and Tracking in Video. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3852, pp. 151–161. Springer, Heidelberg (2006)
Mariano, V.Y., Min, J., Park, J.H., Kasturi, R., Mihalcik, D., Li, H., Doermann, D.S., Drayer, T.: Performance evaluation of object detection algorithms. In: International Conference on Pattern Recognition, ICPR (2002)
Morales, S., Vaudrey, T., Klette, R.: Robustness evaluation of stereo algorithms on long stereo sequences. In: IEEE Intelligent Vehicles Symposium (IV), pp. 347–352 (2009)
Müller, T., Rannacher, J., Rabe, C., Franke, U.: Feature and depth-supported modified total variation optical flow for 3d motion field estimation in real scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA, pp. 1193–1200 (June 2011)
Pfeiffer, D., Franke, U.: Efficient representation of traffic scenes by means of dynamic Stixels. In: IEEE Intelligent Vehicles Symposium (IV), San Diego, CA, USA, pp. 217–224 (June 2010)
Pfeiffer, D., Morales, S., Barth, A., Franke, U.: Ground truth evaluation of the Stixel representation using laser scanners. In: IEEE Conference on Intelligent Transportation Systems (ITSC), Maideira Island, Portugal (September 2010)
Scharstein, D., Szeliski, R.: Middlebury online stereo evaluation (2002), http://vision.middlebury.edu/stereo
Schneider, N.: Evaluation of stereo-based scene analysis under real-world conditions. Master’s thesis, Brunel University (July 2011)
Continental Automotive Industrial Sensors. ARS 300 Long Range Radar Sensor 77 GHz (July 2011), http://www.conti-online.com/generator/www/de/en/continental/industrial_sensors/themes/ars_300/ars_300_en.html
Stein, F.: Efficient Computation of Optical Flow Using the Census Transform. In: Rasmussen, C.E., Bülthoff, H.H., Schölkopf, B., Giese, M.A. (eds.) DAGM 2004. LNCS, vol. 3175, pp. 79–86. Springer, Heidelberg (2004)
Steingrube, P., Gehrig, S., Franke, U.: Performance Evaluation of Stereo Algorithms for Automotive Applications. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 285–294. Springer, Heidelberg (2009)
Tech-News. Toyota’ lexus ls 460 employs stereo camera, http://techon.nikkeibp.co.jp/english/NEWS_EN/20060301/113832/ (viewed April 15, 2009)
Tistarelli, M.: Multiple Constraints for Optical Flow. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 800, pp. 61–70. Springer, Heidelberg (1994)
Tomasi, C., Shi, J.: Good features to track. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA, pp. 593–600 (June 1994)
Vaudrey, T., Rabe, C., Klette, R., Milburn, J.: Differences between stereo and motion behaviour on synthetic and real-world stereo sequences. In: 23rd International Conference on Image and Vision Computing New Zealand, IVCNZ 2008, November 26-28, pp. 1–6 (2008)
Yamada, K., Mochizuki, K., Aizawa, K., Saito, T.: Motion Segmentation with Census Transform. In: Shum, H.-Y., Liao, M., Chang, S.-F. (eds.) PCM 2001. LNCS, vol. 2195, pp. 903–908. Springer, Heidelberg (2001)
Yilmaz, A.: Sensor fusion in computer vision. In: EEE GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas (2007)
Zach, C., Pock, T., Bischof, H.: A Duality Based Approach for Realtime TV–L1 Optical Flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007)
Zanibbi, R., Blostein, D., Cordy, J.R.: White-box evaluation of computer vision algorithms through explicit decision-making (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Schneider, N., Gehrig, S., Pfeiffer, D., Banitsas, K. (2012). An Evaluation Framework for Stereo-Based Driver Assistance. In: Dellaert, F., Frahm, JM., Pollefeys, M., Leal-Taixé, L., Rosenhahn, B. (eds) Outdoor and Large-Scale Real-World Scene Analysis. Lecture Notes in Computer Science, vol 7474. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34091-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-34091-8_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34090-1
Online ISBN: 978-3-642-34091-8
eBook Packages: Computer ScienceComputer Science (R0)
