Advertisement

International Journal of Computer Vision

, Volume 95, Issue 1, pp 29–51 | Cite as

Stereoscopic Scene Flow Computation for 3D Motion Understanding

  • Andreas Wedel
  • Thomas Brox
  • Tobi Vaudrey
  • Clemens Rabe
  • Uwe Franke
  • Daniel Cremers
Article

Abstract

Building upon recent developments in optical flow and stereo matching estimation, we propose a variational framework for the estimation of stereoscopic scene flow, i.e., the motion of points in the three-dimensional world from stereo image sequences. The proposed algorithm takes into account image pairs from two consecutive times and computes both depth and a 3D motion vector associated with each point in the image. In contrast to previous works, we partially decouple the depth estimation from the motion estimation, which has many practical advantages. The variational formulation is quite flexible and can handle both sparse or dense disparity maps. The proposed method is very efficient; with the depth map being computed on an FPGA, and the scene flow computed on the GPU, the proposed algorithm runs at frame rates of 20 frames per second on QVGA images (320×240 pixels). Furthermore, we present solutions to two important problems in scene flow estimation: violations of intensity consistency between input images, and the uncertainty measures for the scene flow result.

Keywords

Scene flow 3D motion 3D reconstruction Structure from motion Motion segmentation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aujol, J. F., Gilboa, G., Chan, T., & Osher, S. (2006). structure-texture image decomposition—modeling, algorithms, and parameter selection. International Journal of Computer Vision, 67(1), 111–136. CrossRefGoogle Scholar
  2. Badino, H. (2004). A robust approach for ego-motion estimation using a mobile stereo platform. In Proc. int. workshop on complex motion (IWCM04) (pp. 198–208). Berlin: Springer. Google Scholar
  3. Black, M. J., & Anandan, P. (1996). The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63(1), 75–104. CrossRefGoogle Scholar
  4. Boykov, Y., & Kolmogorov, V. (2004). An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(9), 1124–1137. CrossRefGoogle Scholar
  5. Brox, T. (2005). From pixels to regions: partial differential equations in image analysis. PhD thesis, Faculty of Mathematics and Computer Science, Saarland University, Germany. Google Scholar
  6. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Proc. European conf. on computer vision (ECCV) (pp. 25–36). Berlin: Springer. Google Scholar
  7. Bruhn, A., & Weickert, J. (2006). Geometric properties for incomplete data: Vols. 283–298. A confidence measure for variational optic flow methods. Berlin: Springer. Google Scholar
  8. Bruhn, A., Weickert, J., Kohlberger, T., & Schnörr, C. (2005). Discontinuity preserving computation of variational optic flow in real-time. In Proc. int. conf. on scale-space (pp. 279–290). Berlin: Springer. Google Scholar
  9. Costeira, J., & Kanande, T. (1995). A multi-body factorization method for motion analysis. In Proc. IEEE int. conf. on computer vision (ICCV) (pp. 1071–1076). CrossRefGoogle Scholar
  10. Franke, U., & Joos, A. (2000). Real-time stereo vision for urban traffic scene understanding. In Proc. IEEE intelligent vehicles symposium (pp. 273–278). Dearborn: IEEE Computer Society. Google Scholar
  11. Gong, M. (2009). Real-time joint disparity and disparity flow estimation on programmable graphics hardware. Computer Vision and Image Understanding, 113(1), 90–100. CrossRefGoogle Scholar
  12. Gong, M., & Yang, Y. H. (2006). Disparity flow estimation using orthogonal reliability-based dynamic programming. In Proc. int. conf. on pattern recognition (ICPR) (pp. 70–73). Los Alamitos: IEEE Computer Society. Google Scholar
  13. Hartley, R. I., & Zisserman, A. (2000). Multiple view geometry in computer vision. Cambridge: Cambridge University Press. ISBN:0521623049. zbMATHGoogle Scholar
  14. Hirschmüller, H. (2006). Stereo vision in structured environments by consistent semi-globalmatching. In Proc. IEEE int. conf. on computer vision and pattern recognition (CVPR) (pp. 2386–2393). Los Alamitos: IEEE Computer Society. Google Scholar
  15. Hirschmüller, H. (2008). Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 328–341. CrossRefGoogle Scholar
  16. Horn, B., & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203. CrossRefGoogle Scholar
  17. Hu, X., & Mordohai, P. (2010). Evaluation of stereo condence indoors and outdoors. In Proc. IEEE int. conf. on computer vision and pattern recognition (CVPR). Los Alamitos: IEEE Computer Society. Google Scholar
  18. Huguet, F., & Devernay, F. (2007). A variational method for scene flow estimation from stereo sequences. In Proc. IEEE int. conf. on computer vision (ICCV) (pp. 1–7). Los Alamitos: IEEE Computer Society. http://www-prima.imag.fr/prima/pub/Publications/2007/HD07/. Google Scholar
  19. Isard, M., & MacCormick, J. (2006). Dense motion and disparity estimation via loopy belief propagation. In Lecture Notes in Computer Science: Vol. 3852. Asian conf. on computer vision (ACCV) (pp. 32–41). Berlin: Springer. http://dblp.uni-trier.de/db/conf/accv/accv2006-2.html#IsardM06. Google Scholar
  20. Kanatani, K., & Sugaya, Y. (2004). Multi-stage optimization for multi-body motion segmentation. IEICE Transactions on Information and Systems E87-D(7), 1935–1942. Google Scholar
  21. Kolmogorov, V., & Zabih, R. (2002). What energy functions can be minimized via graph cuts? In Proc. European conf. on computer vision (ECCV) (pp. 65–81). Google Scholar
  22. Mahalanobis, P. C. (1936). On the generalised distance in statistics. In Proc. of the National Institute of Science of India 12 (pp. 49–55). Google Scholar
  23. Mémin, E., & Pérez, P. (1998). Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Transactions on Image Processing, 7(5), 703–719. CrossRefGoogle Scholar
  24. Min, D., & Sohn, K. (2006). Edge-preserving simultaneous joint motion-disparity estimation. In Proc. int. conf. on pattern recognition (ICPR) (pp. 74–77). Washington: IEEE Computer Society. doi: 10.1109/ICPR.2006.470. Google Scholar
  25. Patras, I., Hendriks, E., & Tziritas, G. (1996). A joint motion/disparity estimation method for the construction of stereo interpolated images in stereoscopic image sequences. In Proc. int. conf. on pattern recognition (ICPR) (p. 359). Heijen: IEEE Computer Society. CrossRefGoogle Scholar
  26. Pons, J. P., Keriven, R., & Faugeras, O. (2007). Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. International Journal of Computer Vision, 72(2), 179–193. doi: 10.1007/s11263-006-8671-5. CrossRefGoogle Scholar
  27. Rabe, C., Franke, U., & Gehrig, S. (2007). Fast detection of moving objects in complex scenarios. In Proc. IEEE intelligent vehicles symposium (pp. 398–403). Los Alamitos: IEEE Computer Society. CrossRefGoogle Scholar
  28. Rao, S. R., Tron, R., Vidal, R., & Ma, Y. (2008). Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories. In Proc. IEEE int. conf. on computer vision and pattern recognition (CVPR). Google Scholar
  29. Rudin, L., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259–268. zbMATHCrossRefGoogle Scholar
  30. Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondencealgorithms. In Proc. IEEE int. conf. on computer vision (ICCV) (pp. 7–42). Los Alamitos: IEEE Computer Society. Google Scholar
  31. Shimizu, M., & Okutomi, M. (2001). Precise sub-pixel estimation on area-based matching. In Proc. IEEE int. conf. on computer vision (ICCV) (pp. 90–97). Los Alamitos: IEEE Computer Society. Google Scholar
  32. Stein, F. (2004). Efficient computation of optical flow using the census transform. In Proc. DAGM (pattern recognition) (pp. 79–86). Berlin: Springer. Google Scholar
  33. Stüben, K., & Trottenberg, U. (1982). Lecture notes in mathematics: Vol. 960. Multigrid methods: fundamental algorithms, model problem analysis and applications. Berlin: Springer. Google Scholar
  34. Tomasi, C., & Kanade, T. (1991). Detection and tracking of point features. Tech. Rep. CMU-CS-91-132, Carnegie Mellon University. citeseer.ist.psu.edu/tomasi91detection.html.
  35. University of Auckland (2008). .enpeda. Image Sequence Analysis Test Site (EISATS). http://www.mi.auckland.ac.nz/EISATS/.
  36. Vaudrey, T., Rabe, C., Klette, R., & Milburn, J. (2008). Differences between stereo and motion behaviour on synthetic and real-world stereo sequences. In Proc. int. conf. on image and vision computing New Zealand (IVCNZ). Los Alamitos: IEEE Computer Society. IEEE Xplore:10.1109/IVCNZ.2008.4762133. Google Scholar
  37. Vaudrey, T., Morales, S., Wedel, A., & Klette, R. (2010). Generalised residual images’ effect on illumination artifact removal for correspondence algorithms. Pattern Recognition. doi: 10.1016/j.patcog.2010.05.036.
  38. Vedula, S., Baker, S., Rander, P., Collins, R., & Kanade, T. (2005). Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3), 475–480. CrossRefGoogle Scholar
  39. Wedel, A., Pock, T., Zach, C., Cremers, D., & Bischof, H. (2008a). An improved algorithm for TV-L1 optical flow. In Revised papers int. Dagstuhl seminar on statistical and geometrical approaches to visual motion analysis (pp. 23–45). Berlin: Springer. Google Scholar
  40. Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., & Cremers, D. (2008b). Efficient dense scene flow from sparse or dense stereo data. In Proc. European conf. on computer vision (ECCV) (pp. 739–751). Berlin: Springer. Google Scholar
  41. Wedel, A., Vaudrey, T., Meissner, A., Rabe, C., Brox, T., Franke, U., & Cremers, D. (2008c). An evaluation approach for scene flow with decoupled motion and position. In Revised papers int. Dagstuhl seminar on statistical and geometrical approaches to visual motion analysis (pp. 46–69). Berlin: Springer. Google Scholar
  42. Wedel, A., Rabe, C., Meissner, A., Franke, U., & Cremers, D. (2009). Detection and segmentation of independently moving objects from dense scene flow. In D. Cremers, Y. Boykov, A. Blake, & F. R. Schmidt (Eds.), Energy minimization methods in computer vision and pattern recognition (EMMCVPR) (Vol. 5681, pp. 14–27). Berlin: Springer. CrossRefGoogle Scholar
  43. Werlberger, M., Pock, T., & Bischof, H. (2010). Motion estimation with non-local total variation regularization. In Proc. IEEE int. conf. on computer vision and pattern recognition (CVPR). Los Alamitos: IEEE Computer Society. Google Scholar
  44. Yan, J., & Pollefeys, M. (2006). A general framework for motion segmentation: independent, articulated, rigid, non-rigid, degenerate and non-degenerate. In LNCS: Vol. 3954. Proc. European conf. on computer vision (ECCV) (pp. 94–106). Berlin: Springer. Google Scholar
  45. Young, D. M. (1971). Iterative solution of large linear systems. New York: Academic Press. zbMATHGoogle Scholar
  46. Zach, C., Pock, T., & Bischof, H. (2007). A duality based approach for realtime tv-L 1 optical flow. In Proc. DAGM (pattern recognition) (pp. 214–223). Berlin: Springer. Google Scholar
  47. Zhang, Y., & Kambhamettu, C. (2001). On 3d scene flow and structure estimation. In Proc. IEEE int. conf. on computer vision and pattern recognition (CVPR) (Vol. 2, pp. 778–778). Los Alamitos: IEEE Computer Society. doi: 10.1109/CVPR.2001.991044. Google Scholar
  48. Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H. P. (2009). Complementary optic flow. In Energy minimization methods in computer vision and pattern recognition (EMMCVPR) (pp. 207–220). Berlin: Springer. CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Andreas Wedel
    • 1
  • Thomas Brox
    • 2
  • Tobi Vaudrey
    • 3
  • Clemens Rabe
    • 1
  • Uwe Franke
    • 1
  • Daniel Cremers
    • 4
  1. 1.GR/PAADaimler ResearchSindelfingenGermany
  2. 2.Albert-Ludwigs-UniversityFreiburgGermany
  3. 3.University of AucklandAucklandNew Zealand
  4. 4.Department of Computer ScienceTechnical University of MunichMunichGermany

Personalised recommendations