Completed Dense Scene Flow in RGB-D Space

  • Yucheng WangEmail author
  • Jian Zhang
  • Zicheng  Liu
  • Qiang Wu
  • Philip Chou
  • Zhengyou Zhang
  • Yunde Jia
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9008)


Conventional scene flow containing only translational vectors is not able to model 3D motion with rotation properly. Moreover, the accuracy of 3D motion estimation is restricted by several challenges such as large displacement, noise, and missing data (caused by sensing techniques or occlusion). In terms of solution, there are two kinds of approaches: local approaches and global approaches. However, local approaches can not generate smooth motion field, and global approaches is difficult to handle large displacement motion. In this paper, a completed dense scene flow framework is proposed, which models both rotation and translation for general motion estimation. It combines both a local method and a global method considering their complementary characteristics to handle large displacement motion and enforce smoothness respectively. The proposed framework is applied on the RGB-D image space where the computation efficiency is further improved. According to the quantitative evaluation based on Middlebury dataset, our method outperforms other published methods. The improved performance is further confirmed on the real data acquired by Kinect sensor.



This work was supported by Microsoft Research, Redmond. We also acknowledge Minqi Li for recording the Kinect RGB-D data sequence in our experiment.


  1. 1.
    Hadfield, S., Bowden, R.: Scene particles: unregularized particle based scene flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 36, 564–576 (2014)CrossRefGoogle Scholar
  2. 2.
    Gottfried, J.-M., Fehr, J., Garbe, C.S.: Computing range flow from multi-modal Kinect data. In: Bebis, G., et al. (eds.) ISVC 2011, Part I. LNCS, vol. 6938, pp. 758–767. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  3. 3.
    Herbst, E., Ren, X., Fox, D.: Rgb-d flow: Dense 3-d motion estimation using color and depth. In: IEEE International Conference on Robotics and Automation (ICRA). IEEE (2013)Google Scholar
  4. 4.
    Quiroga, J., Devernay, F., Crowley, J.L., et al.: Local/global scene flow estimation. In: ICIP-IEEE International Conference on Image Processing (2013)Google Scholar
  5. 5.
    Zhang, X., Chen, D., Yuan, Z., Zheng, N.: Dense scene flow based on depth and multi-channel bilateral filter. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012, Part III. LNCS, vol. 7726, pp. 140–151. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  6. 6.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics-TOG 28, 24 (2009)Google Scholar
  7. 7.
    Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized patchmatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  8. 8.
    Korman, S., Avidan, S.: Coherency sensitive hashing. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1607–1614. IEEE (2011)Google Scholar
  9. 9.
    Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 500–513 (2011)CrossRefGoogle Scholar
  10. 10.
    Hornacek, M., Fitzgibbon, A., Carsten, R.: Sphereflow: 6 dof scene flow from rgb-d pairs. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2014)Google Scholar
  11. 11.
    Junejo, I.N., Dexter, E., Laptev, I., Perez, P.: View-independent action recognition from temporal self-similarities. IEEE Trans. Pattern Anal. Mach. Intell. 33, 172–185 (2011)CrossRefGoogle Scholar
  12. 12.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92, 1–31 (2011)CrossRefGoogle Scholar
  13. 13.
    Vedula, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. IEEE Trans. Pattern Anal. Mach. Intell. 27, 475–480 (2005)CrossRefGoogle Scholar
  14. 14.
    Vogel, C., Schindler, K., Roth, S.: Piecewise rigid scene flow. In: 2013 IEEE International Conference on Computer Vision (ICCV). IEEE (2013)Google Scholar
  15. 15.
    Basha, T., Moses, Y., Kiryati, N.: Multi-view scene flow estimation: a view centered variational approach. Int. J. Comput. Vis. 101, 6–21 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Huguet, F., Devernay, F.: A variational method for scene flow estimation from stereo sequences. In: IEEE 11th International Conference on Computer Vision, ICCV 2007, pp. 1–7. IEEE (2007)Google Scholar
  17. 17.
    HaCohen, Y., Shechtman, E., Goldman, D.B., Lischinski, D.: Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Graph. (TOG) 30, 70:1–70:9 (2011)CrossRefGoogle Scholar
  18. 18.
    Eshet, Y., Korman, S., Ofek, E., Avidan, S.: Dcsh-matching patches in rgbd images. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 89–96. IEEE (2013)Google Scholar
  19. 19.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)CrossRefGoogle Scholar
  20. 20.
    Lempitsky, V., Rother, C., Roth, S., Blake, A.: Fusion moves for markov random field optimization. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1392–1405 (2010)CrossRefGoogle Scholar
  21. 21.
    Boros, E., Hammer, P.L.: Pseudo-boolean optimization. Discrete Appl. Math. 123, 155–225 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  22. 22.
    Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1744–1757 (2012)CrossRefGoogle Scholar
  23. 23.
    Willimon, B., Hickson, S., Walker, I., Birchfield, S.: An energy minimization approach to 3d non-rigid deformable surface estimation using rgbd data. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2711–2717. IEEE (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Yucheng Wang
    • 1
    • 3
    Email author
  • Jian Zhang
    • 1
  • Zicheng  Liu
    • 2
  • Qiang Wu
    • 1
  • Philip Chou
    • 2
  • Zhengyou Zhang
    • 2
  • Yunde Jia
    • 3
  1. 1.Advanced Analytics InstituteUniversity of TechnologySydneyAustralia
  2. 2.Microsoft ResearchRedmondUSA
  3. 3.Beijing Lab of Intelligent Information TechnologyBeijing Institute of TechnologyBeijingChina

Personalised recommendations