Motion-Aware Structured Light Using Spatio-Temporal Decodable Patterns

  • Yuichi Taguchi
  • Amit Agrawal
  • Oncel Tuzel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7576)


Single-shot structured light methods allow 3D reconstruction of dynamic scenes. However, such methods lose spatial resolution and perform poorly around depth discontinuities. Previous single-shot methods project the same pattern repeatedly; thereby spatial resolution is reduced even if the scene is static or has slowly moving parts. We present a structured light system using a sequence of shifted stripe patterns that is decodable both spatially and temporally. By default, our method allows single-shot 3D reconstruction with any of our projected patterns by using spatial windows. Moreover, the sequence is designed so as to progressively improve the reconstruction quality around depth discontinuities by using temporal windows.

Our method enables motion-aware reconstruction for each pixel: The best spatio-temporal window is automatically selected depending on the scene structure, motion, and the number of available images. This significantly reduces the number of pixels around discontinuities where depth cannot be recovered in traditional approaches. Our decoding scheme extends the adaptive window matching commonly used in stereo by incorporating temporal windows with 1D spatial windows. We demonstrate the advantages of our approach for a variety of scenarios including thin structures, dynamic scenes, and scenes containing both static and dynamic regions.


Structured light motion-aware 3D reconstruction spatio-temporal decoding adaptive window matching 


  1. 1.
    Posdamer, J., Altschuler, M.: Surface measurement by space-encoded projected beam systems. Computer Graphics and Image Processing 18, 1–17 (1982)CrossRefGoogle Scholar
  2. 2.
    Inokuchi, S., Sato, K., Matsuda, F.: Range imaging system for 3-D object recognition. In: Proc. Int’l Conf. Pattern Recognition (ICPR), pp. 806–808 (1984)Google Scholar
  3. 3.
    Altschuler, M., Altschuler, B., Dijaki, J., Tamburino, L., Woolford, B.: Robot vision by encoded light beams. Three-Dimensional Machine Vision 87, 97–149 (1987)CrossRefGoogle Scholar
  4. 4.
    Pagès, J., Salvi, J., Collewet, C., Forest, J.: Optimised De Bruijn patterns for one-shot shape acquisition. Image and Vision Computing 23, 707–720 (2005)CrossRefGoogle Scholar
  5. 5.
    Hügli, H., Maïtre, G.: Generation and use of color pseudo random sequences for coding structured light in active ranging. In: Proc. SPIE Industrial Inspection, vol. 1010, pp. 75–82 (1989)Google Scholar
  6. 6.
    Forster, F., Rummel, P., Lang, M., Radig, B.: The HISCORE camera: A real time three dimensional and color camera. In: Proc. IEEE Int’l Conf. Image Processing (ICIP), vol. 2, pp. 598–601 (2001)Google Scholar
  7. 7.
    Zhang, L., Curless, B., Seitz, S.M.: Rapid shape acquisition using color structured light and multi-pass dynamic programming. In: Proc. Int’l Symp. 3D Data Processing, Visualization, and Transmission (3DPVT), pp. 24–36 (2002)Google Scholar
  8. 8.
    Koninckx, T., Van Gool, L.: Real-time range acquisition by adaptive structured light. IEEE Trans. Pattern Anal. Mach. Intell. 28, 432–445 (2006)CrossRefGoogle Scholar
  9. 9.
    Salvi, J., Batlle, J., Mouaddib, E.: A robust-coded pattern projection for dynamic 3d scene measurement. Pattern Recognition Letters 19, 1055–1065 (1998)CrossRefGoogle Scholar
  10. 10.
    Sagawa, R., Ota, Y., Yagi, Y., Furukawa, R., Asada, N., Kawasaki, H.: Dense 3D reconstruction method using a single pattern for fast moving object. In: Proc. IEEE Int’l Conf. Computer Vision (ICCV), pp. 1779–1786 (2009)Google Scholar
  11. 11.
    Ulusoy, A.O., Calakli, F., Taubin, G.: One-shot scanning using De Bruijn spaced grids. In: Proc. Int’l Conf. 3-D Digital Imaging and Modeling (3DIM) (2009)Google Scholar
  12. 12.
    Griffin, P., Narasimhan, L., Yee, S.: Generation of uniquely encoded light patterns for range data acquisition. Pattern Recognition 25, 609–616 (1992)CrossRefGoogle Scholar
  13. 13.
    Morano, R., Ozturk, C., Conn, R., Dubin, S., Zietz, S., Nissanov, J.: Structured light using pseudorandom codes. IEEE Trans. Pattern Anal. Mach. Intell. 20, 322–327 (1998)CrossRefGoogle Scholar
  14. 14.
    Claes, K.: Structured light adapted to control a robot arm. PhD thesis, K.U.Leuven (2008)Google Scholar
  15. 15.
    Maurice, X., Graebling, P., Doignon, C.: A pattern framework driven by the hamming distance for structured light-based reconstruction with a single image. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 2497–2504 (2011)Google Scholar
  16. 16.
    Collins, R.T.: A space-sweep approach to true multi-image matching. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 358–363 (1996)Google Scholar
  17. 17.
    Salvi, J., Fernandez, S., Pribanic, T., Llado, X.: A state of the art in structured light patterns for surface profilometry. Pattern Recognition 43, 2666–2680 (2010)zbMATHCrossRefGoogle Scholar
  18. 18.
    Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from single depth images. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1297–1304 (2011)Google Scholar
  19. 19.
    Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., Fitzgibbon, A.: KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In: ACM Symp. User Interface Software and Technology, UIST (2011)Google Scholar
  20. 20.
    Ishii, I., Yamamoto, K., Doi, K., Tsuji, T.: High-speed 3D image acquisition using coded structured light projection. In: Proc. IEEE/RSJ Int’l Conf. Intelligent Robots and Systems (IROS), pp. 925–930 (2007)Google Scholar
  21. 21.
    Kanade, T., Gruss, A., Carley, L.R.: A very fast VLSI rangefinder. In: Proc. IEEE Int’l Conf. Robotics Automation (ICRA), vol. 2, pp. 1322–1329 (1991)Google Scholar
  22. 22.
    Curless, B., Levoy, M.: Better optical triangulation through spacetime analysis. In: Proc. IEEE Int’l Conf. Computer Vision (ICCV), pp. 987–994 (1995)Google Scholar
  23. 23.
    Gupta, M., Agrawal, A., Veeraraghavan, A., Narasimhan, S.G.: Flexible Voxels for Motion-Aware Videography. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 100–114. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  24. 24.
    Zhang, L., Curless, B., Seitz, S.M.: Spacetime stereo: Shape recovery for dynamic scenes. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 367–374 (2003)Google Scholar
  25. 25.
    Davis, J., Ramamoorthi, R., Rusinkiewicz, S.: Spacetime stereo: A unifying framework for depth from triangulation. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 359–366 (2003)Google Scholar
  26. 26.
    Kanade, T., Okutomi, M.: A stereo matching algorithm with an adaptive window: Theory and experiment. IEEE Trans. Pattern Anal. Mach. Intell. 16, 920–932 (1994)CrossRefGoogle Scholar
  27. 27.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int’l J. Computer Vision 47, 7–42 (2002)zbMATHCrossRefGoogle Scholar
  28. 28.
    Lewis, J.P.: Fast normalized cross-correlation. Vision Interface 10 (1995)Google Scholar
  29. 29.
    Lanman, D., Taubin, G.: Build your own 3D scanner: 3D photograhy for beginners. In: ACM SIGGRAPH 2009 Courses, pp. 1–87 (2009)Google Scholar
  30. 30.
    Caspi, D., Kiryati, N., Shamir, J.: Range imaging with adaptive color structured light. IEEE Trans. Pattern Anal. Mach. Intell. 20, 470–480 (1998)CrossRefGoogle Scholar
  31. 31.
    Gupta, M., Agrawal, A., Veeraraghavan, A., Narasimhan, S.G.: Structured light 3D scanning in the presence of global illumination. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 713–720 (2011)Google Scholar
  32. 32.
    Ikeuchi, K.: Generating an interpretation tree from a CAD model for 3D-object recognition in bin-picking tasks. Int’l J. Computer Vision 1, 145–165 (1987)CrossRefGoogle Scholar
  33. 33.
    Choi, C., Taguchi, Y., Tuzel, O., Liu, M.Y., Ramalingam, S.: Voting-based pose estimation for robotic assembly using a 3D sensor. In: Proc. IEEE Int’l Conf. Robotics Automation (ICRA), pp. 1724–1731 (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Yuichi Taguchi
    • 1
  • Amit Agrawal
    • 1
  • Oncel Tuzel
    • 1
  1. 1.Mitsubishi Electric Research Labs (MERL)CambridgeUSA

Personalised recommendations