International Journal of Computer Vision

, Volume 107, Issue 2, pp 123–138 | Cite as

Decomposing Global Light Transport Using Time of Flight Imaging

  • Di Wu
  • Andreas VeltenEmail author
  • Matthew O’Toole
  • Belen Masia
  • Amit Agrawal
  • Qionghai Dai
  • Ramesh Raskar


Global light transport is composed of direct and indirect components. In this paper, we take the first steps toward analyzing light transport using the high temporal resolution information of time of flight (ToF) images. With pulsed scene illumination, the time profile at each pixel of these images separates different illumination components by their finite travel time and encodes complex interactions between the incident light and the scene geometry with spatially-varying material properties. We exploit the time profile to decompose light transport into its constituent direct, subsurface scattering, and interreflection components. We show that the time profile is well modelled using a Gaussian function for the direct and interreflection components, and a decaying exponential function for the subsurface scattering component. We use our direct, subsurface scattering, and interreflection separation algorithm for five computer vision applications: recovering projective depth maps, identifying subsurface scattering objects, measuring parameters of analytical subsurface scattering models, performing edge detection using ToF images and rendering novel images of the captured scene with adjusted amounts of subsurface scattering.


Light transport analysis Direct/global separation  Time of flight imaging Transient imaging Femto-photography 



The work of the MIT affiliated coauthors was funded by the Media Lab Consortium Members, DARPA through the DARPA YFA grant, and the Institute for Soldier Nanotechnologies and U.S. Army Research Office under contract W911NF-07-D-0004. The work of the Tsinghua affiliated coauthors was supported by the National Basic Research Project (No.2010CB731800) of China and the Key Project of NSFC (No. 61120106003 and 60932007). O’Toole received the support of the NSERC PGS-D and GRAND NCE programs. Masia was additionally funded by an FPU grant, project TIN2010-21543 and an NVIDIA Graduate Fellowship.

Supplementary material

Supplementary material 1 (mp4 4953 KB)

Supplementary material 2 (mp4 3233 KB)

Supplementary material 3 (mp4 3881 KB)

Supplementary material 4 (mp4 12513 KB)

Supplementary material 5 (mp4 9305 KB)

Supplementary material 6 (mp4 3621 KB)


  1. Abramson, N. (1978). Light-in-flight recording by holography. Optics Letters, 3(4), 121–123.CrossRefGoogle Scholar
  2. Arvo, J. (1993). Transfer equations in global illumination. SIGGRAPH Course Notes.Google Scholar
  3. Bai, J., Chandraker, M., Ng, T., & Ramamoorthi, R. (2010). A dual theory of inverse and forward light transport. European Conference on Computer Vision, 2010, 294–307.Google Scholar
  4. Brooker, G. (2009). Introduction to sensors for ranging and imaging. Raleigh: Scitech.Google Scholar
  5. Busck, J., & Heiselberg, H. (2004). Gated viewing and high-accuracy three-dimensional laser radar. Applied Optics, 43(24), 4705–4710.CrossRefGoogle Scholar
  6. Chen, T., Lensch, H., Fuchs, C., & Seidel, H. (2007). Polarization and phase-shifting for 3d scanning of translucent objects. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1–8).Google Scholar
  7. Gleckler, A., & Gelbart, A. (2000). Multiple-slit streak tube imaging lidar (ms-stil) applications. Proceedings SPIE (Vol. 4035, pp. 266–278).Google Scholar
  8. Gupta, M., Agrawal, A., Veeraraghavan, A. & Narasimhan, S. (2011). Structured light 3d scanning in the presence of global illumination. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 713–720).Google Scholar
  9. Gupta, M., Tian, Y., Narasimhan, S., & Zhang, L. (2009). (De) focusing on global light transport for active scene recovery. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2969–2976).Google Scholar
  10. Gupta, O., Willwacher, T., Velten, A., Veeraraghavan, A., & Raskar, R. (2012). Reconstruction of hidden 3d shapes using diffuse reflections. Optics Express, 20(17), 19096–19108.CrossRefGoogle Scholar
  11. Hamamatsu. (2012). Guide to streak cameras.
  12. Heide, F., Hullin, M., Gregson, J., & Heidrich, W. (2013). Low-budget transient imaging using photonic mixer devices. Proceedings of ACM SIGGRAPH, ACM Transactions on Graphics, 32(4), 146:1–146:12.Google Scholar
  13. Holroyd, M., Lawrence, J., & Zickler, T. (2010). A coaxial optical scanner for synchronous acquisition of 3d geometry and surface reflectance. ACM Transactions on Graphics, 29(4), 99.CrossRefGoogle Scholar
  14. Huang, D., Swanson, E., Lin, C., Schuman, J., Stinson, W., Chang, W., et al. (1991). Optical coherence tomography. Science, 254(5035), 1178–1181.CrossRefGoogle Scholar
  15. Jarabo, A., Masia, B., Velten, A., Barsi, C., Raskar, R., & Gutierrez, D. (2013). Rendering relativistic effects in transient imaging. Proceedings of CEIG.Google Scholar
  16. Kajiya, J. (1986). The rendering equation. ACM SIGGRAPH Computer Graphics, 20(4), 143–150.CrossRefGoogle Scholar
  17. Liu, S., Ng, T., & Matsushita, Y. (2010). Shape from second-bounce of light transport. European Conference on Computer Vision, 2010, 280–293.Google Scholar
  18. Mukaigawa, Y., Yagi, Y., & Raskar, R. (2010). Analysis of light transport in scattering media. Proceedings of Computer Vision and Pattern Recognition (CVPR) (pp. 153–160).Google Scholar
  19. Naik, N., Zhao, S., Velten, A., Raskar, R., & Bala, K. (2011). Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Transaction on Graphics, 30(6), 171.CrossRefGoogle Scholar
  20. Nayar, S., Krishnan, G., Grossberg, M., & Raskar, R. (2006). Fast separation of direct and global components of a scene using high frequency illumination. ACM Transaction on Graphics, 25(3), 935–944.CrossRefGoogle Scholar
  21. Ng, T., Pahwa, R., Bai, J., Tan, K., & Ramamoorthi, R. (2012). From the rendering equation to stratified light transport inversion. International Journal of Computer Vision, 96(2), 235–251.Google Scholar
  22. O’Toole, M., Raskar, R., & Kutulakos, K. (2012). Primal-dual coding to probe light transport. ACM Transactions on Graphics, 31(4), 39.Google Scholar
  23. Pandharkar, R., Velten, A., Bardagjy, A., Lawson, E., Bawendi, M., & Raskar, R. (2011). Estimating motion and size of moving non-line-of-sight objects in cluttered environments. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 265–272). Google Scholar
  24. Reddy, D., Ramamoorthi, R., & Curless, B. (2012). Frequency-space decomposition and acquisition of light transport under spatially varying illumination. European Conference on Computer Vision (pp. 596–610).Google Scholar
  25. Repasi, E., Lutzmann, P., Steinvall, O., Elmqvist, M., Göhler, B., & Anstett, G. (2009). Advanced short-wavelength infrared range-gated imaging for ground applications in monostatic and bistatic configurations. Applied Optics, 48(31), 5956–5969.CrossRefGoogle Scholar
  26. Seitz, S., Matsushita, Y., & Kutulakos, K. (2005). A theory of inverse light transport. Tenth IEEE International Conference on Computer Vision, (Vol. 2, pp. 1440–1447).Google Scholar
  27. Sen, P., Chen, B., Garg, G., Marschner, S., Horowitz, M., Levoy, M., et al. (2005). Dual photography. ACM Transactions on Graphics, 24(3), 745–755.CrossRefGoogle Scholar
  28. Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M. G., & Raskar, R. (2012). Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communication, 3, 745–758.CrossRefGoogle Scholar
  29. Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., et al. (2012). Relativistic ultrafast rendering using time-of-flight imaging. In ACM SIGGRAPH Talks, 41(1).Google Scholar
  30. Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., & Raskar, R. (2013). Femto-photography: Capturing and visualizing the propagation of light. Proceedindgs of SIGGRAPH, ACM Transaction on Graphics 32(4).Google Scholar
  31. Wang, L., & Wu, H. (2007). Biomedical optics: Principles and imaging. Hoboken, NJ: Wiley.Google Scholar
  32. Wetzstein, G., & Bimber, O. (2007). Radiometric compensation through inverse light transport. Proceedings of the 15th Pacific Conference on Computer Graphics and Applications (pp. 391–399).Google Scholar
  33. Wu, D., Wetzstein, G., Barsi, C., Willwacher, T., O’Toole, M., Naik, N., et al. (2012). Frequency analysis of transient light transport with applications in bare sensor imaging. European Conference on Computer Vision, 2012, 542–555.Google Scholar
  34. Wyant, J. (2002). White light interferometry. In AeroSense 2002 (pp. 98–107). Bellingham, WA: International Society for Optics and Photonics.Google Scholar
  35. Xia, H., & Zhang, C. (2009). Ultrafast ranging lidar based on real-time Fourier transformation. Optics Letters, 34, 2108–2110.CrossRefGoogle Scholar
  36. Zhang, L., & Nayar, S. (2006). Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 25(3), 907–915.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Di Wu
    • 1
    • 2
  • Andreas Velten
    • 3
    • 4
    Email author
  • Matthew O’Toole
    • 1
    • 5
  • Belen Masia
    • 1
    • 6
  • Amit Agrawal
    • 7
  • Qionghai Dai
    • 8
  • Ramesh Raskar
    • 1
  1. 1.MIT Media LabCambridgeUSA
  2. 2.Tsinghua UniversityBeijingChina
  3. 3.Medical EngineeringMorgridge Institute for ResearchMadisonUSA
  4. 4.Laboratory for Optical and Computational InstrumentationUniversity of Wisconsin at MadisonMadisonUSA
  5. 5.University of TorontoTorontoCanada
  6. 6.Universidad de ZaragozaSaragossaSpain
  7. 7.Mitsubishi Electric Research LabsCambridgeUSA
  8. 8.Department of AutomationTsinghua UniversityRoom 410, Central Main BuildingChina

Personalised recommendations