Advertisement

Intrinsic Video

  • Naejin Kong
  • Peter V. Gehler
  • Michael J. Black
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8690)

Abstract

Intrinsic images such as albedo and shading are valuable for later stages of visual processing. Previous methods for extracting albedo and shading use either single images or images together with depth data. Instead, we define intrinsic video estimation as the problem of extracting temporally coherent albedo and shading from video alone. Our approach exploits the assumption that albedo is constant over time while shading changes slowly. Optical flow aids in the accurate estimation of intrinsic video by providing temporal continuity as well as putative surface boundaries. Additionally, we find that the estimated albedo sequence can be used to improve optical flow accuracy in sequences with changing illumination. The approach makes only weak assumptions about the scene and we show that it substantially outperforms existing single-frame intrinsic image methods. We evaluate this quantitatively on synthetic sequences as well on challenging natural sequences with complex geometry, motion, and illumination.

Keywords

intrinsic images video temporal coherence optical flow 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

978-3-319-10605-2_24_MOESM1_ESM.pdf (23.3 mb)
Electronic Supplementary Material (PDF 23,849 KB)

References

  1. 1.
    Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. International Journal of Computer Vision (IJCV) 92(1), 1–31 (2011)CrossRefGoogle Scholar
  2. 2.
    Barron, J.T., Malik, J.: Color constancy, intrinsic images, and shape estimation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part IV. LNCS, vol. 7575, pp. 57–70. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  3. 3.
    Barron, J.T., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 334–341 (2012)Google Scholar
  4. 4.
    Barron, J.T., Malik, J.: Intrinsic scene properties from a single RGB-D image. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17–24 (2013)Google Scholar
  5. 5.
    Barrow, H.G., Tenenbaum, J.M.: Recovering intrinsic scene characteristics from images. In: Computer Vision Systems, pp. 3–26 (1978)Google Scholar
  6. 6.
    Bell, M., Freeman, W.T.: Learning local evidence for shading and reflectance. In: Proc. IEEE International Conference on Computer Vision (ICCV), pp. 670–677 (2001)Google Scholar
  7. 7.
    Black, M.J., Anandan, P.: The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63(1), 75–104 (1996)CrossRefGoogle Scholar
  8. 8.
    Bousseau, A., Paris, S., Durand, F.: User-assisted intrinsic images. ACM Trans. Graphics (TOG) – Proc. SIGGRAPH Asia 28(5), 130:1–130:10 (2009)Google Scholar
  9. 9.
    Bruhn, A., Weickert, J., Schnorr, C.: Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. International Journal of Computer Vision (IJCV) 61, 211–231 (2005)Google Scholar
  10. 10.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  11. 11.
    Chen, Q., Koltun, V.: A simple model for intrinsic image decomposition with depth cues. In: Proc. IEEE International Conference on Computer Vision (ICCV), pp. 241–248 (2013)Google Scholar
  12. 12.
    Gehler, P., Rother, C., Kiefel, M., Zhang, L., Schölkopf, B.: Recovering intrinsic images with a global sparsity prior on reflectance. In: Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F.C.N., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems (NIPS), pp. 765–773 (2011)Google Scholar
  13. 13.
    Grosse, R., Johnson, M.K., Adelson, E.H., Freeman, W.T.: Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In: Proc. IEEE International Conference on Computer Vision (ICCV), pp. 2335–2342 (2009)Google Scholar
  14. 14.
    Hauagge, D., Wehrwein, S., Bala, K., Snavely, N.: Photometric ambient occlusion. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2515–2522 (2013)Google Scholar
  15. 15.
    Horn, B.K.P., Schunk, B.G.: Determining optical flow. Artificial Intelligence 17, 185–203 (1981)CrossRefGoogle Scholar
  16. 16.
    Horn, B.K.P.: Determining lightness from an image. Computer Graphics and Image Processing 3(1), 277–299 (1974)CrossRefGoogle Scholar
  17. 17.
    Laffont, P.Y., Bousseau, A., Drettakis, G.: Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Transactions on Visualization and Computer Graphics 19(2), 210–224 (2013)Google Scholar
  18. 18.
    Laffont, P.Y., Bousseau, A., Paris, S., Durand, F., Drettakis, G.: Coherent intrinsic images from photo collections. ACM Transactions on Graphics (TOG) – Proc. SIGGRAPH Asia 31(6), 202:1–202:11 (2012)Google Scholar
  19. 19.
    Land, E.H., McCann, J.J.: Lightness and Retinex theory. Journal of the Optical Society of America 61(1), 1–11 (1971)CrossRefGoogle Scholar
  20. 20.
    Lee, K.J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S.U., Tan, P., Lin, S.: Estimation of intrinsic image sequences from image+depth video. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 327–340. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  21. 21.
    Li, Y., Osher, S.: A new median formula with applications to PDE based denoising. Communications in Mathematical Sciences 7(3), 741–753 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  22. 22.
    Paris, S., Kornprobst, P., Tumblin, J., Durand, F.: Bilateral filtering: Theory and applications. Foundations and Trends in Computer Graphics and Vision 4(1), 1–73 (2009)CrossRefGoogle Scholar
  23. 23.
    Sand, P., Teller, S.: Particle video: Long-range motion estimation using point trajectories. International Journal of Computer Vision 80(1), 72–91 (2008)CrossRefGoogle Scholar
  24. 24.
    Shen, J., Yang, X., Jia, Y., Li, X.: Intrinsic images using optimization. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3481–3487. IEEE (2011)Google Scholar
  25. 25.
    Shen, L., Tan, P., Lin, S.: Intrinsic image decomposition with non-local texture cues. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–7 (2008)Google Scholar
  26. 26.
    Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal of Computer Vision (IJCV) 106(2), 115–137 (2014)CrossRefGoogle Scholar
  27. 27.
    Tappen, M.F., Adelson, E.H., Freeman, W.T.: Estimating intrinsic component images using non-linear regression. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. II: 1992–1999 (2006)Google Scholar
  28. 28.
    Tappen, M.F., Freeman, W.T., Adelson, E.H.: Recovering intrinsic images from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(9), 1459–1472 (2005)CrossRefGoogle Scholar
  29. 29.
    Weiss, Y.: Deriving intrinsic images from image sequences. In: Proc. IEEE International Conference on Computer Vision (ICCV), pp. II: 68–75 (2001)Google Scholar
  30. 30.
    Ye, G., Garces, E., Liu, Y., Dai, Q., Gutierrez, D.: Intrinsic video and applications. ACM Transactions on Graphics 33(4) (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Naejin Kong
    • 1
  • Peter V. Gehler
    • 1
  • Michael J. Black
    • 1
  1. 1.Max Planck Institute for Intelligent SystemsTübingenGermany

Personalised recommendations