On Refractive Optical Flow

  • Sameer Agarwal
  • Satya P. Mallick
  • David Kriegman
  • Serge Belongie
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3022)


This paper presents a novel generalization of the optical flow equation to the case of refraction, and it describes a method for recovering the refractive structure of an object from a video sequence acquired as the background behind the refracting object moves. By structure here we mean a representation of how the object warps and attenuates (or amplifies) the light passing through it. We distinguish between the cases when the background motion is known and unknown. We show that when the motion is unknown, the refractive structure can only be estimated up to a six-parameter family of solutions without additional sources of information. Methods for solving for the refractive structure are described in both cases. The performance of the algorithm is demonstrated on real data, and results of applying the estimated refractive structure to the task of environment matting and compositing are presented.


Background Image Environment Matting Attenuation Function Background Motion Specular Surface 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Irani, M., Rousso, B., Peleg, S.: Computing occluding and transparent motion. International Journal of Computer Vision 12, 5–16 (1994)CrossRefGoogle Scholar
  2. 2.
    Ju, S.X., Black, M.J., Jepson, A.D.: Skin and bones: Multilayer, locally affine, optical flow and regularization with transparency. In: CVPR 1996, pp. 307–314 (1996)Google Scholar
  3. 3.
    Levin, A., Zomet, A., Weiss, Y.: Learning to perceive transparency from the statistics of natural scenes. In: Neural Information Processing Systems (2002)Google Scholar
  4. 4.
    Schechner, Y., Kiryati, N., Shamir, J.: Blind recovery of transparent and semireflected scenes. In: CVPR 2000, vol. 1, pp. 38–43. IEEE Computer Society, Los Alamitos (2000)Google Scholar
  5. 5.
    Chuang, Y.Y., Zongker, D.E., Hindorff, J., Curless, B., Salesin, D.H., Szeliski, R.: Environment matting extensions: Towards higher accuracy and real-time capture. In: Proc. of ACM SIGGRAPH 2000, pp. 121–130. ACM Press, New York (2000)Google Scholar
  6. 6.
    Wexler, Y., Fitzgibbon, A.W., Zisserman, A.: Image-based environment matting. In: Proc. of the 13th Eurographics workshop on Rendering (2002)Google Scholar
  7. 7.
    Zongker, D.E., Werner, D.M., Curless, B., Salesin, D.H.: Environment matting and compositing. In: Proc. of ACM SIGGRAPH 1999, pp. 205–214. ACM Press, New York (1999)Google Scholar
  8. 8.
    Murase, H.: Surface shape reconstruction of a nonrigid transparent object using refraction and motion. IEEE Trans. on Pattern Analysis and Machine Intelligence 14, 1045–1052 (1992)CrossRefGoogle Scholar
  9. 9.
    Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artificial Intelligence 17 (1981)Google Scholar
  10. 10.
    Horn, B.: Robot Vision. MIT Press, Cambridge (1986)Google Scholar
  11. 11.
    Perona, P., Malik, J., Shiota, T.: Anisotropic Diffusion. In: Geometry-Driven Diffusion in Computer Vision, pp. 73–92. Kluwer, Amsterdam (1995)Google Scholar
  12. 12.
    Savarese, S., Perona, P.: Local analysis for 3D reconstruction of specular surfaces. In: CVPR 2001, pp. 738–745 (2001)Google Scholar
  13. 13.
    Savarese, S., Perona, P.: Local analysis for 3D reconstruction of specular surfaces: Part II. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 759–774. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  14. 14.
    Oren, M., Nayar, S.: A theory of specular surface geometry. IJCV 24, 105–124 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Sameer Agarwal
    • 1
  • Satya P. Mallick
    • 1
  • David Kriegman
    • 1
  • Serge Belongie
    • 1
  1. 1.University of CaliforniaSan Diego, La JollaUSA

Personalised recommendations