Computational Schlieren Photography with Light Field Probes


We introduce a new approach to capturing refraction in transparent media, which we call light field background oriented Schlieren photography. By optically coding the locations and directions of light rays emerging from a light field probe, we can capture changes of the refractive index field between the probe and a camera or an observer. Our prototype capture setup consists of inexpensive off-the-shelf hardware, including inkjet-printed transparencies, lenslet arrays, and a conventional camera. By carefully encoding the color and intensity variations of 4D light field probes, we show how to code both spatial and angular information of refractive phenomena. Such coding schemes are demonstrated to allow for a new, single image approach to reconstructing transparent surfaces, such as thin solids or surfaces of fluids. The captured visual information is used to reconstruct refractive surface normals and a sparse set of control points independently from a single photograph.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20


  1. 1.

    The connection between Snell’s law and the ray equation of geometric optics (Eq. 1) is well-known in physics, see e.g.,_particular_case_of_electromagnetism.


  1. Agarwal, S., Mallick, S.P., Kriegman, D., & Belongie, S. (2004). On refractive optical flow. In Proceeding of ECCV (pp. 483–494).

  2. Agrawal, A., Raskar, R., & Chellappa, R. (2006) What is the range of surface reconstructions from a gradient field? In Proceedings of ECCV (pp. 483–494).

  3. Arpa, A., Wetzstein, G., Lanman, D., & Raskar, R. (2012). Single lens off-chip cellphone microscopy. In Proceedings of IEEE PROCAMS.

  4. Atcheson, B., Heide, F., & Heidrich, W. (2010). CALTag: High precision fiducial markers for camera calibration. In Proceedings of VMV.

  5. Atcheson, B., Ihrke, I., Heidrich, W., Tevs, A., Bradley, D., Magnor, M., et al. (2008). Time-resolved 3D capture of non-stationary gas flows. ACM Transactions on Graphics (Siggraph Asia), 27, 132.

    Google Scholar 

  6. Ben-Ezra, M., & Nayar, S.K. (2003) What does motion reveal about transparency? In Proceeding of ICCV (pp. 1025–1032).

  7. Bonfort, T., Sturm, P., & Gargallo, P. (2006). General specular surface triangulation. In Proceedings of ACCV (pp. 872–881).

  8. Born, M., & Wolf, E. (1999). Principles of Optics (7th ed.). Cambridge: Cambridge University Press.

    Google Scholar 

  9. Dalziel, S., Hughes, G., & Sutherland, B. (2000). Whole-field density measurements by synthetic schlieren. Experiments in Fluids, 28, 322–335.

    Article  Google Scholar 

  10. Gao, C., & Ahuja, N. (2006). A refractive camera for acquiring stereo and super-resolution images. In Proceeding of CVPR (pp. 2316–2323).

  11. Hargather, M. J., & Settles, G. S. (2009). Natural-background-oriented Schlieren imaging. Experiments in Fluids, 48, 59–68.

    Article  Google Scholar 

  12. Hernandez, C., Vogiatzis, G., Brostow, G., Stenger, B., & Cipolla, R. (2007). Non-rigid photometric stereo with colored lights. In Proceeding of ICCV.

  13. Horovitz, I., & Kiryati, N. (2004). Depth from gradient fields and control points: Bias correction in photometric stereo. In Proceedings of ICIVC (pp. 681–694).

  14. Howes, W. L. (1984). Rainbow schlieren and its applications. Applied Optics, 23, 2449–2460.

    Article  Google Scholar 

  15. Hullin, M. B., Fuchs, M., Ihrke, I., Seidel, H. P., & Lensch, H. P. A. (2008). Fluorescent Immersion Range Scanning. ACM Transactions on Graphics (Siggraph), 87(1–87), 10.

    Google Scholar 

  16. Huynh, C. P., Robles-Kelly, A., & Hancock, E. (2010). Shape and refractive index recovery from single-view polarisation images. In Proceeding of CVPR.

  17. Ihrke, I., Goldluecke, B., & Magnor, M. (2005). Reconstructing the geometry of flowing water. In Proceedings of ICCV (pp. 1055– 1060).

  18. Ihrke, I., Kutulakos, K. N., Lensch, H. P. A., Magnor, M., & Heidrich, W. (2010). Transparent and specular object reconstruction. Computer Graphics Forum, 29, 2400–2426.

    Article  Google Scholar 

  19. Ihrke, I., Ziegler, G., Tevs, A., Theobalt, C., Magnor, M., & Seidel, H.P. (2007). Eikonal rendering: Efficient light transport in refractive objects. In Proceeding of SIGGRAPH. ACM Transactions on Graphics, 26:59.

  20. Kutulakos, K.N., & Steger, E. (2005) A theory of refractive and specular 3D shape by light-path triangulation. In Proceeding of ICCV (1448–1455).

  21. Levoy, M., Zhang, Z., & McDowall, I. (2009). Recording and controlling the 4D light field in a microscope. Journal of Microscopy, 235, 144–162.

    MathSciNet  Article  Google Scholar 

  22. Merzkirch, W. (1987). Flow visualization. New York: Academic Press.

    Google Scholar 

  23. Miyazaki, D., & Ikeuchi, K. (2005). Inverse polarization raytracing: Estimating surface shapes of transparent objects. In Proceeding of CVPR (pp. 910–917).

  24. Morris, N. J. W., & Kutulakos, K. N. (2005). Dynamic refraction stereo. In Proceedings of ICCV.

  25. Morris, N. J. W., & Kutulakos, K. N. (2007). Reconstructing the surface of inhomogeneous transparent scenes by scatter trace photography. In Proceedings of ICCV.

  26. Murase, H. (1990) Surface shape reconstruction of an undulating transparent object. In Proceedings of ICCV (pp. 313–317).

  27. Murphy, D. B. (2001). Fundamentals of light microscopy and electronic imaging. New York: Wiley.

    Google Scholar 

  28. Nehab, D., Rusinkiewicz, S., Davis, J., & Ramamoorthi, R. (2005). Efficiently combining positions and normals for precise 3D geometry. ACM Transactions on Graphics (Siggraph) 24.

  29. Ng, H. S., Wu, T. P., & Tang, C. K. (2010). Surface-from-gradients without discrete integrability enforcement: A Gaussian Kernel approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32, 2085–2099.

    Article  Google Scholar 

  30. Okano, F., Arai, J., Hoshino, H., & Yuyama, I. (1999). Three-dimensional video system based on integral photography. Optical Engineering, 38, 1072–1077.

    Article  Google Scholar 

  31. Savarese, S., & Perona, P. (2002). Local analysis for 3D reconstruction of specular surfaces—part II. In Proceedings of ECCV (pp. 759–774).

  32. Schardin, H. (1942). Die Schlierenverfahren und ihre Anwendungen. Ergebnisse der Exakten Naturwissenschaften. (English translation available as NASA TT F-12731), April 1970 (N70–25586) 20, 303–439.

  33. Settles, G. S. (2001). Schlieren and shadowgraph techniques. Cambridge: Cambridge University Press.

    Google Scholar 

  34. Settles, G. S. (2010). Important Developments in Schlieren and Shadowgraph Visualization during the Last Decade. In Proceedings of International Symposium on Flow Visualization (p. 267).

  35. Shimizu, M., & Okutomi, M. (2006) Reflection stereo—Novel monocular stereo using a transparent plate. In Proceeding of CRV (pp. 1–14).

  36. Tarini, M., Lensch, H. P., Goesele, M., & Seidel, H. P. (2005). 3D acquisition of mirroring objects using striped patterns. Graphical Models, 67, 233–259.

    Article  Google Scholar 

  37. Trifonov, B., Bradley, D., & Heidrich, W. (2006). Tomographic reconstruction of transparent objects. In Proceedings of EGSR (pp. 51–60).

  38. Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2012). Tensor displays: Compressive light field synthesis using multilayer displays with directional backlighting. ACM Transactions on Graphics (SIGGRAPH), 31, 1–11.

    Article  Google Scholar 

  39. Wetzstein, G., Raskar, R., & Heidrich, W. (2011) Hand-held schlieren photography with light field probes. In Proceeding of ICCP (pp. 1–8).

  40. Wetzstein, G., Roodnick, D., Raskar, R., & Heidrich, W. (2011). Refractive Shape from light field distortion. In International Conference on Computer Vision (ICCV).

  41. Woodham, R. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144.

    Google Scholar 

  42. Zhang, X., & Cox, C. S. (1994). Measuring the two-dimensional structure of a wavy water surface optically: A surface gradient detector. Experiments in Fluids, 17, 225–237.

    Article  MATH  Google Scholar 

  43. Zongker, D.E., Werner, D.M., Curless, B., & Salesin, D.H. (1999). Environment matting and compositing. In Proceeding of SIGGRAPH. Computer Graphics (pp. 205–214).

Download references


Gordon Wetzstein was supported by an NSERC Postdoctoral Fellowship. Ramesh Raskar was supported by an Alfred P. Sloan Research Fellowship and a DARPA Young Faculty Award.

Author information



Corresponding author

Correspondence to Gordon Wetzstein.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Wetzstein, G., Heidrich, W. & Raskar, R. Computational Schlieren Photography with Light Field Probes. Int J Comput Vis 110, 113–127 (2014).

Download citation


  • Computational photography
  • Light transport
  • Fluid imaging
  • Shape from x