International Journal of Computer Vision

, Volume 110, Issue 2, pp 113–127 | Cite as

Computational Schlieren Photography with Light Field Probes

  • Gordon WetzsteinEmail author
  • Wolfgang Heidrich
  • Ramesh Raskar


We introduce a new approach to capturing refraction in transparent media, which we call light field background oriented Schlieren photography. By optically coding the locations and directions of light rays emerging from a light field probe, we can capture changes of the refractive index field between the probe and a camera or an observer. Our prototype capture setup consists of inexpensive off-the-shelf hardware, including inkjet-printed transparencies, lenslet arrays, and a conventional camera. By carefully encoding the color and intensity variations of 4D light field probes, we show how to code both spatial and angular information of refractive phenomena. Such coding schemes are demonstrated to allow for a new, single image approach to reconstructing transparent surfaces, such as thin solids or surfaces of fluids. The captured visual information is used to reconstruct refractive surface normals and a sparse set of control points independently from a single photograph.


Computational photography Light transport Fluid imaging Shape from x 



Gordon Wetzstein was supported by an NSERC Postdoctoral Fellowship. Ramesh Raskar was supported by an Alfred P. Sloan Research Fellowship and a DARPA Young Faculty Award.


  1. Agarwal, S., Mallick, S.P., Kriegman, D., & Belongie, S. (2004). On refractive optical flow. In Proceeding of ECCV (pp. 483–494).Google Scholar
  2. Agrawal, A., Raskar, R., & Chellappa, R. (2006) What is the range of surface reconstructions from a gradient field? In Proceedings of ECCV (pp. 483–494).Google Scholar
  3. Arpa, A., Wetzstein, G., Lanman, D., & Raskar, R. (2012). Single lens off-chip cellphone microscopy. In Proceedings of IEEE PROCAMS.Google Scholar
  4. Atcheson, B., Heide, F., & Heidrich, W. (2010). CALTag: High precision fiducial markers for camera calibration. In Proceedings of VMV.Google Scholar
  5. Atcheson, B., Ihrke, I., Heidrich, W., Tevs, A., Bradley, D., Magnor, M., et al. (2008). Time-resolved 3D capture of non-stationary gas flows. ACM Transactions on Graphics (Siggraph Asia), 27, 132.Google Scholar
  6. Ben-Ezra, M., & Nayar, S.K. (2003) What does motion reveal about transparency? In Proceeding of ICCV (pp. 1025–1032).Google Scholar
  7. Bonfort, T., Sturm, P., & Gargallo, P. (2006). General specular surface triangulation. In Proceedings of ACCV (pp. 872–881).Google Scholar
  8. Born, M., & Wolf, E. (1999). Principles of Optics (7th ed.). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  9. Dalziel, S., Hughes, G., & Sutherland, B. (2000). Whole-field density measurements by synthetic schlieren. Experiments in Fluids, 28, 322–335.CrossRefGoogle Scholar
  10. Gao, C., & Ahuja, N. (2006). A refractive camera for acquiring stereo and super-resolution images. In Proceeding of CVPR (pp. 2316–2323).Google Scholar
  11. Hargather, M. J., & Settles, G. S. (2009). Natural-background-oriented Schlieren imaging. Experiments in Fluids, 48, 59–68.CrossRefGoogle Scholar
  12. Hernandez, C., Vogiatzis, G., Brostow, G., Stenger, B., & Cipolla, R. (2007). Non-rigid photometric stereo with colored lights. In Proceeding of ICCV.Google Scholar
  13. Horovitz, I., & Kiryati, N. (2004). Depth from gradient fields and control points: Bias correction in photometric stereo. In Proceedings of ICIVC (pp. 681–694).Google Scholar
  14. Howes, W. L. (1984). Rainbow schlieren and its applications. Applied Optics, 23, 2449–2460.CrossRefGoogle Scholar
  15. Hullin, M. B., Fuchs, M., Ihrke, I., Seidel, H. P., & Lensch, H. P. A. (2008). Fluorescent Immersion Range Scanning. ACM Transactions on Graphics (Siggraph), 87(1–87), 10.Google Scholar
  16. Huynh, C. P., Robles-Kelly, A., & Hancock, E. (2010). Shape and refractive index recovery from single-view polarisation images. In Proceeding of CVPR.Google Scholar
  17. Ihrke, I., Goldluecke, B., & Magnor, M. (2005). Reconstructing the geometry of flowing water. In Proceedings of ICCV (pp. 1055– 1060).Google Scholar
  18. Ihrke, I., Kutulakos, K. N., Lensch, H. P. A., Magnor, M., & Heidrich, W. (2010). Transparent and specular object reconstruction. Computer Graphics Forum, 29, 2400–2426.CrossRefGoogle Scholar
  19. Ihrke, I., Ziegler, G., Tevs, A., Theobalt, C., Magnor, M., & Seidel, H.P. (2007). Eikonal rendering: Efficient light transport in refractive objects. In Proceeding of SIGGRAPH. ACM Transactions on Graphics, 26:59.Google Scholar
  20. Kutulakos, K.N., & Steger, E. (2005) A theory of refractive and specular 3D shape by light-path triangulation. In Proceeding of ICCV (1448–1455).Google Scholar
  21. Levoy, M., Zhang, Z., & McDowall, I. (2009). Recording and controlling the 4D light field in a microscope. Journal of Microscopy, 235, 144–162.MathSciNetCrossRefGoogle Scholar
  22. Merzkirch, W. (1987). Flow visualization. New York: Academic Press.zbMATHGoogle Scholar
  23. Miyazaki, D., & Ikeuchi, K. (2005). Inverse polarization raytracing: Estimating surface shapes of transparent objects. In Proceeding of CVPR (pp. 910–917).Google Scholar
  24. Morris, N. J. W., & Kutulakos, K. N. (2005). Dynamic refraction stereo. In Proceedings of ICCV.Google Scholar
  25. Morris, N. J. W., & Kutulakos, K. N. (2007). Reconstructing the surface of inhomogeneous transparent scenes by scatter trace photography. In Proceedings of ICCV.Google Scholar
  26. Murase, H. (1990) Surface shape reconstruction of an undulating transparent object. In Proceedings of ICCV (pp. 313–317).Google Scholar
  27. Murphy, D. B. (2001). Fundamentals of light microscopy and electronic imaging. New York: Wiley.Google Scholar
  28. Nehab, D., Rusinkiewicz, S., Davis, J., & Ramamoorthi, R. (2005). Efficiently combining positions and normals for precise 3D geometry. ACM Transactions on Graphics (Siggraph) 24.Google Scholar
  29. Ng, H. S., Wu, T. P., & Tang, C. K. (2010). Surface-from-gradients without discrete integrability enforcement: A Gaussian Kernel approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32, 2085–2099.CrossRefGoogle Scholar
  30. Okano, F., Arai, J., Hoshino, H., & Yuyama, I. (1999). Three-dimensional video system based on integral photography. Optical Engineering, 38, 1072–1077.CrossRefGoogle Scholar
  31. Savarese, S., & Perona, P. (2002). Local analysis for 3D reconstruction of specular surfaces—part II. In Proceedings of ECCV (pp. 759–774).Google Scholar
  32. Schardin, H. (1942). Die Schlierenverfahren und ihre Anwendungen. Ergebnisse der Exakten Naturwissenschaften. (English translation available as NASA TT F-12731), April 1970 (N70–25586) 20, 303–439.Google Scholar
  33. Settles, G. S. (2001). Schlieren and shadowgraph techniques. Cambridge: Cambridge University Press.CrossRefzbMATHGoogle Scholar
  34. Settles, G. S. (2010). Important Developments in Schlieren and Shadowgraph Visualization during the Last Decade. In Proceedings of International Symposium on Flow Visualization (p. 267).Google Scholar
  35. Shimizu, M., & Okutomi, M. (2006) Reflection stereo—Novel monocular stereo using a transparent plate. In Proceeding of CRV (pp. 1–14).Google Scholar
  36. Tarini, M., Lensch, H. P., Goesele, M., & Seidel, H. P. (2005). 3D acquisition of mirroring objects using striped patterns. Graphical Models, 67, 233–259.CrossRefGoogle Scholar
  37. Trifonov, B., Bradley, D., & Heidrich, W. (2006). Tomographic reconstruction of transparent objects. In Proceedings of EGSR (pp. 51–60).Google Scholar
  38. Wetzstein, G., Lanman, D., Hirsch, M., & Raskar, R. (2012). Tensor displays: Compressive light field synthesis using multilayer displays with directional backlighting. ACM Transactions on Graphics (SIGGRAPH), 31, 1–11.CrossRefGoogle Scholar
  39. Wetzstein, G., Raskar, R., & Heidrich, W. (2011) Hand-held schlieren photography with light field probes. In Proceeding of ICCP (pp. 1–8).Google Scholar
  40. Wetzstein, G., Roodnick, D., Raskar, R., & Heidrich, W. (2011). Refractive Shape from light field distortion. In International Conference on Computer Vision (ICCV). Google Scholar
  41. Woodham, R. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144.Google Scholar
  42. Zhang, X., & Cox, C. S. (1994). Measuring the two-dimensional structure of a wavy water surface optically: A surface gradient detector. Experiments in Fluids, 17, 225–237.CrossRefzbMATHGoogle Scholar
  43. Zongker, D.E., Werner, D.M., Curless, B., & Salesin, D.H. (1999). Environment matting and compositing. In Proceeding of SIGGRAPH. Computer Graphics (pp. 205–214).Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Gordon Wetzstein
    • 1
    Email author
  • Wolfgang Heidrich
    • 2
  • Ramesh Raskar
    • 1
  1. 1.MIT Media LabCambridgeUSA
  2. 2.University of British ColumbiaVancouverCanada

Personalised recommendations