Skip to main content
SpringerLink
Log in

Estimating the Natural Illumination Conditions from a Single Outdoor Image

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Given a single outdoor image, we present a method for estimating the likely illumination conditions of the scene. In particular, we compute the probability distribution over the sun position and visibility. The method relies on a combination of weak cues that can be extracted from different portions of the image: the sky, the vertical surfaces, the ground, and the convex objects in the image. While no single cue can reliably estimate illumination by itself, each one can reinforce the others to yield a more robust estimate. This is combined with a data-driven prior computed over a dataset of 6 million photos. We present quantitative results on a webcam dataset with annotated sun positions, as well as quantitative and qualitative results on consumer-grade photographs downloaded from Internet. Based on the estimated illumination, we show how to realistically insert synthetic 3-D objects into the scene, and how to transfer appearance across images while keeping the illumination consistent.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3), 239–257.

    Article  Google Scholar 

  • Bird, R. E. (1984). A simple spectral model for direct normal and diffuse horizontal irradiance. Solar Energy, 32, 461–471.

    Article  Google Scholar 

  • Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P. N., & Nayar, S. K. (2008). Face swapping: automatically replacing faces in photographs. ACM Transactions on Graphics (SIGGRAPH 2008), 27(3), 39:1–39:8.

    Google Scholar 

  • Blinn, J. F., & Newell, M. E. (1976). Texture and reflection in computer generated images. In Proceedings of ACM SIGGRAPH 1976.

    Google Scholar 

  • Buluswar, S. D., & Draper, B. A. (2002). Color models for outdoor machine vision. Computer Vision and Image Understanding, 85(2), 71–99.

    Article  MATH  Google Scholar 

  • Cavanagh, P. (2005). The artist as neuroscientist. Nature, 434, 301–307.

    Article  Google Scholar 

  • Chang, C.-C., & Lin, C.-J. (2001). Libsvm: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.

  • Chen, H., Belhumeur, P., & Jacobs, D. (2000). In search of illumination invariants. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Chong, H. Y., Gortler, S. J., & Zickler, T. (2008). A perception-based color space for illumination-invariant image processing. ACM Transactions on Graphics (SIGGRAPH 2008) (pp. 61:1–61:7).

  • Collins, M., Shapire, R., & Singer, Y. (2002). Logistic regression, adaboost and Bregman distances. Machine Learning, 48(1), 158–169.

    Article  Google Scholar 

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Dale, K., Johnson, M. K., Sunkavalli, K., Matusik, W., & Pfister, H. (2009). Image restoration using online photo collections. In International conference on computer vision.

    Google Scholar 

  • Debevec, P. (1998). Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of ACM SIGGRAPH 1998.

    Google Scholar 

  • Debevec, P., & Malik, J. (1997). Recovering high dynamic range radiance maps from photographs. In Proceedings of ACM SIGGRAPH 1997, August.

    Google Scholar 

  • Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., & Martinez, P. (2004). Estimating surface reflectance properties of a complex scene under captured natural illumination (Technical Report ICT-TR-06.2004, USC ICT).

  • Dollár, P., Wojek, C., Schiele, B., & Perona, P. (2009). Pedestrian detection: a benchmark. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Dror, R. O., Willsky, A. S., & Adelson, E. H. (2004). Statistical characterization of real-world illumination. Journal of Vision, 4, 821–837.

    Article  Google Scholar 

  • Felzenszwalb, P., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence (pp. 1627–1645).

  • Finlayson, G., Drew, M., & Funt, B. (1993). Diagonal transforms suffice for color constancy. In International conference on computer vision.

    Google Scholar 

  • Finlayson, G. D., Drew, M. S., & Lu, C. (2004). Intrinsic images by entropy minimization. In European conference on computer vision.

    Google Scholar 

  • Finlayson, G. D., Fredembach, C., & Drew, M. S. (2007). Detecting illumination in images. In IEEE international conference on computer vision.

    Google Scholar 

  • Finlayson, G. D., Hordley, S. D., & Drew, M. S. (2002). Removing shadows from images. In European conference on computer vision.

    Google Scholar 

  • Hays, J., & Efros, A. A. (2008). im2gps: estimating geographic information from a single image. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Healey, G., & Slater, D. (1999). Models and methods for automated material identification in hyperspectral imagery acquired under unknown illumination and atmospheric conditions. IEEE Transactions on Geoscience and Remote Sensing, 37(6), 2706–2717.

    Article  Google Scholar 

  • Hill, R. (1994). Theory of geolocation by light levels. In B. J. LeBouef & R. M. Laws (Eds.), Elephant seals: population ecology, behavior, and physiology (pp. 227–236). Berkeley: University of California Press. Chapter 12.

    Google Scholar 

  • Hoiem, D., Efros, A. A., & Hebert, M. (2005). Automatic photo pop-up. ACM Transactions on Graphics (SIGGRAPH 2005), 24(3), 577–584.

    Article  Google Scholar 

  • Hoiem, D., Efros, A. A., & Hebert, M. (2007a). Recovering surface layout from an image. International Journal of Computer Vision, 75(1), 151–172.

    Article  Google Scholar 

  • Hoiem, D., Stein, A., Efros, A. A., & Hebert, M. (2007b). Recovering occlusion boundaries from a single image. In IEEE international conference on computer vision.

    Google Scholar 

  • Jacobs, N., Roman, N., & Pless, R. (2007). Consistent temporal variations in many outdoor scenes. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Judd, D. B., Macadam, D. L., Wyszecki, G., Budde, H. W., Condit, H. R., Henderson, S. T., & Simonds, J. L. (1964). Spectral distribution of typical daylight as a function of correlated color temperature. Journal of the Optical Society of America A, 54(8), 1031–1036.

    Article  Google Scholar 

  • Junejo, I. N., & Foroosh, H. (2008). Estimating geo-temporal location of stationary cameras using shadow trajectories. In European conference on computer vision.

    Google Scholar 

  • Kasten, F., & Young, A. T. (1989). Revised optical air mass tables and approximation formula. Applied Optics, 28, 4735–4738.

    Article  Google Scholar 

  • Ke, Y., Tang, X., & Jing, F. (2006). The design of high-level features for photo quality assessment. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Khan, E. A., Reinhard, E., Fleming, R., & Büelthoff, H. (2006). Image-based material editing. ACM Transactions on Graphics (ACM SIGGRAPH 2006), August (pp. 654–663).

  • Kim, T., & Hong, K.-S. (2005). A practical single image based approach for estimating illumination distribution from shadows. In IEEE international conference on computer vision.

    Google Scholar 

  • Koenderink, J. J., van Doorn, A. J., & Pont, S. C. (2004). Light direction from shad(ow)ed random gaussian surfaces. Perception, 33(12), 1405–1420.

    Article  Google Scholar 

  • Košecká, J., & Zhang, W. (2002). Video compass. In European conference on computer vision.

    Google Scholar 

  • Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2009). Estimating natural illumination from a single outdoor image. In IEEE international conference on computer vision.

    Google Scholar 

  • Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2009). Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. ACM Transactions on Graphics (SIGGRAPH Asia 2009), 28(5), December (pp. 131:1–131:10).

  • Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2010). Detecting ground shadows in outdoor consumer photographs. In European conference on computer vision.

    Google Scholar 

  • Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2010). Ground shadow boundary dataset. http://graphics.cs.cmu.edu/projects/shadows, September.

  • Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., & Criminisi, A. (2007). Photo clip art. ACM Transactions on Graphics (SIGGRAPH 2007).

  • Lalonde, J.-F., Narasimhan, S. G., & Efros, A. A. (2010). What do the sun and the sky tell us about the camera? International Journal of Computer Vision, 88(1), 24–51.

    Article  Google Scholar 

  • Langer, M. S., & Büelthoff, H. H. (2001). A prior for global convexity in local shape-from-shading. Perception, 30(4), 403–410.

    Article  Google Scholar 

  • Li, Y., Lin, S., Lu, H., & Shum, H.-Y. (2003). Multiple-cue illumination estimation in textured scenes. In IEEE international conference on computer vision.

    Google Scholar 

  • Manduchi, R. (2006). Learning outdoor color classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11), 1713–1723.

    Article  Google Scholar 

  • Maxwell, B. A., Friedhoff, R. M., & Smith, C. A. (2008). A bi-illuminant dichromatic reflection model for understanding images. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Mills, D. (2004). Advances in solar thermal electricity and technology. Solar Energy, 76, 19–31.

    Article  Google Scholar 

  • Narasimhan, S. G., Ramesh, V., & Nayar, S. K. (2005). A class of photometric invariants: Separating material from shape and illumination. In IEEE international conference on computer vision.

    Google Scholar 

  • Park, D., Ramanan, D., & Fowlkes, C. C. (2010). Multiresolution models for object detection. In European conference on computer vision.

    Google Scholar 

  • Perez, R., Seals, R., & Michalsky, J. (1993). All-weather model for sky luminance distribution—preliminary configuration and validation. Solar Energy, 50(3), 235–245.

    Article  Google Scholar 

  • Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers.

  • Preetham, A. J., Shirley, P., & Smits, B. (1999). A practical analytic model for daylight. In Proceedings of ACM SIGGRAPH 1999, August.

    Google Scholar 

  • Ramamoorthi, R., & Hanrahan, P. (2001). A signal-processing framework for inverse rendering. In Proceedings of ACM SIGGRAPH 2001.

    Google Scholar 

  • Reda, I., & Andreas, A. (2005). Solar position algorithm for solar radiation applications (Technical Report NREL/TP-560-34302). National Renewable Energy Laboratory, November.

  • Reinhart, C. F., Mardaljevic, J., & Rogers, Z. (2006). Dynamic daylight performance metrics for sustainable building design. Leukos, 3(1), 1–25.

    Google Scholar 

  • Romeiro, F., & Zickler, T. (2010). Blind reflectometry. In European conference on computer vision.

    Google Scholar 

  • Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173.

    Article  Google Scholar 

  • Sato, I., Sato, Y., & Ikeuchi, K. (2003). Illumination from shadows. IEEE Transactions on Pattern Matching and Machine Intelligence, 25(3), 290–300.

    Article  Google Scholar 

  • Sato, Y., & Ikeuchi, K. (1995). Reflectance analysis under solar illumination. In Proceedings of the IEEE workshop on physics-based modeling and computer vision (pp. 180–187).

    Chapter  Google Scholar 

  • Slater, D., & Healey, G. (1998). Analyzing the spectral dimensionality of outdoor visible and near-infrared illumination functions. Journal of the Optical Society of America, 15(11), 2913–2920.

    Article  Google Scholar 

  • Stauffer, C. (1999). Adaptive background mixture models for real-time tracking. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Stumpfel, J., Jones, A., Wenger, A., Tchou, C., Hawkins, T., & Debevec, P. (2004). Direct HDR capture of the sun and sky. In Proceedings of AFRIGRAPH.

    Google Scholar 

  • Sun, M., Schindler, G., Turk, G., & Dellaert, F. (2009). Color matching and illumination estimation for urban scenes. In IEEE international workshop on 3-D digital imaging and modeling.

    Google Scholar 

  • Sunkavalli, K., Romeiro, F., Matusik, W., Zickler, T., & Pfister, H. (2008). What do color changes reveal about an outdoor scene. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Tian, J., Sun, J., & Tang, Y. (2009). Tricolor attenuation model for shadow detection. IEEE Transactions on Image Processing, 18(10), 2355–2363.

    Article  MathSciNet  Google Scholar 

  • Tsin, Y., Collins, R. T., Ramesh, V., & Kanade, T. (2001). Bayesian color constancy for outdoor object recognition. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Ward, G. (1994). The RADIANCE lighting simulation and rendering system. In Proceedings of ACM SIGGRAPH 1994.

    Google Scholar 

  • Weiss, Y. (2001). Deriving intrinsic images from image sequences. In IEEE international conference on computer vision.

    Google Scholar 

  • Wu, T.-P., & Tang, C.-K. (2005). A Bayesian approach for shadow extraction from a single image. In IEEE international conference on computer vision.

    Google Scholar 

  • Yu, Y., & Malik, J. (1998). Recovering photometric properties of architectural scenes from photographs. In Proceedings of ACM SIGGRAPH 1998, July.

    Google Scholar 

  • Zhu, J., Samuel, K. G. G., Masood, S. Z., & Tappen, M. F. (2010). Learning to recognize shadows in monochromatic natural images. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

  • Zickler, T., Mallick, S. P., Kriegman, D. J., & Belhumeur, P. N. (2006). Color subspaces as photometric invariants. In IEEE conference on computer vision and pattern recognition.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA

    Jean-François Lalonde, Alexei A. Efros & Srinivasa G. Narasimhan

Authors
  1. Jean-François Lalonde
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexei A. Efros
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Srinivasa G. Narasimhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jean-François Lalonde.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lalonde, JF., Efros, A.A. & Narasimhan, S.G. Estimating the Natural Illumination Conditions from a Single Outdoor Image. Int J Comput Vis 98, 123–145 (2012). https://doi.org/10.1007/s11263-011-0501-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-011-0501-8

Keywords

Search

Navigation