International Journal of Computer Vision

, Volume 98, Issue 2, pp 123–145 | Cite as

Estimating the Natural Illumination Conditions from a Single Outdoor Image

  • Jean-François Lalonde
  • Alexei A. Efros
  • Srinivasa G. Narasimhan
Article

Abstract

Given a single outdoor image, we present a method for estimating the likely illumination conditions of the scene. In particular, we compute the probability distribution over the sun position and visibility. The method relies on a combination of weak cues that can be extracted from different portions of the image: the sky, the vertical surfaces, the ground, and the convex objects in the image. While no single cue can reliably estimate illumination by itself, each one can reinforce the others to yield a more robust estimate. This is combined with a data-driven prior computed over a dataset of 6 million photos. We present quantitative results on a webcam dataset with annotated sun positions, as well as quantitative and qualitative results on consumer-grade photographs downloaded from Internet. Based on the estimated illumination, we show how to realistically insert synthetic 3-D objects into the scene, and how to transfer appearance across images while keeping the illumination consistent.

Keywords

Illumination estimation Data-driven methods Shadow detection Scene understanding Image synthesis 

References

  1. Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3), 239–257. CrossRefGoogle Scholar
  2. Bird, R. E. (1984). A simple spectral model for direct normal and diffuse horizontal irradiance. Solar Energy, 32, 461–471. CrossRefGoogle Scholar
  3. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P. N., & Nayar, S. K. (2008). Face swapping: automatically replacing faces in photographs. ACM Transactions on Graphics (SIGGRAPH 2008), 27(3), 39:1–39:8. Google Scholar
  4. Blinn, J. F., & Newell, M. E. (1976). Texture and reflection in computer generated images. In Proceedings of ACM SIGGRAPH 1976. Google Scholar
  5. Buluswar, S. D., & Draper, B. A. (2002). Color models for outdoor machine vision. Computer Vision and Image Understanding, 85(2), 71–99. MATHCrossRefGoogle Scholar
  6. Cavanagh, P. (2005). The artist as neuroscientist. Nature, 434, 301–307. CrossRefGoogle Scholar
  7. Chang, C.-C., & Lin, C.-J. (2001). Libsvm: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.
  8. Chen, H., Belhumeur, P., & Jacobs, D. (2000). In search of illumination invariants. In IEEE conference on computer vision and pattern recognition. Google Scholar
  9. Chong, H. Y., Gortler, S. J., & Zickler, T. (2008). A perception-based color space for illumination-invariant image processing. ACM Transactions on Graphics (SIGGRAPH 2008) (pp. 61:1–61:7). Google Scholar
  10. Collins, M., Shapire, R., & Singer, Y. (2002). Logistic regression, adaboost and Bregman distances. Machine Learning, 48(1), 158–169. CrossRefGoogle Scholar
  11. Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In IEEE conference on computer vision and pattern recognition. Google Scholar
  12. Dale, K., Johnson, M. K., Sunkavalli, K., Matusik, W., & Pfister, H. (2009). Image restoration using online photo collections. In International conference on computer vision. Google Scholar
  13. Debevec, P. (1998). Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of ACM SIGGRAPH 1998. Google Scholar
  14. Debevec, P., & Malik, J. (1997). Recovering high dynamic range radiance maps from photographs. In Proceedings of ACM SIGGRAPH 1997, August. Google Scholar
  15. Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., & Martinez, P. (2004). Estimating surface reflectance properties of a complex scene under captured natural illumination (Technical Report ICT-TR-06.2004, USC ICT). Google Scholar
  16. Dollár, P., Wojek, C., Schiele, B., & Perona, P. (2009). Pedestrian detection: a benchmark. In IEEE conference on computer vision and pattern recognition. Google Scholar
  17. Dror, R. O., Willsky, A. S., & Adelson, E. H. (2004). Statistical characterization of real-world illumination. Journal of Vision, 4, 821–837. CrossRefGoogle Scholar
  18. Felzenszwalb, P., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence (pp. 1627–1645). Google Scholar
  19. Finlayson, G., Drew, M., & Funt, B. (1993). Diagonal transforms suffice for color constancy. In International conference on computer vision. Google Scholar
  20. Finlayson, G. D., Drew, M. S., & Lu, C. (2004). Intrinsic images by entropy minimization. In European conference on computer vision. Google Scholar
  21. Finlayson, G. D., Fredembach, C., & Drew, M. S. (2007). Detecting illumination in images. In IEEE international conference on computer vision. Google Scholar
  22. Finlayson, G. D., Hordley, S. D., & Drew, M. S. (2002). Removing shadows from images. In European conference on computer vision. Google Scholar
  23. Hays, J., & Efros, A. A. (2008). im2gps: estimating geographic information from a single image. In IEEE conference on computer vision and pattern recognition. Google Scholar
  24. Healey, G., & Slater, D. (1999). Models and methods for automated material identification in hyperspectral imagery acquired under unknown illumination and atmospheric conditions. IEEE Transactions on Geoscience and Remote Sensing, 37(6), 2706–2717. CrossRefGoogle Scholar
  25. Hill, R. (1994). Theory of geolocation by light levels. In B. J. LeBouef & R. M. Laws (Eds.), Elephant seals: population ecology, behavior, and physiology (pp. 227–236). Berkeley: University of California Press. Chapter 12. Google Scholar
  26. Hoiem, D., Efros, A. A., & Hebert, M. (2005). Automatic photo pop-up. ACM Transactions on Graphics (SIGGRAPH 2005), 24(3), 577–584. CrossRefGoogle Scholar
  27. Hoiem, D., Efros, A. A., & Hebert, M. (2007a). Recovering surface layout from an image. International Journal of Computer Vision, 75(1), 151–172. CrossRefGoogle Scholar
  28. Hoiem, D., Stein, A., Efros, A. A., & Hebert, M. (2007b). Recovering occlusion boundaries from a single image. In IEEE international conference on computer vision. Google Scholar
  29. Jacobs, N., Roman, N., & Pless, R. (2007). Consistent temporal variations in many outdoor scenes. In IEEE conference on computer vision and pattern recognition. Google Scholar
  30. Judd, D. B., Macadam, D. L., Wyszecki, G., Budde, H. W., Condit, H. R., Henderson, S. T., & Simonds, J. L. (1964). Spectral distribution of typical daylight as a function of correlated color temperature. Journal of the Optical Society of America A, 54(8), 1031–1036. CrossRefGoogle Scholar
  31. Junejo, I. N., & Foroosh, H. (2008). Estimating geo-temporal location of stationary cameras using shadow trajectories. In European conference on computer vision. Google Scholar
  32. Kasten, F., & Young, A. T. (1989). Revised optical air mass tables and approximation formula. Applied Optics, 28, 4735–4738. CrossRefGoogle Scholar
  33. Ke, Y., Tang, X., & Jing, F. (2006). The design of high-level features for photo quality assessment. In IEEE conference on computer vision and pattern recognition. Google Scholar
  34. Khan, E. A., Reinhard, E., Fleming, R., & Büelthoff, H. (2006). Image-based material editing. ACM Transactions on Graphics (ACM SIGGRAPH 2006), August (pp. 654–663). Google Scholar
  35. Kim, T., & Hong, K.-S. (2005). A practical single image based approach for estimating illumination distribution from shadows. In IEEE international conference on computer vision. Google Scholar
  36. Koenderink, J. J., van Doorn, A. J., & Pont, S. C. (2004). Light direction from shad(ow)ed random gaussian surfaces. Perception, 33(12), 1405–1420. CrossRefGoogle Scholar
  37. Košecká, J., & Zhang, W. (2002). Video compass. In European conference on computer vision. Google Scholar
  38. Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2009). Estimating natural illumination from a single outdoor image. In IEEE international conference on computer vision. Google Scholar
  39. Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2009). Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. ACM Transactions on Graphics (SIGGRAPH Asia 2009), 28(5), December (pp. 131:1–131:10). Google Scholar
  40. Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2010). Detecting ground shadows in outdoor consumer photographs. In European conference on computer vision. Google Scholar
  41. Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2010). Ground shadow boundary dataset. http://graphics.cs.cmu.edu/projects/shadows, September.
  42. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., & Criminisi, A. (2007). Photo clip art. ACM Transactions on Graphics (SIGGRAPH 2007). Google Scholar
  43. Lalonde, J.-F., Narasimhan, S. G., & Efros, A. A. (2010). What do the sun and the sky tell us about the camera? International Journal of Computer Vision, 88(1), 24–51. CrossRefGoogle Scholar
  44. Langer, M. S., & Büelthoff, H. H. (2001). A prior for global convexity in local shape-from-shading. Perception, 30(4), 403–410. CrossRefGoogle Scholar
  45. Li, Y., Lin, S., Lu, H., & Shum, H.-Y. (2003). Multiple-cue illumination estimation in textured scenes. In IEEE international conference on computer vision. Google Scholar
  46. Manduchi, R. (2006). Learning outdoor color classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11), 1713–1723. CrossRefGoogle Scholar
  47. Maxwell, B. A., Friedhoff, R. M., & Smith, C. A. (2008). A bi-illuminant dichromatic reflection model for understanding images. In IEEE conference on computer vision and pattern recognition. Google Scholar
  48. Mills, D. (2004). Advances in solar thermal electricity and technology. Solar Energy, 76, 19–31. CrossRefGoogle Scholar
  49. Narasimhan, S. G., Ramesh, V., & Nayar, S. K. (2005). A class of photometric invariants: Separating material from shape and illumination. In IEEE international conference on computer vision. Google Scholar
  50. Park, D., Ramanan, D., & Fowlkes, C. C. (2010). Multiresolution models for object detection. In European conference on computer vision. Google Scholar
  51. Perez, R., Seals, R., & Michalsky, J. (1993). All-weather model for sky luminance distribution—preliminary configuration and validation. Solar Energy, 50(3), 235–245. CrossRefGoogle Scholar
  52. Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers. Google Scholar
  53. Preetham, A. J., Shirley, P., & Smits, B. (1999). A practical analytic model for daylight. In Proceedings of ACM SIGGRAPH 1999, August. Google Scholar
  54. Ramamoorthi, R., & Hanrahan, P. (2001). A signal-processing framework for inverse rendering. In Proceedings of ACM SIGGRAPH 2001. Google Scholar
  55. Reda, I., & Andreas, A. (2005). Solar position algorithm for solar radiation applications (Technical Report NREL/TP-560-34302). National Renewable Energy Laboratory, November. Google Scholar
  56. Reinhart, C. F., Mardaljevic, J., & Rogers, Z. (2006). Dynamic daylight performance metrics for sustainable building design. Leukos, 3(1), 1–25. Google Scholar
  57. Romeiro, F., & Zickler, T. (2010). Blind reflectometry. In European conference on computer vision. Google Scholar
  58. Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173. CrossRefGoogle Scholar
  59. Sato, I., Sato, Y., & Ikeuchi, K. (2003). Illumination from shadows. IEEE Transactions on Pattern Matching and Machine Intelligence, 25(3), 290–300. CrossRefGoogle Scholar
  60. Sato, Y., & Ikeuchi, K. (1995). Reflectance analysis under solar illumination. In Proceedings of the IEEE workshop on physics-based modeling and computer vision (pp. 180–187). CrossRefGoogle Scholar
  61. Slater, D., & Healey, G. (1998). Analyzing the spectral dimensionality of outdoor visible and near-infrared illumination functions. Journal of the Optical Society of America, 15(11), 2913–2920. CrossRefGoogle Scholar
  62. Stauffer, C. (1999). Adaptive background mixture models for real-time tracking. In IEEE conference on computer vision and pattern recognition. Google Scholar
  63. Stumpfel, J., Jones, A., Wenger, A., Tchou, C., Hawkins, T., & Debevec, P. (2004). Direct HDR capture of the sun and sky. In Proceedings of AFRIGRAPH. Google Scholar
  64. Sun, M., Schindler, G., Turk, G., & Dellaert, F. (2009). Color matching and illumination estimation for urban scenes. In IEEE international workshop on 3-D digital imaging and modeling. Google Scholar
  65. Sunkavalli, K., Romeiro, F., Matusik, W., Zickler, T., & Pfister, H. (2008). What do color changes reveal about an outdoor scene. In IEEE conference on computer vision and pattern recognition. Google Scholar
  66. Tian, J., Sun, J., & Tang, Y. (2009). Tricolor attenuation model for shadow detection. IEEE Transactions on Image Processing, 18(10), 2355–2363. MathSciNetCrossRefGoogle Scholar
  67. Tsin, Y., Collins, R. T., Ramesh, V., & Kanade, T. (2001). Bayesian color constancy for outdoor object recognition. In IEEE conference on computer vision and pattern recognition. Google Scholar
  68. Ward, G. (1994). The RADIANCE lighting simulation and rendering system. In Proceedings of ACM SIGGRAPH 1994. Google Scholar
  69. Weiss, Y. (2001). Deriving intrinsic images from image sequences. In IEEE international conference on computer vision. Google Scholar
  70. Wu, T.-P., & Tang, C.-K. (2005). A Bayesian approach for shadow extraction from a single image. In IEEE international conference on computer vision. Google Scholar
  71. Yu, Y., & Malik, J. (1998). Recovering photometric properties of architectural scenes from photographs. In Proceedings of ACM SIGGRAPH 1998, July. Google Scholar
  72. Zhu, J., Samuel, K. G. G., Masood, S. Z., & Tappen, M. F. (2010). Learning to recognize shadows in monochromatic natural images. In IEEE conference on computer vision and pattern recognition. Google Scholar
  73. Zickler, T., Mallick, S. P., Kriegman, D. J., & Belhumeur, P. N. (2006). Color subspaces as photometric invariants. In IEEE conference on computer vision and pattern recognition. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Jean-François Lalonde
    • 1
  • Alexei A. Efros
    • 1
  • Srinivasa G. Narasimhan
    • 1
  1. 1.School of Computer ScienceCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations