International Journal of Computer Vision

, Volume 88, Issue 1, pp 24–51

What Do the Sun and the Sky Tell Us About the Camera?

  • Jean-François Lalonde
  • Srinivasa G. Narasimhan
  • Alexei A. Efros
Article

Abstract

As the main observed illuminant outdoors, the sky is a rich source of information about the scene. However, it is yet to be fully explored in computer vision because its appearance in an image depends on the sun position, weather conditions, photometric and geometric parameters of the camera, and the location of capture. In this paper, we analyze two sources of information available within the visible portion of the sky region: the sun position, and the sky appearance. By fitting a model of the predicted sun position to an image sequence, we show how to extract camera parameters such as the focal length, and the zenith and azimuth angles. Similarly, we show how we can extract the same parameters by fitting a physically-based sky model to the sky appearance. In short, the sun and the sky serve as geometric calibration targets, which can be used to annotate a large database of image sequences. We test our methods on a high-quality image sequence with known camera parameters, and obtain errors of less that 1% for the focal length, 1° for azimuth angle and 3° for zenith angle. We also use our methods to calibrate 22 real, low-quality webcam sequences scattered throughout the continental US, and show deviations below 4% for focal length, and 3° for the zenith and azimuth angles. Finally, we demonstrate that by combining the information available within the sun position and the sky appearance, we can also estimate the camera geolocation, as well as its geometric parameters. Our method achieves a mean localization error of 110 km on real, low-quality Internet webcams. The estimated viewing and illumination geometry of the scene can be useful for a variety of vision and graphics tasks such as relighting, appearance analysis and scene recovery.

Keywords

Sky Camera calibration Physics-based vision Time-lapse video Camera geolocation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P. N., & Nayar, S. K. (2008). Face swapping: automatically replacing faces in photographs. ACM Transactions on Graphics (SIGGRAPH 2008), 27(3). Google Scholar
  2. Buluswar, S. D., & Draper, B. A. (2002). Color models for outdoor machine vision. Computer Vision and Image Understanding, 85(2), 71–99. MATHCrossRefGoogle Scholar
  3. Chiao, C.-C., & Cronin, T. W. (2000). Color signals in natural scenes: characteristics of reflectance spectra and effects of natural illumination. Journal of Optical Society of America, 17(2), February. Google Scholar
  4. Committee, C. T. (1994). Spatial distribution of daylight—luminance distributions of various reference skies (Technical Report CIE-110-1994). International Commission on Illumination. Google Scholar
  5. Cozman, F., & Krotkov, E. (1995). Robot localization using a computer vision sextant. In IEEE international conference on robotics and automation. Google Scholar
  6. Debevec, P. (1998). Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of ACM SIGGRAPH 1998. Google Scholar
  7. Debevec, P., & Malik, J. (1997). Recovering high dynamic range radiance maps from photographs. In Proceedings of ACM SIGGRAPH 1997, August 1997. Google Scholar
  8. Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., & Martinez, P. (2004). Estimating surface reflectance properties of a complex scene under captured natural illumination (Technical Report ICT-TR-06.2004). USC ICT. Google Scholar
  9. Dror, R. O., Willsky, A. S., & Adelson, E. H. (2004). Statistical characterization of real-world illumination. Journal of Vision, 4, 821–837. CrossRefGoogle Scholar
  10. Forsyth, D. A., & Ponce, J. (2003). Computer vision a modern approach. New York: Prentice Hall. Google Scholar
  11. Gross, R., Baker, S., Matthews, I., & Kanade, T. (2004). Face recognition across pose and illumination. In S. Z. Li & A. K. Jain (Eds.), Handbook of face recognition. Berlin: Springer. Google Scholar
  12. Hill, R. (1994). Theory of geolocation by light levels. In B. J. LeBouef & R. M. Laws (Eds.), Elephant seals: population ecology, behavior, and physiology (pp. 227–236). Berkeley: University of California Press. Chap. 12. Google Scholar
  13. Hoiem, D., Efros, A. A., & Hebert, M. (2005). Geometric context from a single image. In IEEE international conference on computer vision. Google Scholar
  14. Ineichen, P., Molineaux, B., & Perez, R. (1994). Sky luminance data validation: comparison of seven models with four data banks. Solar Energy, 52(4), 337–346. CrossRefGoogle Scholar
  15. Jacobs, N., Roman, N., & Pless, R. (2007a). Consistent temporal variations in many outdoor scenes. In IEEE conference on computer vision and pattern recognition. Google Scholar
  16. Jacobs, N., Satkin, S., Roman, N., Speyer, R., & Pless, R. (2007b). Geolocating static cameras. In IEEE international conference on computer vision. Google Scholar
  17. Jacobs, N., Roman, N., & Pless, R. (2008). Toward fully automatic geo-location and geo-orientation of static outdoor cameras. In Workshop on applications of computer vision. Google Scholar
  18. Khan, E. A., Reinhard, E., Fleming, R., & Büelthoff, H. (2006). Image-based material editing. ACM Transactions on Graphics (SIGGRAPH 2006), August 2006. Google Scholar
  19. Kim, S. J., Frahm, J.-M., & Polleyfeys, M. (2008). Radiometric calibration with illumination change for outdoor scene analysis. In IEEE conference on computer vision and pattern recognition. Google Scholar
  20. Kim, S. J., & Polleyfeys, M. (2008). Robust radiometric calibration and vignetting correction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(4), April 2008. Google Scholar
  21. Koppal, S. J., & Narasimhan, S. G. (2006). Clustering appearance for scene analysis. In IEEE international conference on computer vision. Google Scholar
  22. Kuthirummal, S., Agarwala, A., Glodman, D. B., & Nayar, S. K. (2008). Priors for large photo collections and what they reveal about cameras. In European conference on computer vision. Google Scholar
  23. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., & Criminisi, A. (2007). Photo clip art. ACM Transactions on Graphics (SIGGRAPH 2007), 26(3), August 2007. Google Scholar
  24. Lalonde, J.-F., Narasimhan, S. G., & Efros, A. A. (2008a). Camera parameters estimation from hand-labelled sun positions in image sequences (Technical Report CMU-RI-TR-08-32). Robotics Institute, Carnegie Mellon University, July 2008. Google Scholar
  25. Lalonde, J.-F., Narasimhan, S. G., & Efros, A. A. (2008b). What does the sky tell us about the camera? In European conference on computer vision. Google Scholar
  26. Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2009). Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. ACM Transactions on Graphics (SIGGRAPH Asia 2009). Google Scholar
  27. Lin, S., Gu, J., Yamazaki, S., & Shum, H.-Y. (2004). Radiometric calibration from a single image. In IEEE conference on computer vision and pattern recognition. Google Scholar
  28. Manduchi, R. (2006). Learning outdoor color classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11), 1713–1723. CrossRefGoogle Scholar
  29. Narasimhan, S. G., Wang, C., & Nayar, S. K. (2002). All images of an outdoor scene. In European conference on computer vision (pp. 148–162). May 2002. Google Scholar
  30. Perez, R., Seals, R., & Michalsky, J. (1993). All-weather model for sky luminance distribution—preliminary configuration and validation. Solar Energy, 50(3), 235–245. CrossRefGoogle Scholar
  31. Preetham, A. J., Shirley, P., & Smits, B. (1999). A practical analytic model for daylight. In Proceedings of ACM SIGGRAPH 1999, August 1999. Google Scholar
  32. Reda, I., & Andreas, A. (2005). Solar position algorithm for solar radiation applications (Technical Report NREL/TP-560-34302). National Renewable Energy Laboratory, November 2005. Google Scholar
  33. Sato, Y., & Ikeuchi, K. (1995). Reflectance analysis under solar illumination. In Proceedings of the IEEE workshop on physics-based modeling and computer vision (pp. 180–187). Google Scholar
  34. Slater, D., & Healey, G. (1998). What is the spectral dimensionality of illumination functions in outdoor scenes? In IEEE conference on computer vision and pattern recognition. Google Scholar
  35. Stumpfel, J., Jones, A., Wenger, A., Tchou, C., Hawkins, T., & Debevec, P. (2004). Direct HDR capture of the sun and sky. In Proceedings of AFRIGRAPH. Google Scholar
  36. Sunkavalli, K., Matusik, W., Pfister, H., & Rusinkiewicz, S. (2007). Factored time-lapse video. ACM Transactions on Graphics (SIGGRAPH 2007), 26(3), August 2007. Google Scholar
  37. Sunkavalli, K., Romeiro, F., Matusik, W., Zickler, T., & Pfister, H. (2008). What do color changes reveal about an outdoor scene? In IEEE conference on computer vision and pattern recognition. Google Scholar
  38. Toyama, K., Krumm, J., Brumitt, B., & Meyers, B. (1999). Wallflower: principles and practice of background maintenance. In IEEE International conference on computer vision. Google Scholar
  39. Trebi-Ollennu, A., Huntsberger, T., Cheng, Y., Baumgartner, E. T., Kennedy, B., & Schenker, P. (2001). Design and analysis of a sun sensor for planetary rover absolute heading detection. IEEE Transactions on Robotics and Automation, 17(6), 939–947. CrossRefGoogle Scholar
  40. Tsin, Y., Collins, R. T., Ramesh, V., & Kanade, T. (2001). Bayesian color constancy for outdoor object recognition. In IEEE conference on computer vision and pattern recognition. Google Scholar
  41. Weiss, Y. (2001). Deriving intrinsic images from image sequences. In IEEE international conference on computer vision. Google Scholar
  42. Yu, Y., & Malik, J. (1998). Recovering photometric properties of architectural scenes from photographs. In Proceedings of ACM SIGGRAPH 1998, July 1998. Google Scholar
  43. Zheng, Y., Lin, S., & Kang, S. B. (2006). Single-image vignetting correction. In IEEE conference on computer vision and pattern recognition. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Jean-François Lalonde
    • 1
  • Srinivasa G. Narasimhan
    • 1
  • Alexei A. Efros
    • 1
  1. 1.School of Computer ScienceCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations