Abstract
As the main observed illuminant outdoors, the sky is a rich source of information about the scene. However, it is yet to be fully explored in computer vision because its appearance in an image depends on the sun position, weather conditions, photometric and geometric parameters of the camera, and the location of capture. In this paper, we analyze two sources of information available within the visible portion of the sky region: the sun position, and the sky appearance. By fitting a model of the predicted sun position to an image sequence, we show how to extract camera parameters such as the focal length, and the zenith and azimuth angles. Similarly, we show how we can extract the same parameters by fitting a physically-based sky model to the sky appearance. In short, the sun and the sky serve as geometric calibration targets, which can be used to annotate a large database of image sequences. We test our methods on a high-quality image sequence with known camera parameters, and obtain errors of less that 1% for the focal length, 1° for azimuth angle and 3° for zenith angle. We also use our methods to calibrate 22 real, low-quality webcam sequences scattered throughout the continental US, and show deviations below 4% for focal length, and 3° for the zenith and azimuth angles. Finally, we demonstrate that by combining the information available within the sun position and the sky appearance, we can also estimate the camera geolocation, as well as its geometric parameters. Our method achieves a mean localization error of 110 km on real, low-quality Internet webcams. The estimated viewing and illumination geometry of the scene can be useful for a variety of vision and graphics tasks such as relighting, appearance analysis and scene recovery.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P. N., & Nayar, S. K. (2008). Face swapping: automatically replacing faces in photographs. ACM Transactions on Graphics (SIGGRAPH 2008), 27(3).
Buluswar, S. D., & Draper, B. A. (2002). Color models for outdoor machine vision. Computer Vision and Image Understanding, 85(2), 71–99.
Chiao, C.-C., & Cronin, T. W. (2000). Color signals in natural scenes: characteristics of reflectance spectra and effects of natural illumination. Journal of Optical Society of America, 17(2), February.
Committee, C. T. (1994). Spatial distribution of daylight—luminance distributions of various reference skies (Technical Report CIE-110-1994). International Commission on Illumination.
Cozman, F., & Krotkov, E. (1995). Robot localization using a computer vision sextant. In IEEE international conference on robotics and automation.
Debevec, P. (1998). Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of ACM SIGGRAPH 1998.
Debevec, P., & Malik, J. (1997). Recovering high dynamic range radiance maps from photographs. In Proceedings of ACM SIGGRAPH 1997, August 1997.
Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., & Martinez, P. (2004). Estimating surface reflectance properties of a complex scene under captured natural illumination (Technical Report ICT-TR-06.2004). USC ICT.
Dror, R. O., Willsky, A. S., & Adelson, E. H. (2004). Statistical characterization of real-world illumination. Journal of Vision, 4, 821–837.
Forsyth, D. A., & Ponce, J. (2003). Computer vision a modern approach. New York: Prentice Hall.
Gross, R., Baker, S., Matthews, I., & Kanade, T. (2004). Face recognition across pose and illumination. In S. Z. Li & A. K. Jain (Eds.), Handbook of face recognition. Berlin: Springer.
Hill, R. (1994). Theory of geolocation by light levels. In B. J. LeBouef & R. M. Laws (Eds.), Elephant seals: population ecology, behavior, and physiology (pp. 227–236). Berkeley: University of California Press. Chap. 12.
Hoiem, D., Efros, A. A., & Hebert, M. (2005). Geometric context from a single image. In IEEE international conference on computer vision.
Ineichen, P., Molineaux, B., & Perez, R. (1994). Sky luminance data validation: comparison of seven models with four data banks. Solar Energy, 52(4), 337–346.
Jacobs, N., Roman, N., & Pless, R. (2007a). Consistent temporal variations in many outdoor scenes. In IEEE conference on computer vision and pattern recognition.
Jacobs, N., Satkin, S., Roman, N., Speyer, R., & Pless, R. (2007b). Geolocating static cameras. In IEEE international conference on computer vision.
Jacobs, N., Roman, N., & Pless, R. (2008). Toward fully automatic geo-location and geo-orientation of static outdoor cameras. In Workshop on applications of computer vision.
Khan, E. A., Reinhard, E., Fleming, R., & Büelthoff, H. (2006). Image-based material editing. ACM Transactions on Graphics (SIGGRAPH 2006), August 2006.
Kim, S. J., Frahm, J.-M., & Polleyfeys, M. (2008). Radiometric calibration with illumination change for outdoor scene analysis. In IEEE conference on computer vision and pattern recognition.
Kim, S. J., & Polleyfeys, M. (2008). Robust radiometric calibration and vignetting correction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(4), April 2008.
Koppal, S. J., & Narasimhan, S. G. (2006). Clustering appearance for scene analysis. In IEEE international conference on computer vision.
Kuthirummal, S., Agarwala, A., Glodman, D. B., & Nayar, S. K. (2008). Priors for large photo collections and what they reveal about cameras. In European conference on computer vision.
Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., & Criminisi, A. (2007). Photo clip art. ACM Transactions on Graphics (SIGGRAPH 2007), 26(3), August 2007.
Lalonde, J.-F., Narasimhan, S. G., & Efros, A. A. (2008a). Camera parameters estimation from hand-labelled sun positions in image sequences (Technical Report CMU-RI-TR-08-32). Robotics Institute, Carnegie Mellon University, July 2008.
Lalonde, J.-F., Narasimhan, S. G., & Efros, A. A. (2008b). What does the sky tell us about the camera? In European conference on computer vision.
Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2009). Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. ACM Transactions on Graphics (SIGGRAPH Asia 2009).
Lin, S., Gu, J., Yamazaki, S., & Shum, H.-Y. (2004). Radiometric calibration from a single image. In IEEE conference on computer vision and pattern recognition.
Manduchi, R. (2006). Learning outdoor color classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11), 1713–1723.
Narasimhan, S. G., Wang, C., & Nayar, S. K. (2002). All images of an outdoor scene. In European conference on computer vision (pp. 148–162). May 2002.
Perez, R., Seals, R., & Michalsky, J. (1993). All-weather model for sky luminance distribution—preliminary configuration and validation. Solar Energy, 50(3), 235–245.
Preetham, A. J., Shirley, P., & Smits, B. (1999). A practical analytic model for daylight. In Proceedings of ACM SIGGRAPH 1999, August 1999.
Reda, I., & Andreas, A. (2005). Solar position algorithm for solar radiation applications (Technical Report NREL/TP-560-34302). National Renewable Energy Laboratory, November 2005.
Sato, Y., & Ikeuchi, K. (1995). Reflectance analysis under solar illumination. In Proceedings of the IEEE workshop on physics-based modeling and computer vision (pp. 180–187).
Slater, D., & Healey, G. (1998). What is the spectral dimensionality of illumination functions in outdoor scenes? In IEEE conference on computer vision and pattern recognition.
Stumpfel, J., Jones, A., Wenger, A., Tchou, C., Hawkins, T., & Debevec, P. (2004). Direct HDR capture of the sun and sky. In Proceedings of AFRIGRAPH.
Sunkavalli, K., Matusik, W., Pfister, H., & Rusinkiewicz, S. (2007). Factored time-lapse video. ACM Transactions on Graphics (SIGGRAPH 2007), 26(3), August 2007.
Sunkavalli, K., Romeiro, F., Matusik, W., Zickler, T., & Pfister, H. (2008). What do color changes reveal about an outdoor scene? In IEEE conference on computer vision and pattern recognition.
Toyama, K., Krumm, J., Brumitt, B., & Meyers, B. (1999). Wallflower: principles and practice of background maintenance. In IEEE International conference on computer vision.
Trebi-Ollennu, A., Huntsberger, T., Cheng, Y., Baumgartner, E. T., Kennedy, B., & Schenker, P. (2001). Design and analysis of a sun sensor for planetary rover absolute heading detection. IEEE Transactions on Robotics and Automation, 17(6), 939–947.
Tsin, Y., Collins, R. T., Ramesh, V., & Kanade, T. (2001). Bayesian color constancy for outdoor object recognition. In IEEE conference on computer vision and pattern recognition.
Weiss, Y. (2001). Deriving intrinsic images from image sequences. In IEEE international conference on computer vision.
Yu, Y., & Malik, J. (1998). Recovering photometric properties of architectural scenes from photographs. In Proceedings of ACM SIGGRAPH 1998, July 1998.
Zheng, Y., Lin, S., & Kang, S. B. (2006). Single-image vignetting correction. In IEEE conference on computer vision and pattern recognition.
Author information
Authors and Affiliations
Corresponding author
Additional information
Rights and permissions
About this article
Cite this article
Lalonde, JF., Narasimhan, S.G. & Efros, A.A. What Do the Sun and the Sky Tell Us About the Camera?. Int J Comput Vis 88, 24–51 (2010). https://doi.org/10.1007/s11263-009-0291-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-009-0291-4