Advertisement

One-Day Outdoor Photometric Stereo Using Skylight Estimation

  • Jiyoung JungEmail author
  • Joon-Young Lee
  • In So Kweon
Article

Abstract

We present an outdoor photometric stereo method using images captured in a single day. We simulate a sky hemisphere for each image according to its GPS and timestamp and parameterize the obtained sky hemisphere into quadratic skylight and Gaussian sunlight distributions. Our previous work recovered an outdoor scene on a clear day, whereas the current paper shows that cloudy days can provide better illumination conditions for surface orientation recovery, and hence we propose a modified sky model to represent a well-conditioned skylight distribution for outdoor photometric stereo. The proposed method models natural illumination according to a sky model, providing sufficient constraints for shape reconstruction from 1-day images. We tested the proposed method to recover various sized objects and scenes from real-world outdoor daylight images and verified the method using synthetic and real data experiments.

Keywords

Photometric stereo Light modeling Outdoor illumination Scene reconstruction Skylight estimation 

Notes

Acknowledgements

This work was supported by a Grant from Kyung Hee University (KHU-20170718) and National Research Foundation of Korea (NRF-2017R1C1B5075945).

References

  1. Abrams, A., Hawley, C., & Pless, R. (2012). Heliometric stereo: Shape from sun position. In Proceedings of European conference on computer vision (ECCV).Google Scholar
  2. Abrams, A., Hawley, C., Miskell, K., Stoica, A., Jacobs, N., & Pless, R. (2013a). Shadow estimation method for “the episolar constraint: Monocular shape from shadow correspondence”. arXiv:1304.4112.
  3. Abrams, A., Miskell, K., & Pless, R. (2013b). The episolar constraint: Monocular shape from shadow correspondence. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  4. Ackermann, J., Langguth, F., Fuhrmann, S., & Goesele, M. (2012). Photometric stereo for outdoor webcams. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  5. Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., & Seitz, S. M. (2011). Building Rome in a day. Communications of the ACM, 54(10), 105–112.Google Scholar
  6. Furukawa, Y., & Ponce, J. (2010). Accurate, dense, and robust multi-view stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 32(8), 1362–1376.Google Scholar
  7. Google Earth. (2015). https://earth.google.com.
  8. Grossmann, P. (1987). Depth from focus. Pattern Recognition Letters, 5(1), 63–69.Google Scholar
  9. Han, Y., Lee, J. Y., & Kweon, I. S. (2013). High quality shape from a single RGB-D image under uncalibrated natural illumination. In Proceedings of international conference on computer vision (ICCV).Google Scholar
  10. Hold-Geoffroy, Y., Zhang, J., Gotardo, P. F. U., & LaLonde, J. F. (2015a). What is a good day for outdoor photometric stereo? In International conference on computational photography.Google Scholar
  11. Hold-Geoffroy, Y., Zhang, J., Gotardo, P. F. U., & LaLonde, J. F. (2015b). x-hour outdoor photometric stereo. In International conference on 3D vision.Google Scholar
  12. Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., & LaLonde, J. F. (2017). Deep outdoor illumination estimation. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  13. Horn, B. K. P. (1989). Obtaining shape from shading information. In Shape from shading (pp. 123–171). Cambridge, MA, USA: MIT Press.Google Scholar
  14. Hošek, L., & Wilkie, A. (2013). Adding a solar radiance function to the Hošek–Wilkie skylight model. IEEE Computer Graphics and Applications, 33(3), 44–52.Google Scholar
  15. Huang, R., & Smith, W. A. P. (2011). Shape-from-shading under complex natural illumination. In Proceedings of international conference on image processing (ICIP).Google Scholar
  16. Hung, C. H., Wu, T. P., Matsushita, Y., Xu, L., Jia, J., & Tang, C. K. (2015). Photometric stereo in the wild. In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV). Waikoloa, HI: IEEE.Google Scholar
  17. Inose, K., Shimizu, S., Kawakami, R., Mukaigawa, Y., & Ikeuchi, K. (2013). Refining outdoor photometric stereo based on sky model. IPSJ Transaction on Computer Vision and Applications, 5(1), 104–108.Google Scholar
  18. Jacobs, N., Roman, N., & Pless, R. (2007). Consistent temporal variations in many outdoor scenes. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  19. Johnson, M. K., & Adelson, E. H. (2011). Shape estimation in natural illumination. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  20. Jung, J. (2016). Outdoor photometric stereo for 3D scene recovery. Ph.D. Thesis, KAIST, Daejeon, Republic of Korea.Google Scholar
  21. Jung, J., Lee, J. Y., Jeong, Y., & Kweon, I. S. (2015a). Time-of-flight sensor calibration for a color and depth camera pair. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 37(7), 1501–1513.Google Scholar
  22. Jung, J., Lee, J. Y., & Kweon, I. S. (2015b). One-day outdoor photometric stereo via skylight estimation. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  23. Kawakami, R., Zhao, H., Tan, R. T., & Ikeuchi, K. (2013). Camera spectral sensitivity and white balance estimation from sky images. International Journal on Computer Vision (IJCV), 105(3), 187–204.zbMATHGoogle Scholar
  24. Koenderink, J. J., van Doorn, A. J., & Kappers, A. M. L. (1992). Surface perception in pictures. Perception and Psychophysics, 52(5), 487–496.Google Scholar
  25. Kolmogorov, V., & Zabih, R. (2004). What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 26(2), 147–159.zbMATHGoogle Scholar
  26. Lalonde, J. F., Narasimhan, S. G., & Efros, A. A. (2010). What do the sun and the sky tell us about the camera? International Journal on Computer Vision (IJCV), 88(1), 24–51.Google Scholar
  27. Lalonde, J. F., Efros, A. A., & Narasimhan, S. G. (2012). Estimating natural illumination from a single outdoor image. International Journal on Computer Vision (IJCV), 98(2), 123–145.MathSciNetGoogle Scholar
  28. Lalonde, J. F., Asselin, L. P., Becirovski, J., Hold-Geoffroy, Y., Garon, M., Gardner, M. A., & Zhang, J. (2016). The Laval HDR sky database. http://www.hdrdb.com. Accessed 1 October 2015.
  29. Lee, J. Y., Matsushita, Y., Shi, B., & Kweon, I. S. (2013). Radiometric calibration by rank minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 35(1), 114–156.Google Scholar
  30. Lombardi, S., & Nishino, K. (2012). Reflectance and illumination from a single image. In Proceedings of European conference on computer vision (ECCV).Google Scholar
  31. Lu, F., Matsushita, Y., Sato, I., Okabe, T., & Sato, Y. (2013). Uncalibrated photometric stereo for unknown isotropic reflectances. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  32. Okutomi, M., & Kanade, T. (1993). A multiple-baseline stereo system. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 15(4), 353–363.Google Scholar
  33. Oxholm, G., & Nishino, K. (2012). Shape and reflectance from natural illumination. In Proceedings of European conference on computer vision (ECCV).Google Scholar
  34. Preetham, A. J., Shirley, P., & Smits, B. (1999). A practical analytic model for daylight. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques (SIGGRAPH ’99) (pp. 91–100). ACM Press/Addison-Wesley Publishing Co.Google Scholar
  35. Ramamoorthi, R., & Hanrahan, P. (2001). On the relationship between radiance and irradiance: Determining the illumination from images of a convex lambertian object. Journal of the Optical Society of America A (JOSA A), 18(10), 2448–2459.MathSciNetGoogle Scholar
  36. Saff, E., & Kuijlaars, A. B. J. (1997). Distributing many points on a sphere. The Mathematical Intelligencer, 19(1), 5–11.MathSciNetzbMATHGoogle Scholar
  37. Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal on Computer Vision (IJCV), 47(13), 7–42.zbMATHGoogle Scholar
  38. Shan, Q., Adams, R., Curless, B., Furukawa, Y., & Seitz, S. M. (2013). The visual turing test for scene reconstruction. In Proceedings of international conference on 3D vision.Google Scholar
  39. Shen, F., Sunkavalli, K., Bonneel, N., Rusinkiewicz, S., Pfister, H., & Tong, X. (2014). Time-lapse photometric stereo and applications. Computer Graphics Forum, 33(7), 359–367.Google Scholar
  40. Shi, B., Matsushita, Y., Wei, Y., Xu, C., & Tan, P. (2010). Self-calibrating photometric stereo. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  41. Simchony, T., Chellappa, R., & Shao, M. (1990). Direct analytical methods for solving poisson equations in computer vision problems. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 12(5), 435–446.zbMATHGoogle Scholar
  42. Stumpfel, J., Tchou, C., Jones, A., Hawkins, T., Wenger, A., & Debevec, P. (2004). Direct HDR capture of the sun and sky. In Proceedings of the 3rd International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa (AFRIGRAPH) (pp. 145–149). ACM.Google Scholar
  43. Subbarao, M., & Surya, G. (1994). Depth from defocus: A spatial domain approach. International Journal on Computer Vision (IJCV), 13(3), 271–294.Google Scholar
  44. Sunkavalli, K., Matusik, W., Pfister, H., & Rusinkiewicz, S. (2007). Factored time-lapse video. ACM Transaction on Graphics (TOG), 26(3), 101–111.Google Scholar
  45. Sunkavalli, K., Romeiro, F., Matusik, W., Zickler, T., & Pfister, H. (2008). What do color changes reveal about an outdoor scene? In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  46. Turk, G., & Levoy, M. (1994). Zippered polygon meshes from range images. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques (SIGGRAPH ’94) (pp. 311-318). ACM.Google Scholar
  47. Woodham, R. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144.Google Scholar
  48. Yoon, K. J., & Kweon, I. S. (2006). Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 28(4), 650–656.Google Scholar
  49. Yu, L. F., Yeung, S. K., Tai, Y. W., & Lin, S. (2013). Shading-based shape refinement of RGB-D images. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  50. Zhang, Q., Ye, M., Yang, R., Matsushita, Y., Wilburn, B., & Yu, H. (2012). Edge-preserving photometric stereo via depth fusion. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Software ConvergenceKyung Hee UniversityYonginRepublic of Korea
  2. 2.Adobe ResearchSan JoseUSA
  3. 3.School of Electrical EngineeringKAISTDaejeonRepublic of Korea

Personalised recommendations