Advertisement

A Dataset of Flash and Ambient Illumination Pairs from the Crowd

  • Yağız Aksoy
  • Changil Kim
  • Petr Kellnhofer
  • Sylvain Paris
  • Mohamed Elgharib
  • Marc Pollefeys
  • Wojciech Matusik
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11213)

Abstract

Illumination is a critical element of photography and is essential for many computer vision tasks. Flash light is unique in the sense that it is a widely available tool for easily manipulating the scene illumination. We present a dataset of thousands of ambient and flash illumination pairs to enable studying flash photography and other applications that can benefit from having separate illuminations. Different than the typical use of crowdsourcing in generating computer vision datasets, we make use of the crowd to directly take the photographs that make up our dataset. As a result, our dataset covers a wide variety of scenes captured by many casual photographers. We detail the advantages and challenges of our approach to crowdsourcing as well as the computational effort to generate completely separate flash illuminations from the ambient light in an uncontrolled setup. We present a brief examination of illumination decomposition, a challenging and underconstrained problem in flash photography, to demonstrate the use of our dataset in a data-driven approach.

Keywords

Flash photography Dataset collection Crowdsourcing Illumination decomposition 

Notes

Acknowledgements

We would like to thank Alexandre Kaspar for his support on crowdsourcing, James Minor and Valentin Deschaintre for their feedback on the text, and Michaël Gharbi for our discussions. Y. Aksoy was supported by QCRI-CSAIL Computer Science Research Program at MIT, and C. Kim was supported by Swiss National Science Foundation fellowship P2EZP2 168785.

Supplementary material

474192_1_En_39_MOESM1_ESM.pdf (1.5 mb)
Supplementary material 1 (pdf 1585 KB)
474192_1_En_39_MOESM2_ESM.pdf (1.6 mb)
Supplementary material 2 (pdf 1614 KB)

References

  1. 1.
    Adams, A., et al.: The frankencamera: an experimental platform for computational photography. Commun. ACM 55(11), 90–98 (2012)CrossRefGoogle Scholar
  2. 2.
    Agrawal, A., Raskar, R., Nayar, S.K., Li, Y.: Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans. Graph. 24(3), 828–835 (2005)CrossRefGoogle Scholar
  3. 3.
    Bartoli, A.: Groupwise geometric and photometric direct image registration. IEEE Trans. Pattern Anal. Mach. Intell. 30(12), 2098–2108 (2008)CrossRefGoogle Scholar
  4. 4.
    Bell, S., Bala, K., Snavely, N.: Intrinsic images in the wild. ACM Trans. Graph. 33(4) (2014)CrossRefGoogle Scholar
  5. 5.
    Bonneel, N., Kovacs, B., Paris, S., Bala, K.: Intrinsic decompositions for image editing. Comput. Graph. Forum 36(2), 593–609 (2017)CrossRefGoogle Scholar
  6. 6.
    Chen, J., Su, G., He, J., Ben, S.: Face image relighting using locally constrained global optimization. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 44–57. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15561-1_4CrossRefGoogle Scholar
  7. 7.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2009)Google Scholar
  8. 8.
    Eisemann, E., Durand, F.: Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 23(3), 673–678 (2004)CrossRefGoogle Scholar
  9. 9.
    Evangelidis, G.: IAT: a Matlab toolbox for image alignment (2013). http://www.iatool.net
  10. 10.
    Gastal, E.S.L., Oliveira, M.M.: Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30(4), 69:1–69:12 (2011)CrossRefGoogle Scholar
  11. 11.
    He, S., Lau, R.W.H.: Saliency detection with flash and no-flash image pairs. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 110–124. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10578-9_8CrossRefGoogle Scholar
  12. 12.
    Hui, Z., Sankaranarayanan, A.C., Sunkavalli, K., Hadap, S.: White balance under mixed illumination using flash photography. In: International Conference on Computational Photography (ICCP) (2016)Google Scholar
  13. 13.
    Hui, Z., Sunkavalli, K., Hadap, S., Sankaranarayanan, A.C.: Illuminant spectra-based source separation using flash photography. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  14. 14.
    Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  15. 15.
    Kaspar, A., Patterson, G., Kim, C., Aksoy, Y., Matusik, W., Elgharib, M.: Crowd-Guided ensembles: how can we choreograph crowd workers for video segmentation? In: ACM CHI Conference on Human Factors in Computing Systems (2018)Google Scholar
  16. 16.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2014)Google Scholar
  17. 17.
    Kovacs, B., Bell, S., Snavely, N., Bala, K.: Shading annotations in the wild. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  18. 18.
    Krishnan, D., Fergus, R.: Dark flash photography. ACM Trans. Graph. 28(3), 96:1–96:11 (2009)CrossRefGoogle Scholar
  19. 19.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Neural Information Processing Systems Conference (NIPS) (2012)Google Scholar
  20. 20.
    Lettry, L., Vanhoey, K., Van Gool, L.: DARN: a deep adversial residual network for intrinsic image decomposition. In: Winter Conference on Applications of Computer Vision (WACV) (2018)Google Scholar
  21. 21.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_48CrossRefGoogle Scholar
  22. 22.
    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence (IJCAI) (1981)Google Scholar
  23. 23.
    Meka, A., Zollhöfer, M., Richardt, C., Theobalt, C.: Live intrinsic video. ACM Trans. Graph. 35(4), 109:1–109:14 (2016)CrossRefGoogle Scholar
  24. 24.
    Murmann, L., Davis, A., Kautz, J., Durand, F.: Computational bounce flash for indoor portraits. ACM Trans. Graph. 35(6), 190:1–190:9 (2016)CrossRefGoogle Scholar
  25. 25.
    Narasimhan, S.G., Wang, C., Nayar, S.K.: All the images of an outdoor scene. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 148–162. Springer, Heidelberg (2002).  https://doi.org/10.1007/3-540-47977-5_10CrossRefGoogle Scholar
  26. 26.
    Peers, P., Tamura, N., Matusik, W., Debevec, P.: Post-production facial performance relighting using reflectance transfer. ACM Trans. Graph. 26(3) (2007)CrossRefGoogle Scholar
  27. 27.
    Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23(3), 664–672 (2004)CrossRefGoogle Scholar
  28. 28.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  29. 29.
    Sajjadi, M.S., Schölkopf, B., Hirsch, M.: EnhanceNet: single image super-resolution through automated texture synthesis. In: International Conference on Computer Vision (ICCV) (2017)Google Scholar
  30. 30.
    Sun, J., Sun, J., Kang, S.B., Xu, Z.B., Tang, X., Shum, H.Y.: Flash cut: foreground extraction with flash and no-flash image pairs. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2007)Google Scholar
  31. 31.
    Sun, J., Li, Y., Kang, S.B., Shum, H.Y.: Flash matting. ACM Trans. Graph. 25(3), 772–778 (2006)CrossRefGoogle Scholar
  32. 32.
    Vonikakis, V., Chrysostomou, D., Kouskouridas, R., Gasteratos, A.: Improving the robustness in feature detection by local contrast enhancement. In: International Conference on Imaging Systems and Techniques (IST) (2012)Google Scholar
  33. 33.
    Weyrich, T., et al.: Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. 25(3), 1013–1024 (2006)CrossRefGoogle Scholar
  34. 34.
    Zhou, C., Troccoli, A., Pulli, K.: Robust stereo with flash and no-flash image pairs. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  35. 35.
    Zhuo, S., Guo, D., Sim, T.: Robust flash deblurring. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2010)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Yağız Aksoy
    • 1
    • 2
  • Changil Kim
    • 1
  • Petr Kellnhofer
    • 1
  • Sylvain Paris
    • 3
  • Mohamed Elgharib
    • 4
  • Marc Pollefeys
    • 2
    • 5
  • Wojciech Matusik
    • 1
  1. 1.MIT CSAILCambridgeUSA
  2. 2.ETH ZürichZürichSwitzerland
  3. 3.Adobe ResearchCambridgeUSA
  4. 4.QCRIDohaQatar
  5. 5.MicrosoftRedmondUSA

Personalised recommendations