A 4D Light-Field Dataset and CNN Architectures for Material Recognition

  • Ting-Chun Wang
  • Jun-Yan Zhu
  • Ebi Hiroaki
  • Manmohan Chandraker
  • Alexei A. Efros
  • Ravi Ramamoorthi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9907)

Abstract

We introduce a new light-field dataset of materials, and take advantage of the recent success of deep learning to perform material recognition on the 4D light-field. Our dataset contains 12 material categories, each with 100 images taken with a Lytro Illum, from which we extract about 30,000 patches in total. To the best of our knowledge, this is the first mid-size dataset for light-field images. Our main goal is to investigate whether the additional information in a light-field (such as multiple sub-aperture views and view-dependent reflectance effects) can aid material recognition. Since recognition networks have not been trained on 4D images before, we propose and compare several novel CNN architectures to train on light-field images. In our experiments, the best performing CNN architecture achieves a 7 % boost compared with 2D image classification (\(70\,\%\rightarrow 77\,\%\)). These results constitute important baselines that can spur further research in the use of CNNs for light-field applications. Upon publication, our dataset also enables other novel applications of light-fields, including object detection, image segmentation and view interpolation.

Keywords

Light-field Material recognition Convolutional neural network 

References

  1. 1.
    Adams, A., Levoy, M., Vaish, V., Wilburn, B., Joshi, N.: Stanford light field archive. http://lightfield.stanford.edu/
  2. 2.
    Bell, S., Upchurch, P., Snavely, N., Bala, K.: Opensurfaces: a richly annotated catalog of surface appearance. ACM Trans. Graph. (TOG) 32(4), 111 (2013)CrossRefGoogle Scholar
  3. 3.
    Bell, S., Upchurch, P., Snavely, N., Bala, K.: Material recognition in the wild with the materials in context database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  4. 4.
    Caputo, B., Hayman, E., Mallikarjuna, P.: Class-specific material categorisation. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2005)Google Scholar
  5. 5.
    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  6. 6.
    Cimpoi, M., Maji, S., Vedaldi, A.: Deep filter banks for texture recognition and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  7. 7.
    Cula, O.G., Dana, K.J.: 3D texture recognition using bidirectional feature histograms. Int. J. Comput. Vis. 59(1), 33–60 (2004)CrossRefGoogle Scholar
  8. 8.
    Dana, K.J., Van Ginneken, B., Nayar, S.K., Koenderink, J.J.: Reflectance and texture of real-world surfaces. ACM Trans. Graph. (TOG) 18(1), 1–34 (1999)CrossRefGoogle Scholar
  9. 9.
    Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 35(8), 1915–1929 (2013)CrossRefGoogle Scholar
  10. 10.
    Hayman, E., Caputo, B., Fritz, M., Eklundh, J.-O.: On the significance of real-world conditions for material classification. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 253–266. Springer, Heidelberg (2004). doi:10.1007/978-3-540-24673-2_21 CrossRefGoogle Scholar
  11. 11.
    He, K., Sun, J., Tang, X.: Guided image filtering. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 1–14. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15549-9_1 CrossRefGoogle Scholar
  12. 12.
    Hu, D., Bo, L., Ren, X.: Toward robust material recognition for everyday objects. In: BMVC (2011)Google Scholar
  13. 13.
    Jarabo, A., Masia, B., Bousseau, A., Pellacini, F., Gutierrez, D.: How do people edit light fields? ACM Trans. Graph. (TOG) 33(4), 146:1–146:10 (2014)CrossRefGoogle Scholar
  14. 14.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia (2014)Google Scholar
  15. 15.
    Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.H.: Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph. (TOG) 32(4), 73 (2013)MATHGoogle Scholar
  16. 16.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)Google Scholar
  17. 17.
    Li, N., Ye, J., Ji, Y., Ling, H., Yu, J.: Saliency detection on light field. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  18. 18.
    Li, W., Fritz, M.: Recognizing materials from virtual examples. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 345–358. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33765-9_25 Google Scholar
  19. 19.
    Liu, C., Sharan, L., Adelson, E.H., Rosenholtz, R.: Exploring features in a bayesian framework for material recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2010)Google Scholar
  20. 20.
    Liu, C., Gu, J.: Discriminative illumination: per-pixel classification of raw materials based on optimal projections of spectral BRDF. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 36(1), 86–98 (2014)CrossRefGoogle Scholar
  21. 21.
    Lombardi, S., Nishino, K.: Single image multimaterial estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  22. 22.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  23. 23.
    Marwah, K., Wetzstein, G., Bando, Y., Raskar, R.: Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Trans. Graph. (TOG) 32(4), 1–11 (2013)CrossRefMATHGoogle Scholar
  24. 24.
    Nicodemus, F.E., Richmond, J.C., Hsia, J.J., Ginsberg, I.W., Limperis, T.: Geometrical considerations and nomenclature for reflectance, vol. 160. US Department of Commerce, National Bureau of Standards Washington, DC, USA (1977)Google Scholar
  25. 25.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  26. 26.
    Qi, X., Xiao, R., Li, C.G., Qiao, Y., Guo, J., Tang, X.: Pairwise rotation invariant co-occurrence local binary pattern. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 36(11), 2199–2213 (2014)CrossRefGoogle Scholar
  27. 27.
    Raghavendra, R., Raja, K.B., Busch, C.: Exploring the usefulness of light field cameras for biometrics: an empirical study on face and iris recognition. IEEE Trans. Inf. Forensics Secur. 11(5), 922–936 (2016)CrossRefGoogle Scholar
  28. 28.
    Schwartz, G., Nishino, K.: Visual material traits: Recognizing per-pixel material context. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops (2013)Google Scholar
  29. 29.
    Sharan, L., Rosenholtz, R., Adelson, E.: Material perception: what can you see in a brief glance? J. Vis. 9(8), 784–784 (2009)CrossRefGoogle Scholar
  30. 30.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  31. 31.
    Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view convolutional neural networks for 3d shape recognition. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2015)Google Scholar
  32. 32.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  33. 33.
    Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2013)Google Scholar
  34. 34.
    Wang, T.C., Efros, A., Ramamoorthi, R.: Occlusion-aware depth estimation using light-field cameras. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2015)Google Scholar
  35. 35.
    Wanner, S., Meister, S., Goldlücke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: Annual Workshop on Vision, Modeling and Visualization, pp. 225–226 (2013)Google Scholar
  36. 36.
    Weinmann, M., Gall, J., Klein, R.: Material classification based on training data synthesized using a BTF database. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 156–171. Springer, Heidelberg (2014). doi:10.1007/978-3-319-10578-9_11 Google Scholar
  37. 37.
    Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., Kweon, I.: Learning a deep convolutional network for light-field image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops (2015)Google Scholar
  38. 38.
    Zhang, H., Dana, K., Nishino, K.: Reflectance hashing for material recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Ting-Chun Wang
    • 1
  • Jun-Yan Zhu
    • 1
  • Ebi Hiroaki
    • 2
  • Manmohan Chandraker
    • 2
  • Alexei A. Efros
    • 1
  • Ravi Ramamoorthi
    • 2
  1. 1.University of CaliforniaBerkeleyUSA
  2. 2.University of CaliforniaSan DiegoUSA

Personalised recommendations