International Journal of Computer Vision

, Volume 103, Issue 3, pp 348–371 | Cite as

Recognizing Materials Using Perceptually Inspired Features

  • Lavanya Sharan
  • Ce Liu
  • Ruth Rosenholtz
  • Edward H. Adelson
Article

Abstract

Our world consists not only of objects and scenes but also of materials of various kinds. Being able to recognize the materials that surround us (e.g., plastic, glass, concrete) is important for humans as well as for computer vision systems. Unfortunately, materials have received little attention in the visual recognition literature, and very few computer vision systems have been designed specifically to recognize materials. In this paper, we present a system for recognizing material categories from single images. We propose a set of low and mid-level image features that are based on studies of human material recognition, and we combine these features using an SVM classifier. Our system outperforms a state-of-the-art system (Varma and Zisserman, TPAMI 31(11):2032–2047, 2009) on a challenging database of real-world material categories (Sharan et al., J Vis 9(8):784–784a, 2009). When the performance of our system is compared directly to that of human observers, humans outperform our system quite easily. However, when we account for the local nature of our image features and the surface properties they measure (e.g., color, texture, local shape), our system rivals human performance. We suggest that future progress in material recognition will come from: (1) a deeper understanding of the role of non-local surface properties (e.g., extended highlights, object identity); and (2) efforts to model such non-local surface properties in images.

Keywords

Material recognition Material classification Texture classification Mechanical Turk Perception 

References

  1. Adelson, E. H. (2001). On seeing stuff: The perception of materials by humans and machines. In SPIE, human vision and electronic imaging VI (Vol. 4299, pp. 1–12).Google Scholar
  2. Bae, S., Paris, S., & Durand, F. (2006). Two-scale tone management for photographic look. In ACM SIGGRAPH, New York.Google Scholar
  3. Belongie, S., Malik, J., & Puzicha, J. (2002). Shape matching and object recognition using shape contexts. TPAMI, 24(4), 509–522.CrossRefGoogle Scholar
  4. Berzhanskaya, J., Swaminathan, G., Beck, J., & Mingolla, E. (2005). Remote effects of highlights on gloss perception. Perception, 34(5), 565–575.CrossRefGoogle Scholar
  5. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.MATHGoogle Scholar
  6. Bloj, M., Kersten, D., & Hurlbert, A. C. (1999). Perception of three-dimensional shape influences color perception through mutual illumination. Nature, 402, 877–879.Google Scholar
  7. Boivin, S., & Gagalowicz, A. (2001). Image-based rendering of diffuse, specular and glossy surfaces from a single image. In ACM SIGGRAPH, Los Angeles (pp. 107–116).Google Scholar
  8. Boyaci, H., Maloney, L. T., & Hersh, S. (2003). The effect of perceived surface orientation on perceived surface albedo in binocularly viewed scenes. Journal of Vision, 3, 541–553.CrossRefGoogle Scholar
  9. Brainard, D. H., Kraft, J. M., & Longere, P. (2003). Color perception: From light to object. In Color constancy: Developing empirical tests of computational models (pp. 307–334). Oxford: Oxford University Press.Google Scholar
  10. Burges, C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2), 121–167.CrossRefGoogle Scholar
  11. Canny, J. (1986). A computational approach to edge detection. TPAMI, 80(6), 679–698.CrossRefGoogle Scholar
  12. Caputo, B., Hayman, E., & Mallikarjuna, P. (2005). Class-specific material categorization. In Proceedings of the ICCV, Beijing (Vol. 2, pp. 1597–1604).Google Scholar
  13. Caputo, B., Hayman, E., Fritz, M., & Jan-Olof E. (2007) Classifying materials in the real world. Martign: IDIAP.Google Scholar
  14. Cula, O. G., & Dana, K. J. (2004a). 3D texture recognition using bidirectional feature histograms. IJCV, 59(1), 33–60.CrossRefGoogle Scholar
  15. Cula, O. G., Dana, K. J., Murphy, F. P., & Rao, B. K. (2004). Bidirectional imaging and modeling of skin texture. IEEE Transactions on Biomedical Engineering, 51(12), 2148–2159.CrossRefGoogle Scholar
  16. Cula, O. J., & Dana, K. J. (2004b). 3d texture recognition using bidirectional feature histograms. International Journal of Computer Vision, 59(1), 33–60.CrossRefGoogle Scholar
  17. Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In CVPR, Montbonnot (Vol. 2, pp. 886–893).Google Scholar
  18. Dana, K. J., & Nayar, S. (1998). Histogram model for 3d textures. In CVPR (pp. 618–624).Google Scholar
  19. Dana, K. J., Van-Ginneken, B., Nayar, S. K., & Koenderink, J. J. (1999). Reflectance and texture of real world surfaces. ACM Transactions on Graphics, 18(1), 1–34.Google Scholar
  20. Debevec, P., Hawkins, T., Tchou, C., Duiker, H. P., Sarokin, W., & Sagar, M. (2000). Acquiring the reflectance field of a human face. In ACM SIGGRAPH, Louisiana (pp. 145–156).Google Scholar
  21. Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., & Martinez, P. (2004). Estimating surface reflectance properties of a complex scene under captured natural illumination. ICT-TR-06, University of Southern California.Google Scholar
  22. Dror, R., Adelson, E. H., & Alan S. Willsky (2001). Recognition of surface reflectance properties from a single image under unknown real-world illumination. In IEEE Workshop on identifying objects across variation in lighting.Google Scholar
  23. Durand, F., & Dorsey, J. (2002). Fast bilateral filtering for the display of high-dynamic-range images. In ACM SIGGRAPH, San Antonio.Google Scholar
  24. Efros, A. A., & Freeman, W. T. (2001). Image quilting for texture synthesis and transfer. In ACM SIGGRAPH, Los Angeles.Google Scholar
  25. Fei-Fei, L., & Perona, P. (2005). A bayesian hierarchical model for learning natural scene categories. In CVPR, San Diego (Vol. 2, pp. 524–531).Google Scholar
  26. Fleming, R. W., & Bülthoff, H. (2005). Low-level image cues in the perception of translucent materials. ACM Transactions on Applied Perception, 2(3), 346–382.CrossRefGoogle Scholar
  27. Fleming, R. W., Dror, R., & Adelson, E. H. (2003). Real world illumination and the perception of surface reflectance properties. Journal of Vision, 3(5), 347–368.CrossRefGoogle Scholar
  28. Fleming, R. W., Torralba, A., & Adelson, E. H. (2004). Specular reflections and the perception of shape. Journal of Vision, 4(9), 798–820.Google Scholar
  29. Forsyth, D., & Fleck, M. M. (1999). Automatic detection of human nudes. IJCV, 32(1), 63–77.CrossRefGoogle Scholar
  30. Fritz, M., Black, M., Bradski, G., & Darrell, T. (2009). An additive latent feature model for transparent object recognition. In NIPS.Google Scholar
  31. Gilchrist, A., Kossyfidis, C., Bonato, F., Agostini, T., Cataliotti, J., Li, X., et al. (1999). An anchoring theory of lightness perception. Psychological Review, 106, 795–834.CrossRefGoogle Scholar
  32. He, X. D., Torrance, K. E., Sillion, F. S., & Greenberg, D. P. (1991). A comprehensive physical model for light reflection. In 18th annual conference on computer graphics and interactive techniques (Vol. 25, pp. 175–186). New York: ACM.Google Scholar
  33. Ho, Y. X., Landy, M. S., & Maloney, L. T. (2008). Conjoint measurement of gloss and surface texture. Psychological Science, 19(2), 196–204.Google Scholar
  34. Hu, D., Bo, L., & Ren, X. (2011). Robust material recognition for everyday objects. In BMVC, Dundee.Google Scholar
  35. Jensen, H. W., Marschner, S., Levoy, M. & Hanrahan, P. (2001). A practical model for subsurface light transport. In ACM SIGGRAPH, Los Angeles (pp. 511–518).Google Scholar
  36. Khan, E. A., Reinhard, E., Fleming, R. W., & H. Bülthoff, H. (2006). Image-based material editing. In ACM SIGGRAPH, Boston (pp. 654–663).Google Scholar
  37. Koenderink, J. J., Van Doorn, A. J., Dana, K. J., & Nayar, S. (1999). Bidirectional reflectance distribution function of thoroughly pitted surfaces. International Journal of Computer Vision, 31, 129–144.CrossRefGoogle Scholar
  38. Koenderink, J. J., & van Doorn, A. J. (1987). Representation of local geometry in the visual system. Biological Cybernetics, 545, 367–375.CrossRefGoogle Scholar
  39. Leung, T., & Malik, J. (2001). Representing and recognizing the visual appearance of materials using three-dimensional textons. IJCV, 43(1), 29–44.MATHCrossRefGoogle Scholar
  40. Liu, C., Sharan, L., Rosenholtz, R., & Adelson, E. H. (2010). Exploring features in a Bayesian framework for material recognition. In CVPR, San Francisco.Google Scholar
  41. Liu, C., Yuen, J., & Torralba, A. (2009) Nonparametric scene parsing: Label transfer via dense scene alignment. In CVPR.Google Scholar
  42. Lowe, D. G. (2004). Distinctive image-features from scale-invariant keypoints. IJCV, 60(2), 91–110.CrossRefGoogle Scholar
  43. Maloney, L. T., & Yang, J. N. (2003) The illumination estimation hypothesis and surface color perception. In Color perception: From light to object (pp. 335–358). Oxford: Oxford University Press.Google Scholar
  44. Marschner, S., Westin, S. H., Arbree, A., & Moon, J. T. (2005) Measuring and modeling the appearance of finished wood. In ACM SIGGRAPH, Los Angeles (pp. 727–734).Google Scholar
  45. Marschner, S., Westin, S. H., LaFortune, E. P. F., Torrance, K. E., & Greenberg, D. P. (1999). Image-based brdf measurement including human skin. In 10th eurographics workshop on rendering, Granada (pp. 139–152).Google Scholar
  46. Matusik, W., Pfister, H., Brand, M., & McMillan, L. (2000). A data-driven reflectance model. In ACM SIGGRAPH, Louisiana (pp. 759–769).Google Scholar
  47. McHenry, K., & Ponce, J. (2005). A geodesic active contour framework for finding glass. In CVPR, San Diego (Vol. 1, pp. 1038–1044).Google Scholar
  48. McHenry, K., Ponce, J., & Forsyth, D. (2005). Finding glass. In CVPR, San Diego (Vol. 2, pp. 973–979).Google Scholar
  49. Motoyoshi, I., Nishida, S., Sharan, L., & Adelson, E. H. (2007). Image statistics and the perception of surface reflectance. Nature, 447, 206–209.CrossRefGoogle Scholar
  50. Nicodemus, F. (1965). Directional reflectance and emissivity of an opaque surface. Applied Optics, 4(7), 767–775.CrossRefGoogle Scholar
  51. Nillius, P., & Eklundh, J. -O. (2004). Classifying materials from their reflectance properties. In ECCV, Prague (Vol. 4, pp. 366–376).Google Scholar
  52. Nishida, S., & Shinya, M. (1998). Use of image-based information in judgments of surface reflectance properties. Journal of the Optical Society of America A, 15, 2951–2965.CrossRefGoogle Scholar
  53. Nishino, K., Zhang, Z., & Ikeuchi, K. (2001). Determining reflectance parameters and illumination distributions from a sparse set of images for view-dependent image synthesis. In ICCV, Vancouver (pp. 599–601).Google Scholar
  54. Oren, M., & Nayar, S. K. (1995). Generalization of the lambertian model and implications for machine vision. International Journal of Computer Vision, 14(3), 227–251.CrossRefGoogle Scholar
  55. Parikh, D., & Zitnick, L. (2010). The role of features, algorithms and data in visual recognition. In CVPR.Google Scholar
  56. Pellacini, F., Ferwerda, J. A., & Greenberg, D. P. (2000). Towards a psychophysically-based light reflection model for image synthesis. In 27th annual conference on computer graphics and interactive techniques, New Orleans (pp. 55–64). New York: ACM.Google Scholar
  57. Phong, B.-T. (1975). llumination for computer generated pictures. Communications of ACM, 18, 311–317.CrossRefGoogle Scholar
  58. Pont, S. C., & Koenderink, J. J. (2005). Bidirectional texture contrast function. IJCV, 62(1/2), 17–34.Google Scholar
  59. Ramamoorthi, R. & Hanrahan, P. (2001). A signal processing framework for inverse rendering. In ACM SIGGRAPH, Los Angeles (pp. 117–128).Google Scholar
  60. Robilotto, R., & Zaidi, Q. (2004). Limits of lightness identification of real objects under natural viewing conditions. Journal of Vision, 4(9), 779–797.CrossRefGoogle Scholar
  61. Romeiro, F., Vasilyev, Y., & Zickler, T. E. (2008). Passive reflectometry. In ECCV (Vol. 4, pp. 859–872).Google Scholar
  62. Romeiro, F., & Zickler, T. E. (2010). Blind reflectometry. In ECCV (Vol. 1, pp. 45–58).Google Scholar
  63. Rosch, E., & Lloyd, B. B. (Eds.). (1978). Cognition and categorization. In Principles of categorization. Hillsdale: Erlbaum.Google Scholar
  64. Sato, Y., Wheeler, M., & Ikeuchi, K. (1997). Object shape and reflectance modeling from observation. In ACM SIGGRAPH (pp. 379–387).Google Scholar
  65. Savarese, S & Criminisi, A. (2004). Classification of folded textiles. URL: http://research.microsoft.com/vision/cambridge/recognition/MSRC_MaterialsImageDatabase.zip, August 2004
  66. Sharan, L., Li, Y., Motoyoshi, I., Nishida, S., & Adelson, E. H. (2008). Image statistics for surface reflectance perception. Journal of the Optical Society of America A, 25(4), 846–865.CrossRefGoogle Scholar
  67. Sharan, L., Rosenholtz, R., & Adelson, E. (2009). Material perception: What can you see in a brief glance? [Abstract]. Journal of Vision, 9(8), 784–784a.CrossRefGoogle Scholar
  68. Todd, J. T., Norman, J. F., & Mingolla, E. (2004). Lightness constancy in the presence of specular highlights. Psychological Science, 15, 33–39.CrossRefGoogle Scholar
  69. Tominaga, S., & Tanaka, N. (2000). Estimating reflection parameters from a single color image. IEEE Computer Graphics and Applications, 20(5), 58–66.Google Scholar
  70. Varma, M., & Zisserman, A. (2005). A statistical approach to texture classification from single images. IJCV, 62(1–2), 61–81.Google Scholar
  71. Varma, M., & Zisserman, A. (2009). A statistical approach to material classification using image patch exemplars. TPAMI, 31(11), 2032–2047.Google Scholar
  72. Ward, G. (1992). Measuring and modeling anisotropic reflection. In 19th annual conference on computer graphics and interactive techniques (Vol. 26, pp. 265–272). New York: ACM.Google Scholar
  73. WordNet. (1998). WordNet: An electronic lexical database. Cambridge, MA: MIT Press. Google Scholar
  74. Xiao, B., & Brainard, D. H. (2008). Surface gloss and color perception of 3d objects. Visual Neuroscience, 25, 371–385.CrossRefGoogle Scholar
  75. Yu, Y., Debevec, P., Malik, J., Hawkins, T. (1999). Inverse global illumination: recovering reflectance models of real scenes from photographs. In ACM SIGGRAPH (pp. 215–224).Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Lavanya Sharan
    • 1
  • Ce Liu
    • 2
  • Ruth Rosenholtz
    • 3
  • Edward H. Adelson
    • 3
  1. 1.Department of Brain and Cognitive SciencesMassachusetts Institute of TechnologyCambridgeUSA
  2. 2.Microsoft Research New EnglandCambridgeUSA
  3. 3.Department of Brain and Cognitive SciencesMassachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations