Skip to main content
SpringerLink
Log in

Recognizing Materials Using Perceptually Inspired Features

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Our world consists not only of objects and scenes but also of materials of various kinds. Being able to recognize the materials that surround us (e.g., plastic, glass, concrete) is important for humans as well as for computer vision systems. Unfortunately, materials have received little attention in the visual recognition literature, and very few computer vision systems have been designed specifically to recognize materials. In this paper, we present a system for recognizing material categories from single images. We propose a set of low and mid-level image features that are based on studies of human material recognition, and we combine these features using an SVM classifier. Our system outperforms a state-of-the-art system (Varma and Zisserman, TPAMI 31(11):2032–2047, 2009) on a challenging database of real-world material categories (Sharan et al., J Vis 9(8):784–784a, 2009). When the performance of our system is compared directly to that of human observers, humans outperform our system quite easily. However, when we account for the local nature of our image features and the surface properties they measure (e.g., color, texture, local shape), our system rivals human performance. We suggest that future progress in material recognition will come from: (1) a deeper understanding of the role of non-local surface properties (e.g., extended highlights, object identity); and (2) efforts to model such non-local surface properties in images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. In this paper, we use the terms “local features” and “non-local features” relative to the size of the surface of interest and not the size of the image. The images we will consider in this paper correspond to the spatial scale depicted in Fig. 3b. For this scale, features such as color, texture, and local shape are considered local features, whereas features such as outline shape and object identity are considered non-local features.

  2. For the spatial scales depicted in FMD images, object properties such as outline shape are “non-local” in nature. Meanwhile, local image properties such as color or texture can vary across the surface of interest, and hence, they are “local” in nature.

  3. Kernel comparison results were obtained by averaging over 14 different splits of FMD into training and tests sets. All other results in Sects. 6.1–6.4 pertain to a single split.

References

  • Adelson, E. H. (2001). On seeing stuff: The perception of materials by humans and machines. In SPIE, human vision and electronic imaging VI (Vol. 4299, pp. 1–12).

  • Bae, S., Paris, S., & Durand, F. (2006). Two-scale tone management for photographic look. In ACM SIGGRAPH, New York.

  • Belongie, S., Malik, J., & Puzicha, J. (2002). Shape matching and object recognition using shape contexts. TPAMI, 24(4), 509–522.

    Article  Google Scholar 

  • Berzhanskaya, J., Swaminathan, G., Beck, J., & Mingolla, E. (2005). Remote effects of highlights on gloss perception. Perception, 34(5), 565–575.

    Article  Google Scholar 

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

    MATH  Google Scholar 

  • Bloj, M., Kersten, D., & Hurlbert, A. C. (1999). Perception of three-dimensional shape influences color perception through mutual illumination. Nature, 402, 877–879.

    Google Scholar 

  • Boivin, S., & Gagalowicz, A. (2001). Image-based rendering of diffuse, specular and glossy surfaces from a single image. In ACM SIGGRAPH, Los Angeles (pp. 107–116).

  • Boyaci, H., Maloney, L. T., & Hersh, S. (2003). The effect of perceived surface orientation on perceived surface albedo in binocularly viewed scenes. Journal of Vision, 3, 541–553.

    Article  Google Scholar 

  • Brainard, D. H., Kraft, J. M., & Longere, P. (2003). Color perception: From light to object. In Color constancy: Developing empirical tests of computational models (pp. 307–334). Oxford: Oxford University Press.

  • Burges, C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2), 121–167.

    Article  Google Scholar 

  • Canny, J. (1986). A computational approach to edge detection. TPAMI, 80(6), 679–698.

    Article  Google Scholar 

  • Caputo, B., Hayman, E., & Mallikarjuna, P. (2005). Class-specific material categorization. In Proceedings of the ICCV, Beijing (Vol. 2, pp. 1597–1604).

  • Caputo, B., Hayman, E., Fritz, M., & Jan-Olof E. (2007) Classifying materials in the real world. Martign: IDIAP.

  • Cula, O. G., & Dana, K. J. (2004a). 3D texture recognition using bidirectional feature histograms. IJCV, 59(1), 33–60.

    Article  Google Scholar 

  • Cula, O. G., Dana, K. J., Murphy, F. P., & Rao, B. K. (2004). Bidirectional imaging and modeling of skin texture. IEEE Transactions on Biomedical Engineering, 51(12), 2148–2159.

    Article  Google Scholar 

  • Cula, O. J., & Dana, K. J. (2004b). 3d texture recognition using bidirectional feature histograms. International Journal of Computer Vision, 59(1), 33–60.

    Article  Google Scholar 

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In CVPR, Montbonnot (Vol. 2, pp. 886–893).

  • Dana, K. J., & Nayar, S. (1998). Histogram model for 3d textures. In CVPR (pp. 618–624).

  • Dana, K. J., Van-Ginneken, B., Nayar, S. K., & Koenderink, J. J. (1999). Reflectance and texture of real world surfaces. ACM Transactions on Graphics, 18(1), 1–34.

    Google Scholar 

  • Debevec, P., Hawkins, T., Tchou, C., Duiker, H. P., Sarokin, W., & Sagar, M. (2000). Acquiring the reflectance field of a human face. In ACM SIGGRAPH, Louisiana (pp. 145–156).

  • Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., & Martinez, P. (2004). Estimating surface reflectance properties of a complex scene under captured natural illumination. ICT-TR-06, University of Southern California.

  • Dror, R., Adelson, E. H., & Alan S. Willsky (2001). Recognition of surface reflectance properties from a single image under unknown real-world illumination. In IEEE Workshop on identifying objects across variation in lighting.

  • Durand, F., & Dorsey, J. (2002). Fast bilateral filtering for the display of high-dynamic-range images. In ACM SIGGRAPH, San Antonio.

  • Efros, A. A., & Freeman, W. T. (2001). Image quilting for texture synthesis and transfer. In ACM SIGGRAPH, Los Angeles.

  • Fei-Fei, L., & Perona, P. (2005). A bayesian hierarchical model for learning natural scene categories. In CVPR, San Diego (Vol. 2, pp. 524–531).

  • Fleming, R. W., & Bülthoff, H. (2005). Low-level image cues in the perception of translucent materials. ACM Transactions on Applied Perception, 2(3), 346–382.

    Article  Google Scholar 

  • Fleming, R. W., Dror, R., & Adelson, E. H. (2003). Real world illumination and the perception of surface reflectance properties. Journal of Vision, 3(5), 347–368.

    Article  Google Scholar 

  • Fleming, R. W., Torralba, A., & Adelson, E. H. (2004). Specular reflections and the perception of shape. Journal of Vision, 4(9), 798–820.

    Google Scholar 

  • Forsyth, D., & Fleck, M. M. (1999). Automatic detection of human nudes. IJCV, 32(1), 63–77.

    Article  Google Scholar 

  • Fritz, M., Black, M., Bradski, G., & Darrell, T. (2009). An additive latent feature model for transparent object recognition. In NIPS.

  • Gilchrist, A., Kossyfidis, C., Bonato, F., Agostini, T., Cataliotti, J., Li, X., et al. (1999). An anchoring theory of lightness perception. Psychological Review, 106, 795–834.

    Article  Google Scholar 

  • He, X. D., Torrance, K. E., Sillion, F. S., & Greenberg, D. P. (1991). A comprehensive physical model for light reflection. In 18th annual conference on computer graphics and interactive techniques (Vol. 25, pp. 175–186). New York: ACM.

  • Ho, Y. X., Landy, M. S., & Maloney, L. T. (2008). Conjoint measurement of gloss and surface texture. Psychological Science, 19(2), 196–204.

    Google Scholar 

  • Hu, D., Bo, L., & Ren, X. (2011). Robust material recognition for everyday objects. In BMVC, Dundee.

  • Jensen, H. W., Marschner, S., Levoy, M. & Hanrahan, P. (2001). A practical model for subsurface light transport. In ACM SIGGRAPH, Los Angeles (pp. 511–518).

  • Khan, E. A., Reinhard, E., Fleming, R. W., & H. Bülthoff, H. (2006). Image-based material editing. In ACM SIGGRAPH, Boston (pp. 654–663).

  • Koenderink, J. J., Van Doorn, A. J., Dana, K. J., & Nayar, S. (1999). Bidirectional reflectance distribution function of thoroughly pitted surfaces. International Journal of Computer Vision, 31, 129–144.

    Article  Google Scholar 

  • Koenderink, J. J., & van Doorn, A. J. (1987). Representation of local geometry in the visual system. Biological Cybernetics, 545, 367–375.

    Article  Google Scholar 

  • Leung, T., & Malik, J. (2001). Representing and recognizing the visual appearance of materials using three-dimensional textons. IJCV, 43(1), 29–44.

    Article  MATH  Google Scholar 

  • Liu, C., Sharan, L., Rosenholtz, R., & Adelson, E. H. (2010). Exploring features in a Bayesian framework for material recognition. In CVPR, San Francisco.

  • Liu, C., Yuen, J., & Torralba, A. (2009) Nonparametric scene parsing: Label transfer via dense scene alignment. In CVPR.

  • Lowe, D. G. (2004). Distinctive image-features from scale-invariant keypoints. IJCV, 60(2), 91–110.

    Article  Google Scholar 

  • Maloney, L. T., & Yang, J. N. (2003) The illumination estimation hypothesis and surface color perception. In Color perception: From light to object (pp. 335–358). Oxford: Oxford University Press.

  • Marschner, S., Westin, S. H., Arbree, A., & Moon, J. T. (2005) Measuring and modeling the appearance of finished wood. In ACM SIGGRAPH, Los Angeles (pp. 727–734).

  • Marschner, S., Westin, S. H., LaFortune, E. P. F., Torrance, K. E., & Greenberg, D. P. (1999). Image-based brdf measurement including human skin. In 10th eurographics workshop on rendering, Granada (pp. 139–152).

  • Matusik, W., Pfister, H., Brand, M., & McMillan, L. (2000). A data-driven reflectance model. In ACM SIGGRAPH, Louisiana (pp. 759–769).

  • McHenry, K., & Ponce, J. (2005). A geodesic active contour framework for finding glass. In CVPR, San Diego (Vol. 1, pp. 1038–1044).

  • McHenry, K., Ponce, J., & Forsyth, D. (2005). Finding glass. In CVPR, San Diego (Vol. 2, pp. 973–979).

  • Motoyoshi, I., Nishida, S., Sharan, L., & Adelson, E. H. (2007). Image statistics and the perception of surface reflectance. Nature, 447, 206–209.

    Article  Google Scholar 

  • Nicodemus, F. (1965). Directional reflectance and emissivity of an opaque surface. Applied Optics, 4(7), 767–775.

    Article  Google Scholar 

  • Nillius, P., & Eklundh, J. -O. (2004). Classifying materials from their reflectance properties. In ECCV, Prague (Vol. 4, pp. 366–376).

  • Nishida, S., & Shinya, M. (1998). Use of image-based information in judgments of surface reflectance properties. Journal of the Optical Society of America A, 15, 2951–2965.

    Article  Google Scholar 

  • Nishino, K., Zhang, Z., & Ikeuchi, K. (2001). Determining reflectance parameters and illumination distributions from a sparse set of images for view-dependent image synthesis. In ICCV, Vancouver (pp. 599–601).

  • Oren, M., & Nayar, S. K. (1995). Generalization of the lambertian model and implications for machine vision. International Journal of Computer Vision, 14(3), 227–251.

    Article  Google Scholar 

  • Parikh, D., & Zitnick, L. (2010). The role of features, algorithms and data in visual recognition. In CVPR.

  • Pellacini, F., Ferwerda, J. A., & Greenberg, D. P. (2000). Towards a psychophysically-based light reflection model for image synthesis. In 27th annual conference on computer graphics and interactive techniques, New Orleans (pp. 55–64). New York: ACM.

  • Phong, B.-T. (1975). llumination for computer generated pictures. Communications of ACM, 18, 311–317.

    Article  Google Scholar 

  • Pont, S. C., & Koenderink, J. J. (2005). Bidirectional texture contrast function. IJCV, 62(1/2), 17–34.

    Google Scholar 

  • Ramamoorthi, R. & Hanrahan, P. (2001). A signal processing framework for inverse rendering. In ACM SIGGRAPH, Los Angeles (pp. 117–128).

  • Robilotto, R., & Zaidi, Q. (2004). Limits of lightness identification of real objects under natural viewing conditions. Journal of Vision, 4(9), 779–797.

    Article  Google Scholar 

  • Romeiro, F., Vasilyev, Y., & Zickler, T. E. (2008). Passive reflectometry. In ECCV (Vol. 4, pp. 859–872).

  • Romeiro, F., & Zickler, T. E. (2010). Blind reflectometry. In ECCV (Vol. 1, pp. 45–58).

  • Rosch, E., & Lloyd, B. B. (Eds.). (1978). Cognition and categorization. In Principles of categorization. Hillsdale: Erlbaum.

  • Sato, Y., Wheeler, M., & Ikeuchi, K. (1997). Object shape and reflectance modeling from observation. In ACM SIGGRAPH (pp. 379–387).

  • Savarese, S & Criminisi, A. (2004). Classification of folded textiles. URL: http://research.microsoft.com/vision/cambridge/recognition/MSRC_MaterialsImageDatabase.zip, August 2004

  • Sharan, L., Li, Y., Motoyoshi, I., Nishida, S., & Adelson, E. H. (2008). Image statistics for surface reflectance perception. Journal of the Optical Society of America A, 25(4), 846–865.

    Article  Google Scholar 

  • Sharan, L., Rosenholtz, R., & Adelson, E. (2009). Material perception: What can you see in a brief glance? [Abstract]. Journal of Vision, 9(8), 784–784a.

    Article  Google Scholar 

  • Todd, J. T., Norman, J. F., & Mingolla, E. (2004). Lightness constancy in the presence of specular highlights. Psychological Science, 15, 33–39.

    Article  Google Scholar 

  • Tominaga, S., & Tanaka, N. (2000). Estimating reflection parameters from a single color image. IEEE Computer Graphics and Applications, 20(5), 58–66.

    Google Scholar 

  • Varma, M., & Zisserman, A. (2005). A statistical approach to texture classification from single images. IJCV, 62(1–2), 61–81.

    Google Scholar 

  • Varma, M., & Zisserman, A. (2009). A statistical approach to material classification using image patch exemplars. TPAMI, 31(11), 2032–2047.

    Google Scholar 

  • Ward, G. (1992). Measuring and modeling anisotropic reflection. In 19th annual conference on computer graphics and interactive techniques (Vol. 26, pp. 265–272). New York: ACM.

  • WordNet. (1998). WordNet: An electronic lexical database. Cambridge, MA: MIT Press.

  • Xiao, B., & Brainard, D. H. (2008). Surface gloss and color perception of 3d objects. Visual Neuroscience, 25, 371–385.

    Article  Google Scholar 

  • Yu, Y., Debevec, P., Malik, J., Hawkins, T. (1999). Inverse global illumination: recovering reflectance models of real scenes from photographs. In ACM SIGGRAPH (pp. 215–224).

Download references

Author information

Authors and Affiliations

  1. Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA

    Lavanya Sharan

  2. Microsoft Research New England, One Memorial Drive, Cambridge, MA, 02142, USA

    Ce Liu

  3. Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA

    Ruth Rosenholtz & Edward H. Adelson

Authors
  1. Lavanya Sharan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ce Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruth Rosenholtz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Edward H. Adelson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lavanya Sharan.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sharan, L., Liu, C., Rosenholtz, R. et al. Recognizing Materials Using Perceptually Inspired Features. Int J Comput Vis 103, 348–371 (2013). https://doi.org/10.1007/s11263-013-0609-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-013-0609-0

Keywords

Search

Navigation