Abstract
Our world consists not only of objects and scenes but also of materials of various kinds. Being able to recognize the materials that surround us (e.g., plastic, glass, concrete) is important for humans as well as for computer vision systems. Unfortunately, materials have received little attention in the visual recognition literature, and very few computer vision systems have been designed specifically to recognize materials. In this paper, we present a system for recognizing material categories from single images. We propose a set of low and mid-level image features that are based on studies of human material recognition, and we combine these features using an SVM classifier. Our system outperforms a state-of-the-art system (Varma and Zisserman, TPAMI 31(11):2032–2047, 2009) on a challenging database of real-world material categories (Sharan et al., J Vis 9(8):784–784a, 2009). When the performance of our system is compared directly to that of human observers, humans outperform our system quite easily. However, when we account for the local nature of our image features and the surface properties they measure (e.g., color, texture, local shape), our system rivals human performance. We suggest that future progress in material recognition will come from: (1) a deeper understanding of the role of non-local surface properties (e.g., extended highlights, object identity); and (2) efforts to model such non-local surface properties in images.
Notes
In this paper, we use the terms “local features” and “non-local features” relative to the size of the surface of interest and not the size of the image. The images we will consider in this paper correspond to the spatial scale depicted in Fig. 3b. For this scale, features such as color, texture, and local shape are considered local features, whereas features such as outline shape and object identity are considered non-local features.
For the spatial scales depicted in FMD images, object properties such as outline shape are “non-local” in nature. Meanwhile, local image properties such as color or texture can vary across the surface of interest, and hence, they are “local” in nature.
References
Adelson, E. H. (2001). On seeing stuff: The perception of materials by humans and machines. In SPIE, human vision and electronic imaging VI (Vol. 4299, pp. 1–12).
Bae, S., Paris, S., & Durand, F. (2006). Two-scale tone management for photographic look. In ACM SIGGRAPH, New York.
Belongie, S., Malik, J., & Puzicha, J. (2002). Shape matching and object recognition using shape contexts. TPAMI, 24(4), 509–522.
Berzhanskaya, J., Swaminathan, G., Beck, J., & Mingolla, E. (2005). Remote effects of highlights on gloss perception. Perception, 34(5), 565–575.
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
Bloj, M., Kersten, D., & Hurlbert, A. C. (1999). Perception of three-dimensional shape influences color perception through mutual illumination. Nature, 402, 877–879.
Boivin, S., & Gagalowicz, A. (2001). Image-based rendering of diffuse, specular and glossy surfaces from a single image. In ACM SIGGRAPH, Los Angeles (pp. 107–116).
Boyaci, H., Maloney, L. T., & Hersh, S. (2003). The effect of perceived surface orientation on perceived surface albedo in binocularly viewed scenes. Journal of Vision, 3, 541–553.
Brainard, D. H., Kraft, J. M., & Longere, P. (2003). Color perception: From light to object. In Color constancy: Developing empirical tests of computational models (pp. 307–334). Oxford: Oxford University Press.
Burges, C. (1998). A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2), 121–167.
Canny, J. (1986). A computational approach to edge detection. TPAMI, 80(6), 679–698.
Caputo, B., Hayman, E., & Mallikarjuna, P. (2005). Class-specific material categorization. In Proceedings of the ICCV, Beijing (Vol. 2, pp. 1597–1604).
Caputo, B., Hayman, E., Fritz, M., & Jan-Olof E. (2007) Classifying materials in the real world. Martign: IDIAP.
Cula, O. G., & Dana, K. J. (2004a). 3D texture recognition using bidirectional feature histograms. IJCV, 59(1), 33–60.
Cula, O. G., Dana, K. J., Murphy, F. P., & Rao, B. K. (2004). Bidirectional imaging and modeling of skin texture. IEEE Transactions on Biomedical Engineering, 51(12), 2148–2159.
Cula, O. J., & Dana, K. J. (2004b). 3d texture recognition using bidirectional feature histograms. International Journal of Computer Vision, 59(1), 33–60.
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In CVPR, Montbonnot (Vol. 2, pp. 886–893).
Dana, K. J., & Nayar, S. (1998). Histogram model for 3d textures. In CVPR (pp. 618–624).
Dana, K. J., Van-Ginneken, B., Nayar, S. K., & Koenderink, J. J. (1999). Reflectance and texture of real world surfaces. ACM Transactions on Graphics, 18(1), 1–34.
Debevec, P., Hawkins, T., Tchou, C., Duiker, H. P., Sarokin, W., & Sagar, M. (2000). Acquiring the reflectance field of a human face. In ACM SIGGRAPH, Louisiana (pp. 145–156).
Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., & Martinez, P. (2004). Estimating surface reflectance properties of a complex scene under captured natural illumination. ICT-TR-06, University of Southern California.
Dror, R., Adelson, E. H., & Alan S. Willsky (2001). Recognition of surface reflectance properties from a single image under unknown real-world illumination. In IEEE Workshop on identifying objects across variation in lighting.
Durand, F., & Dorsey, J. (2002). Fast bilateral filtering for the display of high-dynamic-range images. In ACM SIGGRAPH, San Antonio.
Efros, A. A., & Freeman, W. T. (2001). Image quilting for texture synthesis and transfer. In ACM SIGGRAPH, Los Angeles.
Fei-Fei, L., & Perona, P. (2005). A bayesian hierarchical model for learning natural scene categories. In CVPR, San Diego (Vol. 2, pp. 524–531).
Fleming, R. W., & Bülthoff, H. (2005). Low-level image cues in the perception of translucent materials. ACM Transactions on Applied Perception, 2(3), 346–382.
Fleming, R. W., Dror, R., & Adelson, E. H. (2003). Real world illumination and the perception of surface reflectance properties. Journal of Vision, 3(5), 347–368.
Fleming, R. W., Torralba, A., & Adelson, E. H. (2004). Specular reflections and the perception of shape. Journal of Vision, 4(9), 798–820.
Forsyth, D., & Fleck, M. M. (1999). Automatic detection of human nudes. IJCV, 32(1), 63–77.
Fritz, M., Black, M., Bradski, G., & Darrell, T. (2009). An additive latent feature model for transparent object recognition. In NIPS.
Gilchrist, A., Kossyfidis, C., Bonato, F., Agostini, T., Cataliotti, J., Li, X., et al. (1999). An anchoring theory of lightness perception. Psychological Review, 106, 795–834.
He, X. D., Torrance, K. E., Sillion, F. S., & Greenberg, D. P. (1991). A comprehensive physical model for light reflection. In 18th annual conference on computer graphics and interactive techniques (Vol. 25, pp. 175–186). New York: ACM.
Ho, Y. X., Landy, M. S., & Maloney, L. T. (2008). Conjoint measurement of gloss and surface texture. Psychological Science, 19(2), 196–204.
Hu, D., Bo, L., & Ren, X. (2011). Robust material recognition for everyday objects. In BMVC, Dundee.
Jensen, H. W., Marschner, S., Levoy, M. & Hanrahan, P. (2001). A practical model for subsurface light transport. In ACM SIGGRAPH, Los Angeles (pp. 511–518).
Khan, E. A., Reinhard, E., Fleming, R. W., & H. Bülthoff, H. (2006). Image-based material editing. In ACM SIGGRAPH, Boston (pp. 654–663).
Koenderink, J. J., Van Doorn, A. J., Dana, K. J., & Nayar, S. (1999). Bidirectional reflectance distribution function of thoroughly pitted surfaces. International Journal of Computer Vision, 31, 129–144.
Koenderink, J. J., & van Doorn, A. J. (1987). Representation of local geometry in the visual system. Biological Cybernetics, 545, 367–375.
Leung, T., & Malik, J. (2001). Representing and recognizing the visual appearance of materials using three-dimensional textons. IJCV, 43(1), 29–44.
Liu, C., Sharan, L., Rosenholtz, R., & Adelson, E. H. (2010). Exploring features in a Bayesian framework for material recognition. In CVPR, San Francisco.
Liu, C., Yuen, J., & Torralba, A. (2009) Nonparametric scene parsing: Label transfer via dense scene alignment. In CVPR.
Lowe, D. G. (2004). Distinctive image-features from scale-invariant keypoints. IJCV, 60(2), 91–110.
Maloney, L. T., & Yang, J. N. (2003) The illumination estimation hypothesis and surface color perception. In Color perception: From light to object (pp. 335–358). Oxford: Oxford University Press.
Marschner, S., Westin, S. H., Arbree, A., & Moon, J. T. (2005) Measuring and modeling the appearance of finished wood. In ACM SIGGRAPH, Los Angeles (pp. 727–734).
Marschner, S., Westin, S. H., LaFortune, E. P. F., Torrance, K. E., & Greenberg, D. P. (1999). Image-based brdf measurement including human skin. In 10th eurographics workshop on rendering, Granada (pp. 139–152).
Matusik, W., Pfister, H., Brand, M., & McMillan, L. (2000). A data-driven reflectance model. In ACM SIGGRAPH, Louisiana (pp. 759–769).
McHenry, K., & Ponce, J. (2005). A geodesic active contour framework for finding glass. In CVPR, San Diego (Vol. 1, pp. 1038–1044).
McHenry, K., Ponce, J., & Forsyth, D. (2005). Finding glass. In CVPR, San Diego (Vol. 2, pp. 973–979).
Motoyoshi, I., Nishida, S., Sharan, L., & Adelson, E. H. (2007). Image statistics and the perception of surface reflectance. Nature, 447, 206–209.
Nicodemus, F. (1965). Directional reflectance and emissivity of an opaque surface. Applied Optics, 4(7), 767–775.
Nillius, P., & Eklundh, J. -O. (2004). Classifying materials from their reflectance properties. In ECCV, Prague (Vol. 4, pp. 366–376).
Nishida, S., & Shinya, M. (1998). Use of image-based information in judgments of surface reflectance properties. Journal of the Optical Society of America A, 15, 2951–2965.
Nishino, K., Zhang, Z., & Ikeuchi, K. (2001). Determining reflectance parameters and illumination distributions from a sparse set of images for view-dependent image synthesis. In ICCV, Vancouver (pp. 599–601).
Oren, M., & Nayar, S. K. (1995). Generalization of the lambertian model and implications for machine vision. International Journal of Computer Vision, 14(3), 227–251.
Parikh, D., & Zitnick, L. (2010). The role of features, algorithms and data in visual recognition. In CVPR.
Pellacini, F., Ferwerda, J. A., & Greenberg, D. P. (2000). Towards a psychophysically-based light reflection model for image synthesis. In 27th annual conference on computer graphics and interactive techniques, New Orleans (pp. 55–64). New York: ACM.
Phong, B.-T. (1975). llumination for computer generated pictures. Communications of ACM, 18, 311–317.
Pont, S. C., & Koenderink, J. J. (2005). Bidirectional texture contrast function. IJCV, 62(1/2), 17–34.
Ramamoorthi, R. & Hanrahan, P. (2001). A signal processing framework for inverse rendering. In ACM SIGGRAPH, Los Angeles (pp. 117–128).
Robilotto, R., & Zaidi, Q. (2004). Limits of lightness identification of real objects under natural viewing conditions. Journal of Vision, 4(9), 779–797.
Romeiro, F., Vasilyev, Y., & Zickler, T. E. (2008). Passive reflectometry. In ECCV (Vol. 4, pp. 859–872).
Romeiro, F., & Zickler, T. E. (2010). Blind reflectometry. In ECCV (Vol. 1, pp. 45–58).
Rosch, E., & Lloyd, B. B. (Eds.). (1978). Cognition and categorization. In Principles of categorization. Hillsdale: Erlbaum.
Sato, Y., Wheeler, M., & Ikeuchi, K. (1997). Object shape and reflectance modeling from observation. In ACM SIGGRAPH (pp. 379–387).
Savarese, S & Criminisi, A. (2004). Classification of folded textiles. URL: http://research.microsoft.com/vision/cambridge/recognition/MSRC_MaterialsImageDatabase.zip, August 2004
Sharan, L., Li, Y., Motoyoshi, I., Nishida, S., & Adelson, E. H. (2008). Image statistics for surface reflectance perception. Journal of the Optical Society of America A, 25(4), 846–865.
Sharan, L., Rosenholtz, R., & Adelson, E. (2009). Material perception: What can you see in a brief glance? [Abstract]. Journal of Vision, 9(8), 784–784a.
Todd, J. T., Norman, J. F., & Mingolla, E. (2004). Lightness constancy in the presence of specular highlights. Psychological Science, 15, 33–39.
Tominaga, S., & Tanaka, N. (2000). Estimating reflection parameters from a single color image. IEEE Computer Graphics and Applications, 20(5), 58–66.
Varma, M., & Zisserman, A. (2005). A statistical approach to texture classification from single images. IJCV, 62(1–2), 61–81.
Varma, M., & Zisserman, A. (2009). A statistical approach to material classification using image patch exemplars. TPAMI, 31(11), 2032–2047.
Ward, G. (1992). Measuring and modeling anisotropic reflection. In 19th annual conference on computer graphics and interactive techniques (Vol. 26, pp. 265–272). New York: ACM.
WordNet. (1998). WordNet: An electronic lexical database. Cambridge, MA: MIT Press.
Xiao, B., & Brainard, D. H. (2008). Surface gloss and color perception of 3d objects. Visual Neuroscience, 25, 371–385.
Yu, Y., Debevec, P., Malik, J., Hawkins, T. (1999). Inverse global illumination: recovering reflectance models of real scenes from photographs. In ACM SIGGRAPH (pp. 215–224).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sharan, L., Liu, C., Rosenholtz, R. et al. Recognizing Materials Using Perceptually Inspired Features. Int J Comput Vis 103, 348–371 (2013). https://doi.org/10.1007/s11263-013-0609-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-013-0609-0