International Journal of Computer Vision

, Volume 14, Issue 1, pp 5–24 | Cite as

Visual learning and recognition of 3-d objects from appearance

  • Hiroshi Murase
  • Shree K. Nayar


The problem of automatically learning object models for recognition and pose estimation is addressed. In contrast to the traditional approach, the recognition problem is formulated as one of matching appearance rather than shape. The appearance of an object in a two-dimensional image depends on its shape, reflectance properties, pose in the scene, and the illumination conditions. While shape and reflectance are intrinsic properties and constant for a rigid object, pose and illumination vary from scene to scene. A compact representation of object appearance is proposed that is parametrized by pose and illumination. For each object of interest, a large set of images is obtained by automatically varying pose and illumination. This image set is compressed to obtain a low-dimensional subspace, called the eigenspace, in which the object is represented as a manifold. Given an unknown input image, the recognition system projects the image to eigenspace. The object is recognized based on the manifold it lies on. The exact position of the projection on the manifold determines the object's pose in the image.

A variety of experiments are conducted using objects with complex appearance characteristics. The performance of the recognition and pose estimation algorithms is studied using over a thousand input images of sample objects. Sensitivity of recognition to the number of eigenspace dimensions and the number of learning samples is analyzed. For the objects used, appearance representation in eigenspaces with less than 20 dimensions produces accurate recognition results with an average pose estimation error of about 1.0 degree. A near real-time recognition system with 20 complex objects in the database has been developed. The paper is concluded with a discussion on various issues related to the proposed learning and recognition methodology.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Besl, P.J., and Jain, R.C. 1985. “Three-Dimensional Object Recognition,”ACM Computing Surveys, Vol. 17, No. 1, pp. 75–145.Google Scholar
  2. Chen, W.H., Smith, H., and Fralick, S.C. 1977. “A Fast Computational Algorithm for the Discrete Cosine Transform,”IEEE Transactions on Communications, Vol.25, pp.1004–1009 Vol. 18, No. 1, pp. 67–108.Google Scholar
  3. Edelman, S. and Weinshall, D. 1991. “A self-organizing multiple-view representation of 3D objects,”Biological Cybernetics Vol. 64, pp. 209–219.Google Scholar
  4. Edelman, S., Bulthoff, H., and Weinshall, D. 1989. “Stimulus familiarity determines recognition strategy for novel 3D objects,” A.I. Memo No. 1138, AI Lab, MIT.Google Scholar
  5. Fan, T.J., Medioni, G., and Nevatia, R. 1988. “Recognizing 3-D Objects Using Surface Descriptions,”Proc. of Intl. Conference on Computer Vision, pp. 474–481, Florida.Google Scholar
  6. Fukunaga, K. 1990.Introduction to Statistical Pattern Recognition, Academic Press, London.Google Scholar
  7. Householder, A.S. 1964.The theory of matrices in numerical analysis, Dover Publications, New York.Google Scholar
  8. Huttenlocher, D.P., and Ullman, S. 1990. “Recognizing solid objects by alignment with an image,”International Journal of Computer Vision, Vol. 5, No. 2, pp. 195–212.Google Scholar
  9. Ikeuchi, K., and Suehiro, T. 1992. “Recognizing Assembly Tasks using Face-Contact Relations,”Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 154–160.Google Scholar
  10. Koenderink, J.J., and van Doorn, A.J. 1979. “The internal representation of solid shape with respect to vision,”Biological Cybernetics Vol. 32, pp. 211–216.Google Scholar
  11. Mukherjee, S., and Nayar, S.K. 1994. “Appearance Based Recognition of 3D Objects using RBF Networks,” Technical Report, Department of Computer Science, Columbia University (in preparation).Google Scholar
  12. Murakami, H., and Kumar, V. 1982. “Efficient Calculation of Primary Images from a Set of Images,”IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 4, No. 5, pp. 511–515.Google Scholar
  13. Murase, H., Kimura, E., Yoshimura, M., and Miyake, Y. 1981. “An Improvement of the Auto-Correlation Matrix in Pattern Matching Method and Its Application to Handprinted ‘HI-RAGANA’,”Trans. IECE, Vol. J64-D, No. 3, pp. 276–283.Google Scholar
  14. Murase, H., and Lindenbaum, M. 1992. “Spatial Temporal Adaptive Method for Partial Eigenstructure Decomposition of Large Images,”NTT Technical Report No. 6527. AlsoIEEE Transactions on Image Processing (in press).Google Scholar
  15. Murase, H., and Nayar, S.K., 1994. “Illumination Planning for Object Recognition in Structured Environments,”Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 31–38.Google Scholar
  16. Nayar, S.K., Murase, H., and Nene, S.A. 1994. “Learning, Positioning, and Tracking Visual Appearance,”Proc. IEEE Conf. on Robotics and Automation, pp. 3237–3244.Google Scholar
  17. Nene, S.A., and Nayar, S.K. 1994. “Binary Search Through Multiple Dimensions,” Technical Report CUCS-018-94, Department of Computer Science, Columbia University.Google Scholar
  18. Oja, E. 1983.Subspace methods of Pattern Recognition, Research Studies Press, Hertfordshire.Google Scholar
  19. Poggio, T., and Girosi, F. 1990. “Networks for Approximation and Learning,”Proceedings of the IEEE, Vol. 78, No. 9, pp. 1481–1497.Google Scholar
  20. Poggio, T., and Edelman, S. 1990. “A networks that learns to recognize three-dimensional objects,”Nature, Vol. 343, pp. 263–266.Google Scholar
  21. Press, W., Flannery, B.P., Teukolsky, S.A., and Vetterling, W.T. 1988.Numerical Recipes in C, Cambridge University Press, Cambridge.Google Scholar
  22. Sirovich, L., and Kirby, M. 1987. “Low dimensional procedure for the characterization of human faces,” Journal of Optical Society of America, Vol. 4, No. 3, pp. 519–524.Google Scholar
  23. Tarr, M., and Pinker, S. 1989. “Mental rotation and orientation- dependence in shape recognition,”Cognitive Psychology, Vol. 21, pp. 233–282.Google Scholar
  24. Turk, M.A., and Pentland, A.P. 1991. “Face Recognition Using Eigenfaces,”Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–591.Google Scholar
  25. Ullman, S., and Basri, R. 1991. “Recognition by Linear Combination of Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 13, No. 10, pp. 992–1006.Google Scholar
  26. Yang, X., Sarkar, T.K., and Arvas, E. 1989. “A Survey of Conjugate Gradient Algorithms for Solution of Extreme Eigen-Problems of a Symmetric Matrix,”IEEE Transactions on Acoustics, Speech and Signal Processing, Vol. 37, No. 10, pp. 1550–1555.Google Scholar

Copyright information

© Kluwer Academic Publishers 1995

Authors and Affiliations

  • Hiroshi Murase
    • 1
  • Shree K. Nayar
    • 2
  1. 1.NTT Basic Research LaboratoryKanagawaJapan
  2. 2.Department of Computer ScienceColumbia UniversityNew York

Personalised recommendations