Image Variability Decomposition for Recovering Pose and Illumination from Object Tracking

  • Peter N. Belhumeur
  • Gregory D. Hager
Conference paper


As an object moves through space, it changes its orientation relative to the viewing camera and relative to light sources which illuminate it. As a consequence, the images of the object produced by the viewing camera may change dramatically. Thus to successfully track a moving object using computer vision, image changes due to varying pose and illumination must be accounted for. In this paper, we develop a method for object tracking that can not only accommodate large changes in object pose and illumination, but can recover these parameters as well. To do this, we separately model the image variation of the object produced by changes in pose and illumination. To track the object through each image in the sequence, we then locally search the models to find the best match, recovering the object’s orientation and illumination in the process. Throughout, we present experimental results, achieved in real-time, demonstrating the effectiveness of our methods.


Computer Vision Target Region Tracking Algorithm Object Tracking Human Face 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    P. Belhumeur and D. Kriegman. What is the set of images of an object under all possible lighting conditions? In IEEE Proc. Conf. Computer Vision and Pattern Recognition, 1996.Google Scholar
  2. [2]
    M. Black and A. Jepson. Eigentracking: Robust matching and tracking of articulated objects using a view-based representation. In European Conf. on Computer Vision, pages I:329–342, 1996.Google Scholar
  3. [3]
    A. Blake, R. Curwen, and A. Zisserman. Affine-invariant contour tracking with automatic control of spatiotemporal scale. In Proc. Int’l Conference on Computer Vision, pages 421–430. IEEE Computer Society Press, 1993.Google Scholar
  4. [4]
    R. Deriche and O. Faugeras. Tracking line segments. In O. Faugeras, editor, Proc. 1st European Conference on Computer Vision, pages 259–268. Springer-Verlag, 1990.Google Scholar
  5. [5]
    R. Epstein, A. Yuille, and P. N. Belhumeur. Learning and recognizing objects using illumination subspaces. In Proc. of the Int. Workshop on Object Representation for Computer Vision, 1996.Google Scholar
  6. [6]
    D. Gavrila and L. Davis. Tracking humans in action: A 3D model-based approach. In Proc. Image Understanding Workshop, pages 737–746, 1996.Google Scholar
  7. [7]
    M. Goresky and R. Macpherson. Stratified Morse Theory. Springer-Verlag, New York, 1980.Google Scholar
  8. [8]
    G. D. Hager and R N. Belhumeur. Real-time tracking of image regions with changes in geometry and illumination. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 403–410. IEEE Computer Society Press, 1996.Google Scholar
  9. [9]
    G. D. Hager and K. Toyama. XVision: A portable substrate for real-time vision applications. Computer Vision and Image Understanding, 69(1), 1998.Google Scholar
  10. [10]
    P. Hallinan. A Deformable Model for Face Recognition Under Arbitrary Lighting Conditions. PhD thesis, Harvard University, 1995.Google Scholar
  11. [11]
    M. Isard and A. Blake. Contour tracking by stochastic propogation of conditional density. In Computer Vision-ECCV’96, volume 1, pages 343–356. Springer Verlag, 1996.Google Scholar
  12. [12]
    D. W. Jacobs. Space efficient 3D model indexing. In Proc. IEEE Conf. on Comp. Vision and Patt. Recog., pages 439–444, 1992.Google Scholar
  13. [13]
    J. Koenderink and A. Van Doorn. Affine structure from motion. J. Opt. Soc. Am., 8(2):337–385, 1991.CrossRefGoogle Scholar
  14. [14]
    L. Sirovitch and M. Kirby. Low-dimensional procedure for the characterization of human faces. J. Optical Soc. of America A,2:519–524, 1987.CrossRefGoogle Scholar
  15. [15]
    D. G. Lowe. Robust model-based motion tracking through the integration of search and estimation. Int’l Journal of Computer Vision, 8(2): 113–122, 1992.CrossRefGoogle Scholar
  16. [16]
    B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proc. of the 7th International Joint Conference on Artificial Intelligence, pages 674–679, 1981.Google Scholar
  17. [17]
    Y. Moses, Y. Adini, and S. Ullman. Face recognition: The problem of compensating for changes in illumination direction. In European Conf. on Computer Vision, pages 286–296, 1994.Google Scholar
  18. [18]
    H. Murase and S. Nayar. Visual learning and recognition of 3-D objects from appearence. Int. J. Computer Vision, 14(5–24), 1995.CrossRefGoogle Scholar
  19. [19]
    T. Poggio and K. Sung. Example-based learning for view-based human face detection. In Proc. Image Understanding Workshop, pages II:843–850, 1994.Google Scholar
  20. [20]
    D. Reynard, A. Wildenberg, A. Blake, and J. Marchant. Learning dynamics of complex motions from image sequences. In Proc. European Conf. on Computer Vision, pages I:357–368, 1996.Google Scholar
  21. [21]
    L. Shapiro, A. Zisserman, and M. Brady. 3D motion recovery via affine epipolar geometry. Int. J. Computer Vision, 16(2):147–182, October 1995.CrossRefGoogle Scholar
  22. [22]
    A. Shashua. Geometry and Photometry in 3D Visual Recognition. PhD thesis, MIT, 1992.Google Scholar
  23. [23]
    J. Shi and C. Tomasi. Good features to track. In Computer Vision and Patt. Recog.,pages 593–600. IEEE Computer Society Press, 1994.Google Scholar
  24. [24]
    C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. Int. J. Computer Vision, 9(2):137–154, 1992.CrossRefGoogle Scholar
  25. [25]
    M. Turk and A. Pentland. Eigenfaces for recognition. J. of Cognitive Neuroscience, 3(1), 1991.Google Scholar
  26. [26]
    S. Ullman and R. Basri. Recognition by linear combinations of models. IEEE Trans. Pattern Analysis and Machine Intelligence, 13(10):992–1006, 1991.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 1999

Authors and Affiliations

  • Peter N. Belhumeur
    • 1
  • Gregory D. Hager
    • 1
  1. 1.Center for Computational Vision and ControlYale UniversityNew HeavenUSA

Personalised recommendations