International Journal of Computer Vision

, Volume 38, Issue 2, pp 99–127 | Cite as

Optical Flow Constraints on Deformable Models with Applications to Face Tracking

  • Douglas DeCarlo
  • Dimitris Metaxas


Optical flow provides a constraint on the motion of a deformable model. We derive and solve a dynamic system incorporating flow as a hard constraint, producing a model-based least-squares optical flow solution. Our solution also ensures the constraint remains satisfied when combined with edge information, which helps combat tracking error accumulation. Constraint enforcement can be relaxed using a Kalman filter, which permits controlled constraint violations based on the noise present in the optical flow information, and enables optical flow and edge information to be combined more robustly and efficiently. We apply this framework to the estimation of face shape and motion using a 3D deformable face model. This model uses a small number of parameters to describe a rich variety of face shapes and facial expressions. We present experiments in extracting the shape and motion of a face from image sequences which validate the accuracy of the method. They also demonstrate that our treatment of optical flow as a hard constraint, as well as our use of a Kalman filter to reconcile these constraints with the uncertainty in the optical flow, are vital for improving the performance of our system.


Kalman Filter Tracking Error Optical Flow Constraint Violation Hard Constraint 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Adiv, G. 1985. Determining 3-d motion and structure from optical flow generated by several moving objects. IEEE Pattern Analysis and Machine Intelligence, 7(4):384–401.Google Scholar
  2. Ayache, N.J. and Faugeras, O.D. 1988. Building, registering, and fusing noisy visual maps. IJRR, 7(6):45–65.Google Scholar
  3. Bar-Shalom, Y. and Fortmann, T. 1988. Tracking and Data Association. Academic Press.Google Scholar
  4. Basu, S., Essa, I., and Pentland, A. 1996. Motion regularization for model-based head tracking. In Proceedings ICPR’ 96, p. C8A.3.Google Scholar
  5. Bergen, J., Anandan, P., Hanna, K., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In Proceedings ECCV’ 92, pp. 237–252.Google Scholar
  6. Black, M. and Yacoob, Y. 1995. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Proceedings ICCV’ 95, pp. 374–381.Google Scholar
  7. Broida, T.J. and Chellappa, R. 1996. Estimation of object motion parameters from noisy images. IEEE Pattern Analysis and Machine Intelligence, 8(1):90–99.Google Scholar
  8. Bryson, A. and Ho, Y. 1975. Applied Optimal Control. Halsted Press.Google Scholar
  9. Choi, C., Aizawa, K., Harashima, H., and Takebe, T. 1994. Analysis and synthesis of facial image sequences in model-based image coding. IEEE Circuits and Systems for Video Technology, 4(3):257–275.Google Scholar
  10. Cootes, T., Edwards, G., and Taylor, C. 1998. Active appearance models. In ECCV98, pp. II:484–498.Google Scholar
  11. DeCarlo, D. and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In Proceedings CVPR’ 96, pp. 231–238.Google Scholar
  12. Durrant-Whyte, H.F. 1987. Consistent integration and propagation of disparate sensor observations. IJRR, 6(3):3–24.Google Scholar
  13. Ekman, P. and Friesen, W. 1978. The Facial Action Coding System. Consulting Psychologist Press, Inc.Google Scholar
  14. Essa, I.A. and Pentland, A.P. 1997. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Pattern Analysis and Machine Intelligence, 19(7):757–763.Google Scholar
  15. Farkas, L. 1987. Anthropometric Facial Proportions in Medicine. Thomas Books.Google Scholar
  16. Farkas, L. 1994. Anthropometry of the Head and Face. Raven Press.Google Scholar
  17. Fua, P. and Brechbuhler, C. 1997. Imposing hard constraints on deformable models through optimization in orthogonal subspaces. Computer Vision and Image Understanding, 65(2):148–162.Google Scholar
  18. Fua, P. and Leclerc, Y.G. 1995. Object-centered surface reconstruction: Combining multi-image stereo and shading. International Journal of Computer Vision, 16(1):35–56.Google Scholar
  19. Gelb, A. 1974. Applied Optimal Estimation. MIT Press.Google Scholar
  20. Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proceedings SIGGRAPH’ 98, pp. 55–66.Google Scholar
  21. Hel-Or, Y. and Werman, M. 1996. Constraint fusion for recognition and localization of articulated objects. International Journal of Computer Vision, 19(1):5–28.Google Scholar
  22. Horn, B. 1986. Robot Vision. McGraw-Hill.Google Scholar
  23. Horn, B. and Weldon, E. 1988. Direct methods for recovering motion. International Journal of Computer Vision, 2(1):51–76.Google Scholar
  24. Hrdlicka, A. 1972. Practical anthropometry. AMS Press.Google Scholar
  25. Kaucic, A. and Blake, A. 1998. Accurate, real-time, unadorned lip tracking. In ICCV98, pp. 370–375.Google Scholar
  26. Kriegman, D.J., Triendl, E., and Binford, T.O. 1989. Stereo vision and navigation in buildings for mobile robots. RA, 5(6):792–803.Google Scholar
  27. Lanitis, A., Taylor, C.J., and Cootes, T.F. 1997. Automatic interpretation and coding of face images using flexible models. IEEE Pattern Analysis and Machine Intelligence, 19(7):743–756.Google Scholar
  28. Li, H., Roivainen, P., and Forchheimer, R. 1993. 3-D motion estimation in model-based facial image coding. IEEE Pattern Analysis and Machine Intelligence, 15(6):545–555.Google Scholar
  29. Lowe, D.G. 1991. Fitting parameterized three-dimensional models to images. IEEE Pattern Analysis and Machine Intelligence, 13(5):441–450.Google Scholar
  30. Maybeck, P. 1979. Stochastic Models, Estimation and Control, Volume 1. Academic Press.Google Scholar
  31. Maybeck, P. 1982. Stochastic Models, Estimation and Control, Volume 2. Academic Press.Google Scholar
  32. Metaxas, D. 1996. Physics-Based Deformable Models: Applications to Computer Vision, Graphics, and Medical Imaging. Kluwer Academic Publishers.Google Scholar
  33. Metaxas, D. and Terzopoulos, D. 1993. Shape and nonrigid motion estimation through physics-based synthesis. IEEE Pattern Analysis and Machine Intelligence, 15(6):580–591.Google Scholar
  34. Moses, Y., Reynard, D., and Blake, A. 1995. Robust real time tracking and classificiation of facial expressions. In Proceedings ICCV’ 95, pp. 296–301.Google Scholar
  35. Negahdaripour, S. and Horn, B. 1987. Direct passive navigation. IEEE Pattern Analysis and Machine Intelligence, 9(1):168–176.Google Scholar
  36. Netravali, A. and Salz, J. 1985. Algorithms for estimation of threedimensional motion. AT&T Technical Journal, 64:335–346.Google Scholar
  37. Pentland, A. and Horowitz, B. 1991. Recovery of nonrigid motion and structure. IEEE Pattern Analysis and Machine Intelligence, 13(7):730–742.Google Scholar
  38. Pentland, A. and Sclaroff, S. 1991. Closed-form solutions for physically based shape modeling and recognition. IEEE Pattern Analysis and Machine Intelligence, 13(7):715–729.Google Scholar
  39. Press, W., Teukolsky, S., Vetterling, W., and Flannery, B. 1992. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press.Google Scholar
  40. Reynard, D., Wildenberg, A., Blake, A., and Marchant, J. 1996. Learning dynamics of complex motions from image sequences. In Proceedings ECCV’ 96, pp. I:357–368.Google Scholar
  41. Shabana, A. 1989. Dynamics of Multibody Systems. Wiley.Google Scholar
  42. Sharma, R., Azoz, Y., and Devi, L. 1998. Reliable tracking of human arm dynamics by multiple cue integration and constraint fusion. In Proceedings CVPR’ 98.Google Scholar
  43. Shi, J. and Tomasi, C. 1994. Good features to track. In Proceedings CVPR’ 94, pp. 593–600.Google Scholar
  44. Simoncelli, E., Adelson, E., and Heeger, D. 1991. Probability distributions of optical flow. In Proceedings CVPR’ 91, pp. 310–315.Google Scholar
  45. Strang, G. 1988. Linear algebra and its applications. Harcourt, Brace, Jovanovich.Google Scholar
  46. Terzopoulos, D. 1993. Physically-based fusion of visual data over space, time and scale. In Multisensor Fusion for Computer Vision, J. Aggarwal (Ed.). Springer-Verlag, pp. 63–69.Google Scholar
  47. Terzopoulos, D. and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Pattern Analysis and Machine Intelligence, 15(6):569–579.Google Scholar
  48. Terzopoulos, D., Witkin, A., and Kass, M. 1988. Constraints on deformable models: Recovering 3D shape and nonrigid motion. Artificial Intelligence, 36(1):91–123.Google Scholar
  49. Yacoob, Y. and Davis, L.S. 1994. Computing spatio-temporal representations of human faces. In Proceedings CVPR’ 94, pp. 70–75.Google Scholar
  50. Yuille, A.L., Cohen, D.S., and Halliman, P. 1992. Feature extraction from faces using deformable templates. International Journal of Computer Vision, 8:104–109.Google Scholar
  51. Zhang, G.H. and Wallace, A. 1993. Physical modeling and combination of range and intensity edge data. Computer Vision, Graphics, and Image Processing, 58(2):191–220.Google Scholar

Copyright information

© Kluwer Academic Publishers 2000

Authors and Affiliations

  • Douglas DeCarlo
    • 1
  • Dimitris Metaxas
    • 2
  1. 1.Department of Computer Science, and Center for Cognitive ScienceRutgers UniversityPiscatawayUSA
  2. 2.Department of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphiaUSA

Personalised recommendations