Real-Time Non-rigid Shape Recovery Via Active Appearance Models for Augmented Reality

  • Jianke Zhu
  • Steven C. H. Hoi
  • Michael R. Lyu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3951)


One main challenge in Augmented Reality (AR) applications is to keep track of video objects with their movement, orientation, size, and position accurately. This poses a challenging task to recover non-rigid shape and global pose in real-time AR applications. This paper proposes a novel two-stage scheme for online non-rigid shape recovery toward AR applications using Active Appearance Models (AAMs). First, we construct 3D shape models from AAMs offline, which do not involve processing of the 3D scan data. Based on the computed 3D shape models, we propose an efficient online algorithm to estimate both 3D pose and non-rigid shape parameters via local bundle adjustment for building up point correspondences. Our approach, without manual intervention, can recover the 3D non-rigid shape effectively from either real-time video sequences or single image. The recovered 3D pose parameters can be used for AR registrations. Furthermore, the facial feature can be tracked simultaneously, which is critical for many face related applications. We evaluate our algorithms on several video sequences. Promising experimental results demonstrate our proposed scheme is effective and significant for real-time AR applications.


Augmented Reality Active Appearance Model Augmented Reality Application Promising Experimental Result IEEE CVPR 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kato, H., Billinghurst, M.: Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In: Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality (2004)Google Scholar
  2. 2.
    Vacchetti, L., Lepetit, V., Fua, P.: Stable real-time 3d tracking using online and offline information. IEEE Trans. PAMI 26 (2004)Google Scholar
  3. 3.
    Cootes, T., Edwards, G., Taylo, C.: Active appearance models. IEEE Trans. PAMI 23 (2001)Google Scholar
  4. 4.
    Cootes, T., Kittipanyangam, P.: Comparing variations on the active appearance model algorithm. In: British Machine Vision ConferenceGoogle Scholar
  5. 5.
    Stegmann, M., Ersboll, B., Larsen, R.: Fame-a flexible appearance modeling environment. IEEE Trans. Medical Imaging 22 (2003)Google Scholar
  6. 6.
    Mittrapiyanumic, P., DeSouza, G., Kak, A.: Calculating the 3d-pose of rigid-objects using active appearance models. In: Proceedings of the International Conference in Robotics and Automation, vol. 5, pp. 5147–5152 (2004)Google Scholar
  7. 7.
    Ahlberg, J.: Using the active appearance algorithm for face and facial feature tracking. In: Proceedings IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, pp. 68–72 (2001)Google Scholar
  8. 8.
    Xiao, J., Baker, S., Matthews, I., Kanade, T.: Real-time combined 2d+3d active appearance models. In: IEEE CVPR 2004, vol. 2, pp. 535–542 (2004)Google Scholar
  9. 9.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3d morphable model. IEEE Trans. PAMI 25 (2003)Google Scholar
  10. 10.
    Hou, X., Zhang, S.Z.L., Cheng, H., Direct, Q.: appearance models. In: IEEE CVPR 2001, vol. 1, pp. 828–833 (2001)Google Scholar
  11. 11.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3d shape from image streams. In: IEEE CVPR 2000, vol. 2, pp. 690–696 (2000)Google Scholar
  12. 12.
    Torresani, L., Yang, D., Alexander, E., Bregler, C.: Tracking and modeling non-rigid objects with rank constraints. In: IEEE CVPR 2001, vol. 1, pp. 493–500 (2001)Google Scholar
  13. 13.
    Medioni, G., Kang, S.B.: Emerging topics in computer vision. Prentice-Hall, Englewood Cliffs (2004)Google Scholar
  14. 14.
    Nielson, G.: Scattered data modeling. IEEE Computer Graphics and Applications 13 (1993)Google Scholar
  15. 15.
    Gross, R., Matthews, I., Baker, S.: Generic vs. person specific active appearance models. In: British Machine Vision Conference (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jianke Zhu
    • 1
  • Steven C. H. Hoi
    • 1
  • Michael R. Lyu
    • 1
  1. 1.Department of Computer Science & EngineeringChinese University of Hong KongShatin, Hong KongChina

Personalised recommendations