3D Structure Refinement of Nonrigid Surfaces through Efficient Image Alignment

  • Yinqiang Zheng
  • Shigeki Sugimoto
  • Masatoshi Okutomi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6495)


Given a template image with known 3D structure, we show how to refine the rough reconstruction of nonrigid surfaces from existing feature-based methods through efficient direct image alignment. Under the mild assumption that the barycentric coordinates of each 3D point on the surface keep constant, we prove that the template and the input image are correlated by piecewise homography, based on which a direct Lucas-Kanade image alignment method is proposed to iteratively recover an inextensible surface even with poor texture and sharp creases. To accelerate the direct Lucas-Kanade method, an equivalent but much more efficient method is proposed as well, in which the most time-consuming part of the Hessian can be pre-computed as a result of combining additive and inverse compositional expressions. Sufficient experiments on both synthetic and real images demonstrate the accuracy and efficiency of our proposed methods.


Input Image Fusion Method Template Image Image Alignment Residual Image 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Cohen, L., Cohen, I.: Finite element methods for active contour models and balloons for 2d and 3d images. PAMI 15, 1131–1147 (1993)CrossRefGoogle Scholar
  2. 2.
    Bhat, K.S., Twigg, C.D., Hodgins, J.K., Khosla, P.K., Popovic, Z., Seitz, S.M.: Estimating cloth simulation parameters from video. In: ACM Symposium on Computer Animation (2003)Google Scholar
  3. 3.
    Salzmann, M., Hartley, R., Fua, P.: Convex optimization for deformable surface 3d tracking. In: ICCV (2007)Google Scholar
  4. 4.
    Zhu, J., Hoi, S.C.H., Xu, Z., Lyu, M.R.: An effective approach to 3d deformable surface tracking. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 766–779. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Varol, A., Salzmann, M., Tola, E., Fua, P.: Template-free monocular reconstruction of deformable surfaces. In: ICCV (2009)Google Scholar
  6. 6.
    Salzmann, M., Moreno-Noguer, F., Lepetit, V., Fua, P.: Closed-form solution to non-rigid 3d surface registration. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 581–594. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  7. 7.
    Moreno-Noguer, F., Salzmann, M., Lepetit, V., Fua, P.: Capturing 3d stretchable surfaces from single images in closed form. In: CVPR (2009)Google Scholar
  8. 8.
    Salzmann, M., Fua, P.: Reconstructing sharply folding surfaces: a convex formulation. In: CVPR (2009)Google Scholar
  9. 9.
    Ilic, S., Salzmann, M., Fua, P.: Implicit meshes for effective silhouette handling. IJCV 72, 159–178 (2007)CrossRefGoogle Scholar
  10. 10.
    Terzopoulos, D., Metaxas, D.: Dynamic 3d models with local and global deformations: deformable superquadrics. PAMI 13, 703–714 (1991)CrossRefGoogle Scholar
  11. 11.
    Vidal, R., Hartley, R.: Perspective nonrigid shape and motion recovery. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 276–289. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  12. 12.
    Xiao, J., Kanade, T.: Uncalibrated perspective reconstruction of deformable structures. In: ICCV (2005)Google Scholar
  13. 13.
    Ecker, A., Jepson, A., Kutulakos, K.: Semidefinite programming heuristics for surface reconstruction ambiguities. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 127–140. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    Zhu, J., Lyu, M.R., Huang, T.S.: A fast 2d shape recovery approach by fusing features and appearance. PAMI 31, 1210–1224 (2009)CrossRefGoogle Scholar
  15. 15.
    Pilet, J., Lepetit, V., Fua, P.: Fast non-rigid surface detection, registration and realistic augmentation. IJCV 76, 109–122 (2008)CrossRefGoogle Scholar
  16. 16.
    Munoz, E., Buenaposada, J., Baumela, L.: Efficient model-based 3d tracking of deformable objects. In: ICCV (2005)Google Scholar
  17. 17.
    Baker, S., Matthews, I.: Lucas-kanade 20 years on: A unifying framework. IJCV 56, 221–255 (2004)CrossRefGoogle Scholar
  18. 18.
    Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: IJCAI (1981)Google Scholar
  19. 19.
    Salzmann, M., Lepetit, V., Fua, P.: Deformable surface tracking ambiguities. In: CVPR (2007)Google Scholar
  20. 20.
    Sugimoto, S., Okutomi, M.: A direct and efficient method for piecewise-planar surface reconstruction. In: CVPR (2007)Google Scholar
  21. 21.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV 20, 91–110 (2004)CrossRefGoogle Scholar
  22. 22.
    Bartoli, A.: Groupwise geometric and photometric direct image registration. PAMI 30, 2098–2108 (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Yinqiang Zheng
    • 1
  • Shigeki Sugimoto
    • 1
  • Masatoshi Okutomi
    • 1
  1. 1.Department of Mechanical and Control EngineeringTokyo Institute of TechnologyJapan

Personalised recommendations