Skip to main content
SpringerLink
Log in

Fast shape-from-template using local features

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Reconstructing the 3D surface of an object using only a single image is a challenging task, which has recently attracted attention. In this paper, a template-based approach is presented to reconstruct the surface of an isometric deformable object. The proposed approach brings a solution for a class of computer vision problems named shape-from-template (SfT). In SfT, the goal is to solve single-image reconstruction for an object given its 3D template model in some rest shape. To this end, corresponding keypoints between the template and the so-called deformed image are first established. Then, a very fast method is used to estimate the first-order differential flow around the extracted keypoint pairs as an affine transformation. This is done using the keypoint pairs’ surrounding texture patch. In our method, we estimate this affine transformation using the keypoint pairs’ closest neighbors. This is both faster and more stable. Finally, the depth of each keypoint in the deformed image is estimated from its associated affine transformation. The robustness of keypoint matching is essential to the process. Indeed, outliers defeat depth estimation dramatically. We propose two new approaches to detect and remove the possible outliers based on geometrical properties of the matched keypoints. These two geometrical outlier removal approaches are faster than existing ones and can be used with almost any image descriptor. Experimental results show that the proposed approaches are very effective and outperform existing ones.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, London (2011)

    Book  MATH  Google Scholar 

  2. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  3. Faugeras, O., Luong, Q.-T., Papadopoulo, T.: The Geometry of Multiple Images. MIT Press, Cambridge (2001)

    Google Scholar 

  4. Li, Y., Wang, Y., Case, M., Chang, S.F., Allen, P.K.: Real-time pose estimation of deformable objects using a volumetric approach. In: International Conference on Intelligent Robots and Systems (2014)

  5. Frank, B., Stachniss, C., Schmedding, R., Teschner, M., Burgard, W.: Learning object deformation models for robot motion planning. Robot. Auton. Syst. 62(8), 1153–1174 (2014)

    Article  Google Scholar 

  6. Malti, A., Bartoli, A., Collins, T.: Template-based conformal shape-from-motion from registered laparoscopic images. In: Medical Image Understanding and Analysis Conference (2011)

  7. Hu, M., Penney, P., Rueckert, D., Edwards, P., Bello, F., Casula, R., Figl, M., Hawkes, D.: Non-rigid reconstruction of the beating heart surface for minimally invasive cardiac surgery. Med. Image Comput. Comput. Assist. Interv. 5761, 34–42 (2009)

    Google Scholar 

  8. Collins, T., Compte, B., Bartoli, A.: Deformable shape-from-motion in laparoscopy using a rigid sliding window. In: Medical Image Understanding and Analysis Conference (2011)

  9. Moreno-noguer, F., Salzmann, F., Lepetit, V., Fua, P.: Capturing 3D stretchable surfaces from single images in closed form. In: Conference on Computer Vision and Pattern Recognition, pp. 1842–1849 (2009)

  10. Salzmann, M., Hartley, R., Fua, P.: Convex optimization for deformable surface 3-D tracking. In: International Conference on Computer Vision, pp. 1–8 (2007)

  11. Bartoli, A., Gerard, Y., Chadebecq, F., Collins, T., Pizarro, D.: Shape-from-template. IEEE Trans. Pattern Anal. Mach. Intell. 37(10), 2099–2118 (2015)

    Article  Google Scholar 

  12. Ecker, A., Jepson, A., Kutulakos, K.: Semidefinite programming heuristics for surface reconstruction ambiguities. In: European Conference on Computer Vision, vol. 5302, pp. 127–140 (2008)

  13. Shen, S., Shi, W., Liu, Y.: Monocular 3D tracking of inextensible deformable surfaces under L2-norm. In: Asian Conference on Computer Vision, vol. 19, no. 2, pp. 512–521 (2009)

  14. Brunet, F., Hartley, R., Bartoli, A., Navab, N., Malgouyres, R.: Monocular template-based reconstruction of smooth and inextensible surfaces. In: Asian Conference on Computer Vision, vol. 6494, pp. 52–66 (2010)

  15. Gumerov, N., Zandifar, A., Duraiswami, R., Davis, L.: Structure of applicable surfaces from single views. In: European Conference on Computer Vision, vol. 3023, pp. 482–496 (2004)

  16. Liang, J., Dementhon, D., Doermann, D.: Flattening curved documents in images. Comput. Vis. Pattern Recognit. 2, 338–345 (2005)

    Google Scholar 

  17. Perriollat, M., Bartoli, A.: A computational model of bounded developable surfaces with application to image-based three-dimensional reconstruction. Comput. Anim. Virtual Worlds 24(5), 459–476 (2013)

    Article  Google Scholar 

  18. Bachelder, I.A., Ullman, S.: Contour matching using local affine transformations. In: Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Champaign, IL, pp. 798–801 (1992)

  19. Salzmann, M., Fua, P.: Deformable Surface 3D Reconstruction from Monocular Images. Morgan & Claypool Publishers, San Rafael (2010)

    Google Scholar 

  20. Salzmann, M.: Toyota Technol. Inst. at Chicago, Chicago, IL, USA, Fua, P.: Linear local models for monocular reconstruction of deformable surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 931–944 (2011)

    Article  Google Scholar 

  21. Ostlund, J., Varol, A., Ngo, D., Fua, P.: Laplacian meshes for monocular 3D shape recovery. In: European Conference on Computer Vision, pp. 412–425 (2012)

  22. Ngo, D. T., Östlund, J., Fua, P.: Template-based monocular 3D shape recovery using Laplacian Meshes. In: IEEE transactions on pattern analysis and machine intelligence, vol. 38, pp. 172–187 (2016)

  23. Locher, A., Elsen, L., Boix, X., Gool, L.V.: Reconstruction of inextensible surfaces on a budget via bootstrapping. In: International Conference on 3D Vision, vol. 1, pp. 240–247 (2014)

  24. Varol, A., Shaji, A., Salzmann, M., Fua, P.: Monocular 3D reconstruction of locally textured surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 34(6), 1118–1130 (2012)

    Article  Google Scholar 

  25. Bartoli, A., Gerard, Y., Chadebecq, F., Collins, T.: On template-based reconstruction from a single view: analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces. In: Computer Vision and Pattern Recognition, pp. 2026–2033 (2012)

  26. Collins, T., Bartoli, A.: Using isometry to classify correct/incorrect 3D–2D correspondences. In: European Conference on Computer Vision, vol. 8692, pp. 325–340 (2014)

  27. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V., Leuven, B.K.U.: Speeded-up robust features (SURF). J. Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Article  Google Scholar 

  28. Ozuysal, M., Calonder, M., Lepetit, V., Fua, P.: Fast keypoint recognition using random ferns. IEEE Trans. Pattern Anal. Mach. Intell. 32(3), 448–461 (2010)

    Article  Google Scholar 

  29. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: International Conference on Computer Vision, pp. 2564–2571 (2011)

  30. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)

    Article  Google Scholar 

  31. Yi, K.M., Trulls, E., Lepetit, V., Fua, P.: LIFT: learned invariant feature transform. In: European Conference on Computer Vision, vol. 9910, pp. 467–483 (2016)

  32. Agamennoni, G.: Bayesian clustering with outliers and missing values. Technical Report ACFR-TR-2013-001. University of Sydney, NSW Australia, Australian Centre for Field Robotics (2013)

  33. Tran, Q.H., Chin, T.J., Carneiro, G., Brown, M.S., Suter, D.: In defence of RANSAC for outlier rejection in deformable registration. In: European Conference on Computer Vision vol. 7575, pp. 274–287 (2012)

  34. Pizarro, D., Bartoli, A.: Feature-based deformable surface detection with self-occlusion reasoning. Int. J. Comput. Vision 97(1), 54–70 (2012)

    Article  MATH  Google Scholar 

  35. Collins, T., Bartoli, A.: Realtime shape-from-template: system and applications. In: IEEE International Symposium on Mixed and Augmented Reality, pp. 116-119 (2015)

  36. Bartoli, A., Perriollat, M., Chambon, S.: Generalized thin-plate spline warps. Int. J. Comput. Vis. 88(1), 85–110 (2010)

    Article  Google Scholar 

Download references

Acknowledgements

We want to express our gratitude to the authors of [22] for the Kinect_paper dataset and implementation of their reconstruction method. Also, we would thank Giuseppe Marchioro for his help during this work. This research has received funding from the Ministry of Science, Research and Technology of Islamic Republic of Iran and Image Science for Interventional Techniques laboratory of the Auvergne University of France. This research has received funding from the EUs FP7 through the ERC research Grant 307483 FLEXABLE.

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, Shiraz University, Shiraz, Iran

    Mahmoud Famouri & Zohreh Azimifar

  2. Institut Pascal, University Clermont Auvergne / CNRS / SIGMA, Clermont-Ferrand, France

    Adrien Bartoli

Authors
  1. Mahmoud Famouri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrien Bartoli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zohreh Azimifar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zohreh Azimifar.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Famouri, M., Bartoli, A. & Azimifar, Z. Fast shape-from-template using local features. Machine Vision and Applications 29, 73–93 (2018). https://doi.org/10.1007/s00138-017-0876-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-017-0876-9

Keywords

Search

Navigation