Skip to main content

The Isowarp: The Template-Based Visual Geometry of Isometric Surfaces

Abstract

Registration maps or warps form a key element in Shape-from-Template (SfT). They relate the template with the input image, which contains the projection of the deformed surface. Recently, it was shown that isometric SfT can be solved analytically if the warp and its first-order derivatives are known. In practice, the warp is recovered by interpolating a set of discrete template-to-image point correspondences. This process relies on smoothness priors but ignores the 3D geometry. This may produce errors in the warp and poor reconstructions. In contrast, we propose to create a 3D consistent warp, which technically is a very challenging task, as the 3D shape variables must be eliminated from the isometric SfT equations to find differential constraints for the warp only. Integrating these constraints in warp estimation yields the isowarp, a warp 3D consistent with isometric SfT. Experimental results show that incorporating the isowarp in the SfT pipeline allows the analytic solution to outperform non-convex 3D shape refinement methods and the recent DNN-based SfT methods. The isowarp can be properly initialized with convex methods and its hyperparameters can be automatically obtained with cross-validation. The isowarp is resistant to 3D ambiguities and less computationally expensive than existing 3D shape refinement methods. The isowarp is thus a theoretical and practical breakthrough in SfT.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

References

  1. Agisoft Photoscan 1.0.4 (2014). http://www.agisoft.ru/products/photoscan

  2. Agudo, A., Agapito, L., Calvo, B., & Montiel, J. M. (2014) Good vibrations: A modal analysis approach for sequential non-rigid structure from motion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1558–1565

  3. Agudo, A., Moreno-Noguer, F., Calvo, B., & Montiel, J. M. M. (2016). Sequential non-rigid structure from motion using physical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(5), 979–994.

    Article  Google Scholar 

  4. Bartoli, A., Gérard, Y., Chadebecq, F., Collins, T., & Pizarro, D. (2015). Shape-from-template. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10), 2099–2118.

    Article  Google Scholar 

  5. Bartoli, A., Perriollat, M., & Chambon, S. (2010). Generalized thin-plate spline warps. International Journal of Computer Vision, 88(1), 85–110.

    Article  Google Scholar 

  6. Bay, H., Tuytelaars, T., Van Gool, L. (2006). Surf: Speeded up robust features. In European conference on computer vision, pp. 404–417. Springer.

  7. Blender (1994). Online Community: Blender - a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam. http://www.blender.org

  8. Bookstein, F. L. (1989). Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Transactions on Pattern Analysis & Machine Intelligence, 6, 567–585.

    Article  Google Scholar 

  9. Brunet, F., Bartoli, A., & Hartley, R. I. (2014). Monocular template-based 3D surface reconstruction: Convex inextensible and nonconvex isometric methods. Computer Vision and Image Understanding, 125, 138–154.

    Article  Google Scholar 

  10. Brunet, F., Bartoli, A., Navab, N., Malgouyres, R. et al. (2009). NURBS warps. In British machine vision conference.

  11. Casillas-Perez, D., Pizarro, D., Fuentes-Jimenez, D., Mazo, M., & Bartoli, A. (2018). Equiareal shape-from-template. Journal of Mathematical Imaging and Vision,. https://doi.org/10.1007/s10851-018-0862-5.

    Article  MATH  Google Scholar 

  12. Chhatkuli, A., Pizarro, D., Bartoli, A., & Collins, T. (2017). A stable analytical framework for isometric shape-from-template by surface integration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(5), 833–850.

    Article  Google Scholar 

  13. Do Carmo, M. P. (2016). Differential geometry of curves and surfaces: Revised and updated second edition. Courier Dover Publications.

  14. Dubitzky, W., Granzow, M., & Berrar, D. P. (2007). Fundamentals of data mining in genomics and proteomics. Berlin: Springer.

  15. Fayad, J., Russell, C., & Agapito, L. (2011). Automated articulated structure and 3d shape recovery from point correspondences. In 2011 International conference on computer vision, pp. 431–438. IEEE.

  16. Fuentes-Jiménez, D., Casillas-Pérez, D., Pizarro-Pérez, D., Collins, T., & Bartoli, A. (2018). Deep shape-from-template: Wide-baseline, dense and fast registration and deformable reconstruction from a single image. CoRR abs/1811.07791. arXiv:1811.07791

  17. Golyanik, V., Shimada, S., Varanasi, K., & Stricker, D. (2018). Hdm-net: Monocular non-rigid 3d reconstruction with learned deformation model. In International conference on virtual reality and augmented reality, pp. 51–72. Berlin: Springer.

  18. Haouchine, N., Dequidt, J., Berger, M. O., & Cotin, S. (2014). Single view augmentation of 3D elastic objects. In ISMAR, pp. 229–236. IEEE.

  19. Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge: Cambridge University Press.

    MATH  Google Scholar 

  20. Horn, B. K., & Schunck, B. G. (1981). Determining optical flow. Artificial intelligence, 17(1–3), 185–203.

    Article  Google Scholar 

  21. Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94.

    Article  Google Scholar 

  22. Malti, A., Bartoli, A., & Hartley, R. (2015). A linear least-squares solution to elastic shape-from-template. In CVPR, pp. 1629–1637.

  23. Malti, A., & Herzet, C. (2017). Elastic shape-from-template with spatially sparse deforming forces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3337–3345.

  24. Ngo, D. T., Östlund, J., & Fua, P. (2016). Template-based monocular 3d shape recovery using Laplacian meshes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 172–187.

    Article  Google Scholar 

  25. Ovsienko, V., & Tabachnikov, S. (2009). What is the schwarzian derivative. Notices of the AMS, 56(1), 34–36.

    MathSciNet  MATH  Google Scholar 

  26. Özgür, E., & Bartoli, A. (2017). Particle-SfT: A provably-convergent, fast shape-from-template algorithm. International Journal of Computer Vision, 123(2), 184–205.

    Article  Google Scholar 

  27. Parashar, S., Pizarro, D., & Bartoli, A. (2020). Local deformable 3d reconstruction with cartan’s connections. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(12), 3011–3026. https://doi.org/10.1109/TPAMI.2019.2920821.

    Article  Google Scholar 

  28. Perriollat, M., Hartley, R., & Bartoli, A. (2011). Monocular template-based reconstruction of inextensible surfaces. International Journal of Computer Vision, 95(2), 124–137.

    MathSciNet  Article  Google Scholar 

  29. Pilet, J., Lepetit, V., & Fua, P. (2008). Fast non-rigid surface detection, registration and realistic augmentation. International Journal of Computer Vision, 76(2), 109–122.

    Article  Google Scholar 

  30. Pizarro, D., & Bartoli, A. (2012). Feature-based deformable surface detection with self-occlusion reasoning. International Journal of Computer Vision, 97(1), 54–70.

    Article  Google Scholar 

  31. Pizarro, D., Bartoli, A., & Collins, T. (2013). Isowarp and conwarp: Warps that exactly comply with weak-perspective projection of deforming objects. In BMVC.

  32. Pizarro, D., Khan, R., & Bartoli, A. (2016). Schwarps: Locally projective image warps based on 2d schwarzian derivatives. International Journal of Computer Vision, 119(2), 93–109. https://doi.org/10.1007/s11263-016-0882-9.

    MathSciNet  Article  MATH  Google Scholar 

  33. Pumarola, A., Agudo, A., Porzi, L., Sanfeliu, A., Lepetit, V., & Moreno-Noguer, F. (2018). Geometry-aware network for non-rigid shape prediction from a single view. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681–4690.

  34. Rueckert, D., Sonoda, L. I., Hayes, C., Hill, D. L. G., Leach, M. O., & Hawkes, D. J. (1999). Nonrigid registration using free-form deformations: Application to breast MR images. IEEE Transactions on Medical Imaging, 18(8), 712–721. https://doi.org/10.1109/42.796284.

    Article  Google Scholar 

  35. Salzmann, M., & Fua, P. (2011). Linear local models for monocular reconstruction of deformable surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5), 931–944.

    Article  Google Scholar 

  36. Shimada, S., Golyanik, V., Theobalt, C., & Stricker, D. (2019). Ismo-gan: Adversarial learning for monocular non-rigid 3d reconstruction. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops.

  37. Sundaram, N., Brox, T., & Keutzer, K. (2010). Dense point trajectories by GPU-accelerated large displacement optical flow. In European conference on computer vision, pp. 438–451. Berlin: Springer.

  38. Varol, A., Salzmann, M., Fua, P., & Urtasun, R. (2012). A constrained latent variable model. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2248–2255. IEEE.

  39. Yu, R., Russell, C., Campbell, N. D., & Agapito, L. (2015). Direct, dense, and deformable: Template-based non-rigid 3D reconstruction from RGB video. In |it Proceedings of the IEEE international conference on computer vision, pp. 918–926.

  40. Zach, C., Pock, T., & Bischof, H. (2007). A duality based approach for realtime tv-l1 optical flow. In F. A. Hamprecht, C. Schnörr, & B. Jähne (Eds.), Pattern recognition (pp. 214–223). Berlin,: Springer.

    Chapter  Google Scholar 

Download references

Acknowledgements

This research has received funding from the Spanish Ministry of Education and Culture under the scholarship FPU, the Spanish Ministry of Economy, Industry and Competitiveness under the project ARTEMISA (TIN2016-80939-R) and the EU’s FP7 through the ERC research grant 307483 FLEXABLE.

Author information

Affiliations

Authors

Corresponding author

Correspondence to David Casillas-Perez.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by Joachim Weickert.

Appendices

Derivation of the Image Embedding

We assume that the image plane is at \(z=1\) in camera coordinates, which is achieved by working in retinal coordinates. The perspective projection of a point (xyz) is then given by:

$$\begin{aligned} \Pi _{p}:\qquad {\mathbb {R}}^{3}\setminus \left\{ (x,y,z)\in {\mathbb {R}}^3\;|\;z=0\right\}&\rightarrow {\mathbb {R}}^2\nonumber \\ \left( x,y,z\right) ^\top&\mapsto \left( \frac{x}{z},\frac{y}{z}\right) ^\top . \end{aligned}$$
(25)

The inverse of the restriction \(\Pi _{p}|_{{\mathcal {S}}}:{\mathcal {S}} \rightarrow {\mathbb {R}}^2\) of \(\Pi _{p}\) is the image embedding. It forms a depth based parametrization of the surface \(({\mathcal {I}},X_{i})\) expressed in terms of the depth function \(\rho :{\mathbb {R}}^2 \rightarrow {\mathbb {R}}\):

$$\begin{aligned} X_{i}(u',v') = \rho (u',v') \left( u',v',1\right) ^\top , \end{aligned}$$
(26)

where \(u'\) and \(v'\) represent the image coordinates. Alternatively to \(\rho \), we define the Euclidean distance between the camera’s projection origin and the surface point as \({\tilde{\rho }}:{\mathbb {R}}^2 \rightarrow {\mathbb {R}}\):

$$\begin{aligned} {\tilde{\rho }}(u',v')=\rho (u',v')\zeta (u',v'), \end{aligned}$$
(27)

where \(\zeta (u',v')=\sqrt{1+u'^2+v'^2}\). The perspective parametrization \(({\mathcal {I}},X_i)\) can be then expressed in terms of \({\tilde{\rho }}\) as:

$$\begin{aligned} X_{i}(u',v') = \frac{{\tilde{\rho }}(u',v')}{\zeta (u',v')}\left( u',v',1\right) ^\top . \end{aligned}$$
(28)

Now, we can define the surface \({\mathcal {S}}\) from the template parametrization domain \({\mathcal {U}}\) by composing the previous parametrization \(X_i\) and the warp function \(\eta \) as follows:

$$\begin{aligned} {\bar{X}}_i(u,v) = X_i \circ \eta = \frac{{\tilde{\rho }}(\eta (u,v))}{\zeta (\eta (u,v))}\left( \eta (u,v),1\right) ^\top , \end{aligned}$$
(29)

where u and v are template domain coordinates.

Defining the depth function \({\bar{\rho }}:{\mathbb {R}}^2 \rightarrow {\mathbb {R}}\) as the depth function \({\tilde{\rho }}\) in u, v coordinates by the composition \({\bar{\rho }} = {\tilde{\rho }}\circ \eta \), we obtain:

$$\begin{aligned} {\bar{X}}_i(u,v) = X_i \circ \eta = \frac{{\bar{\rho }}(u,v)}{\zeta (\eta (u,v))}\left( \eta (u,v),1\right) ^\top . \end{aligned}$$
(30)

Working with the parametrization \(({\mathcal {U}},{\bar{X}}_i)\) of \({\mathcal {S}}\) has two principal advantages. First, it allows us to compute the first fundamental form, also known as the metric tensor, over the same parametrization domain as the template, which is essential to obtain the isowarp equations. Second, it greatly simplifies these equations.

Derivation of the Isowarp Equations

We give Matlab code to establish the Isowarp equations (11). These equations are too lengthy to be reproduced fully expanded. However we recall that, importantly, they depend on the known template and the unknown warp \(\eta \) only. More specifically, they are quadratic of the second-order in \(\eta \).

figuree

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Casillas-Perez, D., Pizarro, D., Fuentes-Jimenez, D. et al. The Isowarp: The Template-Based Visual Geometry of Isometric Surfaces. Int J Comput Vis 129, 2194–2222 (2021). https://doi.org/10.1007/s11263-021-01472-w

Download citation

Keywords

  • Image warp
  • SfT
  • Template-based
  • Non-rigid reconstruction