Advertisement

Cyclic Functional Mapping: Self-supervised Correspondence Between Non-isometric Deformable Shapes

Conference paper
  • 720 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12350)

Abstract

We present the first spatial-spectral joint consistency network for self-supervised dense correspondence mapping between non-isometric shapes. The task of alignment in non-Euclidean domains is one of the most fundamental and crucial problems in computer vision. As 3D scanners can generate highly complex and dense models, the mission of finding dense mappings between those models is vital. The novelty of our solution is based on a cyclic mapping between metric spaces, where the distance between a pair of points should remain invariant after the full cycle. As the same learnable rules that generate the point-wise descriptors apply in both directions, the network learns invariant structures without any labels while coping with non-isometric deformations. We show here state-of-the-art-results by a large margin for a variety of tasks compared to known self-supervised and supervised methods .

Keywords

Dense shape correspondence Self-supervision One-shot learning Spectral decomposition 3D alignment 

Notes

Acknowledgment

D.R. is partially funded by the Zimin Institute for Engineering Solutions Advancing BetterLives, the Israeli consortiums for soft robotics and autonomous driving, and the Shlomo Shmeltzer Institute for Smart Transportation.

References

  1. 1.
    Abadi, M., et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems (2015). Software available from http://tensorflow.org/
  2. 2.
    Aflalo, Y., Bronstein, A., Kimmel, R.: On convex relaxation of graph isomorphism. Proc. Natl. Acad. Sci. 112(10), 2942–2947 (2015)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Aflalo, Y., Dubrovina, A., Kimmel, R.: Spectral generalized multi-dimensional scaling. Int. J. Comput. Vision 118(3), 380–392 (2016)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Aflalo, Y., Kimmel, R., Raviv, D.: Scale invariant geometry for nonrigid shapes. SIAM J. Imaging Sci. 6(3), 1579–1597 (2013)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Aubry, M., Schlickewei, U., Cremers, D.: The wave kernel signature: a quantum mechanical approach to shape analysis. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1626–1633. IEEE (2011)Google Scholar
  6. 6.
    Ben-Chen, M., Gotsman, C., Bunin, G.: Conformal flattening by curvature prescription and metric scaling. Comput. Graph. Forum 27, 449–458 (2008). Wiley Online LibraryCrossRefGoogle Scholar
  7. 7.
    Bogo, F., Romero, J., Loper, M., Black, M.J.: FAUST: dataset and evaluation for 3D mesh registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, June 2014Google Scholar
  8. 8.
    Boscaini, D., Masci, J., Rodolà, E., Bronstein, M.M., Cremers, D.: Anisotropic diffusion descriptors. Comput. Graph. Forum 35, 431–441 (2016)CrossRefGoogle Scholar
  9. 9.
    Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proc. Natl. Acad. Sci. 103(5), 1168–1172 (2006)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Numerical Geometry of Non-Rigid Shapes. Springer, New York (2008)Google Scholar
  11. 11.
    Bronstein, M.M., Bronstein, A.M., Kimmel, R., Yavneh, I.: Multigrid multidimensional scaling. Numer. Linear Algebra Appl. 13(2–3), 149–171 (2006)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Bronstein, M.M., Kokkinos, I.: Scale-invariant heat kernel signatures for non-rigid shape recognition. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1704–1711. IEEE (2010)Google Scholar
  13. 13.
    Chen, Q., Koltun, V.: Robust nonrigid registration by convex optimization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2039–2047 (2015)Google Scholar
  14. 14.
    Cosmo, L., Rodolà, E., Bronstein, M.M., Torsello, A., Cremers, D., Sahillioglu, Y.: SHREC’16: Partial matching of deformable shapesGoogle Scholar
  15. 15.
    Elad, A., Kimmel, R.: On bending invariant signatures for surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 25(10), 1285–1295 (2003)CrossRefGoogle Scholar
  16. 16.
    Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 209–216. ACM Press/Addison-Wesley Publishing Co. (1997)Google Scholar
  17. 17.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: 3D-CODED : 3D Correspondences by Deep Deformation. CoRR abs/1806.05228 (2018), http://arxiv.org/abs/1806.05228
  18. 18.
    Halimi, O., Litany, O., Rodola, E., Bronstein, A.M., Kimmel, R.: Unsupervised learning of dense shape correspondence. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4370–4379 (2019)Google Scholar
  19. 19.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385
  20. 20.
    Huang, Q.X., Guibas, L.: Consistent shape maps via semidefinite programming. Comput. Graph. Forum 32, 177–186 (2013). Wiley Online LibraryCrossRefGoogle Scholar
  21. 21.
    Huang, Q., Wang, F., Guibas, L.: Functional map networks for analyzing and exploring large shape collections. ACM Trans. Graph. (TOG) 33(4), 1–11 (2014)zbMATHGoogle Scholar
  22. 22.
    Kim, V.G., Lipman, Y., Funkhouser, T.: Blended intrinsic maps. ACM Trans. Graphics (TOG) 30, 79 (2011). ACMGoogle Scholar
  23. 23.
    Li, C.L., Simon, T., Saragih, J., Póczos, B., Sheikh, Y.: LBS Autoencoder: self-supervised fitting of articulated meshes to point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11967–11976 (2019)Google Scholar
  24. 24.
    Lipman, Y., Daubechies, I.: Conformal Wasserstein distances: comparing surfaces in polynomial time. Adv. Math. 227(3), 1047–1077 (2011)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Litany, O., Remez, T., Rodolà, E., Bronstein, A.M., Bronstein, M.M.: Deep functional maps: structured prediction for dense shape correspondence. CoRR abs/1704.08686 (2017). http://arxiv.org/abs/1704.08686
  26. 26.
    Litany, O., Rodolà, E., Bronstein, A.M., Bronstein, M.M.: Fully spectral partial shape matching. Comput. Graph. Forum 36, 247–258 (2017). Wiley Online LibraryCrossRefGoogle Scholar
  27. 27.
    Litman, R., Bronstein, A.M.: Learning spectral descriptors for deformable shape correspondence. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 171–180 (2013)CrossRefGoogle Scholar
  28. 28.
    Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances in Neural Information Processing Systems, pp. 700–708 (2017)Google Scholar
  29. 29.
    Masci, J., Boscaini, D., Bronstein, M., Vandergheynst, P.: Geodesic convolutional neural networks on Riemannian manifolds. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 37–45 (2015)Google Scholar
  30. 30.
    Ovsjanikov, M., Ben-Chen, M., Solomon, J., Butscher, A., Guibas, L.: Functional maps: a flexible representation of maps between shapes. ACM Trans. Graph. (TOG) 31(4), 30 (2012)CrossRefGoogle Scholar
  31. 31.
    Pottmann, H., Wallner, J., Huang, Q.X., Yang, Y.L.: Integral invariants for robust geometry processing. Comput. Aided Geometr. Des. 26(1), 37–60 (2009)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Raviv, D., Bronstein, A.M., Bronstein, M.M., Waisman, D., Sochen, N., Kimmel, R.: Equi-affine invariant geometry for shape analysis. J. Math. Imaging Vis. 50(1–2), 144–163 (2014)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Raviv, D., Bronstein, M.M., Bronstein, A.M., Kimmel, R., Sochen, N.: Affine-invariant diffusion geometry for the analysis of deformable 3D shapes. In: CVPR 2011, pp. 2361–2367. IEEE (2011)Google Scholar
  34. 34.
    Raviv, D., Dubrovina, A., Kimmel, R.: Hierarchical matching of non-rigid shapes. In: Bruckstein, A.M., ter Haar Romeny, B.M., Bronstein, A.M., Bronstein, M.M. (eds.) SSVM 2011. LNCS, vol. 6667, pp. 604–615. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-24785-9_51CrossRefGoogle Scholar
  35. 35.
    Raviv, D., Dubrovina, A., Kimmel, R.: Hierarchical framework for shape correspondence. Numer. Math. Theory Methods Appl. 6(1), 245–261 (2013)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Raviv, D., Kimmel, R.: Affine invariant geometry for non-rigid shapes. Int. J. Comput. Vision 111(1), 1–11 (2015)MathSciNetCrossRefGoogle Scholar
  37. 37.
    Raviv, D., Raskar, R.: Scale invariant metrics of volumetric datasets. SIAM J. Imaging Sci. 8(1), 403–425 (2015)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Rodolà, E., Cosmo, L., Bronstein, M.M., Torsello, A., Cremers, D.: Partial functional correspondence. Comput. Graph. Forum 36, 222–236 (2017). Wiley Online LibraryCrossRefGoogle Scholar
  39. 39.
    Rodolà, E., Rota Bulo, S., Windheuser, T., Vestner, M., Cremers, D.: Dense non-rigid shape correspondence using random forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4177–4184 (2014)Google Scholar
  40. 40.
    Roufosse, J.M., Sharma, A., Ovsjanikov, M.: Unsupervised deep learning for structured shape matching. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1617–1627 (2019)Google Scholar
  41. 41.
    Rustamov, R.M.: Laplace-Beltrami eigenfunctions for deformation invariant shape representation. In: Proceedings of the Fifth Eurographics Symposium on Geometry Processing, SGP 2007, pp. 225–233. Eurographics Association, Aire-la-Ville (2007). http://dl.acm.org/citation.cfm?id=1281991.1282022
  42. 42.
    Rustamov, R.M., Ovsjanikov, M., Azencot, O., Ben-Chen, M., Chazal, F., Guibas, L.: Map-based exploration of intrinsic shape differences and variability. ACM Trans. Graph. (TOG) 32(4), 1–12 (2013)CrossRefGoogle Scholar
  43. 43.
    Sethian, J.A.: A fast marching level set method for monotonically advancing fronts. Proc. Natl. Acad. Sci. 93(4), 1591–1595 (1996)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Starck, J., Hilton, A.: Spherical matching for temporal correspondence of non-rigid surfaces. In: Tenth IEEE International Conference on Computer Vision (ICCV 2005), Volume 1, vol. 2, pp. 1387–1394. IEEE (2005)Google Scholar
  45. 45.
    Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multi-scale signature based on heat diffusion. Comput. Graph. Forum 28, 1383–1392 (2009). Wiley Online LibraryCrossRefGoogle Scholar
  46. 46.
    Szeliski, R., et al.: SCAPE: shape completion and animation of people, vol. 24 (2005)Google Scholar
  47. 47.
    Tevs, A., Berner, A., Wand, M., Ihrke, I., Seidel, H.P.: Intrinsic shape matching by planned landmark sampling. Comput. Graph. Forum 30, 543–552 (2011). Wiley Online LibraryCrossRefGoogle Scholar
  48. 48.
    Tombari, F., Salti, S., Di Stefano, L.: Unique signatures of histograms for local surface description. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 356–369. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15558-1_26CrossRefGoogle Scholar
  49. 49.
    Vestner, M., et al.: Efficient deformable shape correspondence via kernel matching. In: 2017 International Conference on 3D Vision (3DV), pp. 517–526. IEEE (2017)Google Scholar
  50. 50.
    Vestner, M., Litman, R., Rodolà, E., Bronstein, A., Cremers, D.: Product manifold filter: non-rigid shape correspondence via kernel density estimation in the product space. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3327–3336 (2017)Google Scholar
  51. 51.
    Yi, Z., Zhang, H., Tan, P., Gong, M.: Dualgan: unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2849–2857 (2017)Google Scholar
  52. 52.
    Zaharescu, A., Boyer, E., Varanasi, K., Horaud, R.: Surface feature detection and description with applications to mesh matching. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 373–380. IEEE (2009)Google Scholar
  53. 53.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Tel Aviv UniversityTel AvivIsrael

Personalised recommendations