Intrinsic Point Cloud Interpolation via Dual Latent Space Navigation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12347)


We present a learning-based method for interpolating and manipulating 3D shapes represented as point clouds, that is explicitly designed to preserve intrinsic shape properties. Our approach is based on constructing a dual encoding space that enables shape synthesis and, at the same time, provides links to the intrinsic shape information, which is typically not available on point cloud data. Our method works in a single pass and avoids expensive optimization, employed by existing techniques. Furthermore, the strong regularization provided by our dual latent space approach also helps to improve shape recovery in challenging settings from noisy point clouds across different datasets. Extensive experiments show that our method results in more realistic and smoother interpolations compared to baselines. Both the code and our pre-trained network can be found online:


3D point clouds 3D reconstruction Deep learning Applications Methodology Theory 



Parts of this work were supported by the KAUST CRG-2017-3426 Award and the ERC Starting Grant No. 758800 (EXPROTEA).

Supplementary material

504434_1_En_39_MOESM1_ESM.pdf (33.5 mb)
Supplementary material 1 (pdf 34351 KB)
504434_1_En_39_MOESM2_ESM.mkv (36 mb)
Supplementary material 2 (mkv 36865 KB)


  1. 1.
    Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and generative models for 3D point clouds. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 40–49. Stockholmsmässan, Stockholm Sweden, 10–15 July 2018Google Scholar
  2. 2.
    Alexa, M., Cohen-Or, D., Levin, D.: As-rigid-as-possible shape interpolation. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 157–164. ACM Press/Addison-Wesley Publishing Co. (2000)Google Scholar
  3. 3.
    Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.: Scape: shape completion and animation of people. In: ACM Transactions on Graphics (TOG), vol. 24, pp. 408–416. ACM (2005)Google Scholar
  4. 4.
    Arun, K.S., Huang, T.S., Blostein, S.D.: Least-squares fitting of two 3-D point sets. IEEE Trans. Pattern Anal. Mach. Intell. 1(5), 698–700 (1987)CrossRefGoogle Scholar
  5. 5.
    Ben-Hamu, H., Maron, H., Kezurer, I., Avineri, G., Lipman, Y.: Multi-chart generative surface modeling. In: SIGGRAPH Asia 2018 Technical Papers, p. 215. ACM (2018)Google Scholar
  6. 6.
    Benamou, J.D., Brenier, Y.: A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem. Numer. Math. 84(3), 375–393 (2000)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Bogo, F., Romero, J., Loper, M., Black, M.J.: FAUST: dataset and evaluation for 3D mesh registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3794–3801 (2014)Google Scholar
  8. 8.
    Bogo, F., Romero, J., Pons-Moll, G., Black, M.J.: Dynamic FAUST: registering human bodies in motion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6233–6242, July 2017Google Scholar
  9. 9.
    Bonneel, N., Rabin, J., Peyré, G., Pfister, H.: Sliced and radon Wasserstein Barycenters of measures. J. Math. Imaging Vis. 51(1), 22–45 (2015)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Boscaini, D., Eynard, D., Kourounis, D., Bronstein, M.M.: Shape-from-operator: recovering shapes from intrinsic operators. In: Computer Graphics Forum, vol. 34, pp. 265–274. Wiley Online Library (2015)Google Scholar
  11. 11.
    Carmo, M.P.D.: Riemannian geometry. Birkhäuser (1992)Google Scholar
  12. 12.
    Chen, N., Klushyn, A., Kurle, R., Jiang, X., Bayer, J., van der Smagt, P.: Metrics for deep generative models. arXiv preprint arXiv:1711.01204 (2017)
  13. 13.
    Chern, A., Knöppel, F., Pinkall, U., Schröder, P.: Shape from metric. ACM Trans. Graph. (TOG) 37(4), 63 (2018)CrossRefGoogle Scholar
  14. 14.
    Corman, E., Solomon, J., Ben-Chen, M., Guibas, L., Ovsjanikov, M.: Functional characterization of intrinsic and extrinsic geometry. ACM Trans. Graph. (TOG) 36(2), 1–17 (2017)CrossRefGoogle Scholar
  15. 15.
    Crane, K., Pinkall, U., Schröder, P.: Spin transformations of discrete surfaces. ACM Trans. Graph. (TOG) 30(4), 104 (2011)CrossRefGoogle Scholar
  16. 16.
    Freifeld, O., Black, M.J.: Lie bodies: a manifold representation of 3D human shape. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 1–14. Springer, Heidelberg (2012). Scholar
  17. 17.
    Frenzel, M.F., Teleaga, B., Ushio, A.: Latent space cartography: generalised metric-inspired measures and measure-based transformations for generative models. arXiv preprint arXiv:1902.02113 (2019)
  18. 18.
    Gao, L., Chen, S.Y., Lai, Y.K., Xia, S.: Data-driven shape interpolation and morphing editing. Comput. Graph. Forum 36(8), 19–31 (2017)CrossRefGoogle Scholar
  19. 19.
    Gao, L., Lai, Y.K., Huang, Q.X., Hu, S.M.: A data-driven approach to realistic shape morphing. Comput. Graph. Forum 32(2pt4), 449–457 (2013)Google Scholar
  20. 20.
    Gluck, H.: Almost all simply connected closed surfaces are rigid. In: Glaser, L.C., Rushing, T.B. (eds.) Geometric Topology. LNM, vol. 438, pp. 225–239. Springer, Heidelberg (1975). Scholar
  21. 21.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: 3D-coded: 3D correspondences by deep deformation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 230–246 (2018)Google Scholar
  22. 22.
    Hasler, N., Stoll, C., Sunkel, M., Rosenhahn, B., Seidel, H.P.: A statistical model of human pose and body shape. Comput. Graph. Forum 28(2), 337–346 (2009)CrossRefGoogle Scholar
  23. 23.
    Heeren, B., Rumpf, M., Schröder, P., Wardetzky, M., Wirth, B.: Exploring the geometry of the space of shells. In: Computer Graphics Forum, vol. 33, pp. 247–256. Wiley Online Library (2014)Google Scholar
  24. 24.
    Heeren, B., Rumpf, M., Schröder, P., Wardetzky, M., Wirth, B.: Splines in the space of shells. Comput. Graph. Forum 35(5), 111–120 (2016)CrossRefGoogle Scholar
  25. 25.
    Heeren, B., Rumpf, M., Wardetzky, M., Wirth, B.: Time-discrete geodesics in the space of shells. Comput. Graph. Forum 31(5), 1755–1764 (2012)CrossRefGoogle Scholar
  26. 26.
    Huang, J., et al.: Subspace gradient domain mesh deformation. ACM Trans. Graph. (TOG) 25(3), 1126–1134 (2006)CrossRefGoogle Scholar
  27. 27.
    Huang, R., Rakotosaona, M.J., Achlioptas, P., Guibas, L., Ovsjanikov, M.: OperatorNet: recovering 3D shapes from difference operators. arXiv preprint arXiv:1904.10754 (2019)
  28. 28.
    Igarashi, T., Moscovich, T., Hughes, J.F.: As-rigid-as-possible shape manipulation. ACM Trans. Graph. (TOG) 24(3), 1134–1141 (2005)CrossRefGoogle Scholar
  29. 29.
    Kendall, D.G.: Shape manifolds, procrustean metrics, and complex projective spaces. Bull. Lond. Math. Soc. 16(2), 81–121 (1984)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Kilian, M., Mitra, N.J., Pottmann, H.: Geometric modeling in shape space. ACM Trans. Graph. (TOG) 26(3), 64 (2007)CrossRefGoogle Scholar
  31. 31.
    Laine, S.: Feature-based metrics for exploring the latent space of generative models (2018)Google Scholar
  32. 32.
    Lazarus, F., Verroust, A.: Three-dimensional metamorphosis: a survey. Vis. Comput. 14(8), 373–389 (1998)CrossRefGoogle Scholar
  33. 33.
    Li, C.L., Zaheer, M., Zhang, Y., Poczos, B., Salakhutdinov, R.: Point cloud GAN. arXiv preprint arXiv:1810.05795 (2018)
  34. 34.
    Lipman, Y., Cohen-Or, D., Gal, R., Levin, D.: Volume and shape preservation via moving frame manipulation. ACM Trans. Graph. (TOG) 26(1), 5 (2007)CrossRefGoogle Scholar
  35. 35.
    Liu, X., Han, Z., Wen, X., Liu, Y.S., Zwicker, M.: L2G auto-encoder: understanding point clouds by local-to-global reconstruction with hierarchical self-attention. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 989–997. ACM (2019)Google Scholar
  36. 36.
    Michor, P.W., Mumford, D.B.: Riemannian geometries on spaces of plane curves. J. Eur. Math. Soc. (2006)Google Scholar
  37. 37.
    Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of the CVPR, pp. 652–660 (2017)Google Scholar
  38. 38.
    Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, pp. 5099–5108 (2017)Google Scholar
  39. 39.
    von Radziewsky, P., Eisemann, E., Seidel, H.P., Hildebrandt, K.: Optimized subspaces for deformation-based modeling and shape interpolation. Comput. Graph. 58, 128–138 (2016)CrossRefGoogle Scholar
  40. 40.
    Sassen, J., Heeren, B., Hildebrandt, K., Rumpf, M.: Solving variational problems using nonlinear rotation-invariant coordinates. In: Bommes, D., Huang, H. (eds.) Symposium on Geometry Processing 2019- Posters. The Eurographics Association (2019)Google Scholar
  41. 41.
    Shao, H., Kumar, A., Thomas Fletcher, P.: The Riemannian geometry of deep generative models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 315–323 (2018)Google Scholar
  42. 42.
    Shen, Y., Feng, C., Yang, Y., Tian, D.: Mining point cloud local structures by kernel correlation and graph pooling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4548–4557 (2018)Google Scholar
  43. 43.
    Solomon, J., et al.: Convolutional Wasserstein distances: efficient optimal transportation on geometric domains. ACM Trans. Graph. (TOG) 34(4), 66 (2015)CrossRefGoogle Scholar
  44. 44.
    Varol, G., et al.: Learning from synthetic humans. In: CVPR (2017)Google Scholar
  45. 45.
    Vaxman, A., Müller, C., Weber, O.: Conformal mesh deformations with Möbius transformations. ACM Trans. Graph. (TOG) 34(4), 55 (2015)CrossRefGoogle Scholar
  46. 46.
    Von Funck, W., Theisel, H., Seidel, H.P.: Vector field based shape deformations. ACM Trans. Graph. (TOG) 25(3), 1118–1125 (2006)CrossRefGoogle Scholar
  47. 47.
    Wang, Y., Liu, B., Tong, Y.: Linear surface reconstruction from discrete fundamental forms on triangle meshes. In: Computer Graphics Forum, vol. 31, pp. 2277–2287. Wiley Online Library (2012)Google Scholar
  48. 48.
    Wirth, B., Bar, L., Rumpf, M., Sapiro, G.: A continuum mechanical approach to geodesics in shape space. Int. J. Comput. Vis. 93(3), 293–318 (2011)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Wu, J., Zhang, C., Xue, T., Freeman, B., Tenenbaum, J.: Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In: Advances in Neural Information Processing Systems, pp. 82–90 (2016)Google Scholar
  50. 50.
    Xu, D., Zhang, H., Wang, Q., Bao, H.: Poisson shape interpolation. Graph. Models 68(3), 268–281 (2006)CrossRefGoogle Scholar
  51. 51.
    Zhang, Z., Li, G., Lu, H., Ouyang, Y., Yin, M., Xian, C.: Fast as-isometric-as-possible shape interpolation. Comput. Graph. 46, 244–256 (2015)CrossRefGoogle Scholar
  52. 52.
    Zuffi, S., Kanazawa, A., Jacobs, D., Black, M.J.: 3D menagerie: modeling the 3D shape and pose of animals. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.LIX, Ecole Polytechnique, IP ParisPalaiseauFrance

Personalised recommendations