Deformation-Aware 3D Model Embedding and Retrieval

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12352)


We introduce a new problem of retrieving 3D models that are deformable to a given query shape and present a novel deep deformation-aware embedding to solve this retrieval task. 3D model retrieval is a fundamental operation for recovering a clean and complete 3D model from a noisy and partial 3D scan. However, given a finite collection of 3D shapes, even the closest model to a query may not be satisfactory. This motivates us to apply 3D model deformation techniques to adapt the retrieved model so as to better fit the query. Yet, certain restrictions are enforced in most 3D deformation techniques to preserve important features of the original model that prevent a perfect fitting of the deformed model to the query. This gap between the deformed model and the query induces asymmetric relationships among the models, which cannot be handled by typical metric learning techniques. Thus, to retrieve the best models for fitting, we propose a novel deep embedding approach that learns the asymmetric relationships by leveraging location-dependent egocentric distance fields. We also propose two strategies for training the embedding network. We demonstrate that both of these approaches outperform other baselines in our experiments with both synthetic and real data. Our project page can be found at


3D model retrieval Deformation-aware embedding Non-metric embedding 



This work is supported by a Google AR/VR University Research Award, a Vannevar Bush Faculty Fellowship, a grant from the Stanford SAIL Toyota Research Center, and gifts from the Adobe Corporation and the Dassault Foundation.

Supplementary material

504444_1_En_24_MOESM1_ESM.pdf (24.6 mb)
Supplementary material 1 (pdf 25139 KB)


  1. 1.
    Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.J.: Learning representations and generative models for 3D point clouds. In: ICML (2018)Google Scholar
  2. 2.
    Ahmed, A., Shervashidze, N., Narayanamurthy, S., Josifovski, V., Smola, A.J.: Distributed large-scale natural graph factorization. In: WWW (2013)Google Scholar
  3. 3.
    Aldrovandi, R., Pereira, J.: An Introduction to Geometrical Physics. World Scientific (1995)Google Scholar
  4. 4.
    Arandjelović, R., Gronat, P., Torii, A., Pajdla, T., Sivic, J.: NetVLAD: CNN architecture for weakly supervised place recognition. In: CVPR (2016)Google Scholar
  5. 5.
    Avetisyan, A., Dahnert, M., Dai, A., Savva, M., Chang, A.X., Nießner, M.: Scan2CAD: learning cad model alignment in RGB-D scans. In: CVPR (2019)Google Scholar
  6. 6.
    Avetisyan, A., Dai, A., Nießner, M.: End-to-end cad model retrieval and 9DoF alignment in 3D scans. In: ICCV (2019)Google Scholar
  7. 7.
    Bellet, A., Habrard, A., Sebban, M.: A survey on metric learning for feature vectors and structured data (2013)Google Scholar
  8. 8.
    Buldygin, V., et al.: Metric Characterization of Random Variables and Random Processes. American Mathematical Society (2000)Google Scholar
  9. 9.
    Bylow, E., Sturm, J., Kerl, C., Kahl, F., Cremers, D.: Real-time camera tracking and 3D reconstruction using signed distance functions. In: RSS (2013)Google Scholar
  10. 10.
    Chang, A.X., et al.: Shapenet: an information-rich 3D model repository (2015)Google Scholar
  11. 11.
    Chechik, G., Sharma, V., Shalit, U., Bengio, S.: Large scale online learning of image similarity through ranking. J. Mach. Learn. Res. 11, 1109–1135 (2010)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Chen, L., Lian, X.: Efficient similarity search in nonmetric spaces with local constant embedding. IEEE Trans. Knowl. Data Eng. 20(3), 321–336 (2008)CrossRefGoogle Scholar
  13. 13.
    Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: CVPR (2005)Google Scholar
  14. 14.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)zbMATHGoogle Scholar
  15. 15.
    Dahnert, M., Dai, A., Guibas, L., Nießner, M.: Joint embedding of 3D scan and cad objects. In: ICCV (2019)Google Scholar
  16. 16.
    Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., Theobalt, C.: BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. In: ACM SIGGRAPH (2017)Google Scholar
  17. 17.
    Dai, A., Ruizhongtai Qi, C., Nießner, M.: Shape completion using 3D-encoder-predictor CNNs and shape synthesis. In: CVPR (2017)Google Scholar
  18. 18.
    Deng, H., Birdal, T., Ilic, S.: PPFNet: global context aware local features for robust 3D point matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 195–205 (2018)Google Scholar
  19. 19.
    Dong, Y., Chawla, N.V., Swami, A.: metapath2vec: scalable representation learning for heterogeneous networks. In: KDD (2017)Google Scholar
  20. 20.
    Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: CVPR (2016)Google Scholar
  21. 21.
    G, V.K.B., Carneiro, G., Reid, I.: Learning local image descriptors with deep siamese and triplet convolutional networks by minimizing global loss functions. In: CVPR (2016)Google Scholar
  22. 22.
    Garcia, N., Vogiatzis, G.: Learning non-metric visual similarity for image retrieval. Image Vis. Comput. 82, 18–25 (2019)CrossRefGoogle Scholar
  23. 23.
    Garland, M., Heckbert, P.S.: Simplifying surfaces with color and texture using quadric error metrics. In: Visualization (1998)Google Scholar
  24. 24.
    Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: Deep self-supervised cycle-consistent deformation for few-shot shape segmentation. In: Eurographics Symposium on Geometry Processing (2019)Google Scholar
  25. 25.
    Grover, A., Leskovec, J.: node2vec: scalable feature learning for networks. In: KDD (2016)Google Scholar
  26. 26.
    Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: CVPR (2006)Google Scholar
  27. 27.
    Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C.: MatchNet: unifying feature and metric learning for patch-based matching. In: CVPR (2015)Google Scholar
  28. 28.
    Hanocka, R., Fish, N., Wang, Z., Giryes, R., Fleishman, S., Cohen-Or, D.: ALIGNet: partial-shape agnostic alignment via unsupervised learning. ACM Trans. Graph. 38(1), 1–14 (2018)CrossRefGoogle Scholar
  29. 29.
    Hinton, G.E., Roweis, S.T.: Stochastic neighbor embedding. In: NIPS (2003)Google Scholar
  30. 30.
    Huang, J., Dai, A., Guibas, L.J., Nießner, M.: 3Dlite: towards commodity 3D scanning for content creation. In: ACM SIGGRAPH Asia (2017)Google Scholar
  31. 31.
    Huang, J., Su, H., Guibas, L.: Robust watertight manifold surface generation method for shapenet models (2018)Google Scholar
  32. 32.
    Igarashi, T., Moscovich, T., Hughes, J.F.: As-rigid-as-possible shape manipulation. In: ACM SIGGRAPH (2005)Google Scholar
  33. 33.
    Jack, D., et al.: Learning free-Form deformations for 3D object reconstruction. In: ICCV (2018)Google Scholar
  34. 34.
    Joshi, P., Meyer, M., DeRose, T., Green, B., Sanocki, T.: Harmonic coordinates for character articulation. In: ACM SIGGRAPH (2007)Google Scholar
  35. 35.
    Ju, T., Schaefer, S., Warren, J.: Mean value coordinates for closed triangular meshes. In: ACM SIGGRAPH (2005)Google Scholar
  36. 36.
    Kraevoy, V., Sheffer, A., Shamir, A., Cohen-Or, D.: Non-homogeneous resizing of complex models. In: ACM SIGGRAPH Asia (2006)Google Scholar
  37. 37.
    Kulis, B., et al.: Metric learning: a survey. Found. Trends® Mach. Learn. 5(4), 287–364 (2013)Google Scholar
  38. 38.
    Kurenkov, A., et al.: DeformNet: free-form deformation network for 3D shape reconstruction from a single image. In: WACV (2018)Google Scholar
  39. 39.
    Li, Y., Dai, A., Guibas, L., Nießner, M.: Database-assisted object retrieval for real-time 3D reconstruction. In: Eurographics (2015)Google Scholar
  40. 40.
    Lipman, Y., Sorkine, O., Cohen-Or, D., Levin, D., Rossi, C., Seidel, H.P.: Differential coordinates for interactive mesh editing. In: Shape Modeling Applications (2004)Google Scholar
  41. 41.
    Lipman, Y., Levin, D., Cohen-Or, D.: Green coordinates. In: ACM SIGGRAPH (2008)Google Scholar
  42. 42.
    Lipman, Y., Sorkine, O., Levin, D., Cohen-Or, D.: Linear rotation-invariant coordinates for meshes. In: ACM SIGGRAPH (2005)Google Scholar
  43. 43.
    Liu, E.Y., Guo, Z., Zhang, X., Jojic, V., Wang, W.: Metric learning from relative comparisons by minimizing squared residual. In: ICDM (2012)Google Scholar
  44. 44.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)zbMATHGoogle Scholar
  45. 45.
    Mahalanobis, P.C.: On the generalized distance in statistics. In: Proceedings of the National Institute of Science. National Institute of Science of India (1936)Google Scholar
  46. 46.
    Morozov, S., Babenko, A.: Non-metric similarity graphs for maximum inner product search. In: Advances in Neural Information Processing Systems (2018)Google Scholar
  47. 47.
    Newcombe, R.A., et al.: KinectFusion: real-time dense surface mapping and tracking. In: ISMAR (2011)Google Scholar
  48. 48.
    Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: online learning of social representations. In: KDD (2013)Google Scholar
  49. 49.
    Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: CVPR (2017)Google Scholar
  50. 50.
    Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: CVPR (2015)Google Scholar
  51. 51.
    Sederberg, T.W., Parry, S.R.: Free-form deformation of solid geometric models. In: ACM SIGGRAPH (1986)Google Scholar
  52. 52.
    Simo-Serra, E., Trulls, E., Ferraz, L., Kokkinos, I., Fua, P., Moreno-Noguer, F.: Discriminative learning of deep convolutional feature point descriptors. In: ICCV (2015)Google Scholar
  53. 53.
    Skopal, T.: On fast non-metric similarity search by metric access methods. In: Ioannidis, Y., et al. (eds.) EDBT 2006. LNCS, vol. 3896, pp. 718–736. Springer, Heidelberg (2006). Scholar
  54. 54.
    Skopal, T., Bustos, B.: On nonmetric similarity search problems in complex domains. ACM Comput. Surv. (CSUR) 43(4), 1–50 (2011)CrossRefGoogle Scholar
  55. 55.
    Skopal, T., Lokoč, J.: NM-tree: flexible approximate similarity search in metric and non-metric spaces. In: Bhowmick, S.S., Küng, J., Wagner, R. (eds.) DEXA 2008. LNCS, vol. 5181, pp. 312–325. Springer, Heidelberg (2008). Scholar
  56. 56.
    Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Eurographics Symposium on Geometry Processing (2007)Google Scholar
  57. 57.
    Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., Seidel, H.P.: Laplacian surface editing. In: Eurographics Symposium on Geometry Processing (2004)Google Scholar
  58. 58.
    Stratasys: GrabCAD community.
  59. 59.
    Tan, X., Chen, S., Li, J., Zhou, Z.H.: Learning non-metric partial similarity based on maximal margin criterion. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 1. IEEE (2006)Google Scholar
  60. 60.
    Tian, Y., Fan, B., Wu, F.: L2-Net: deep learning of discriminative patch descriptor in euclidean space. In: CVPR (2017)Google Scholar
  61. 61.
    Trimble: 3D warehouse.
  62. 62.
    TurboSquid: TurboSquid.
  63. 63.
    Wang, W., Ceylan, D., Mech, R., Neumann, U.: 3DN: 3D deformation network. In: CVPR (2019)Google Scholar
  64. 64.
    Weber, O., Ben-Chen, M., Gotsman, C.: Complex barycentric coordinates with applications to planar shape deformation. In: Eurographics (2009)Google Scholar
  65. 65.
    Whelan, T., Leutenegger, S., Salas-Moreno, R.F., Glocker, B., Davison, A.J.: ElasticFusion: dense slam without a pose graph. Robot.: Sci. Syst. (2011)Google Scholar
  66. 66.
    Hamilton, W.L., Ying, R., Leskovec, J.: Representation learning on graphs: methods and applications. IEEE Data Eng. Bull. (2017)Google Scholar
  67. 67.
    Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: Pointflow: 3D point cloud generation with continuous normalizing flows. In: ICCV (2019)Google Scholar
  68. 68.
    Yifan, W., Aigerman, N., Kim, V., Chaudhuri, S., Sorkine-Hornung, O.: Neural cages for detail-preserving 3D deformations (2019)Google Scholar
  69. 69.
    Yumer, E., Mitra, N.J.: Learning semantic deformation flows with 3D convolutional networks. In: ECCV (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Stanford UniversityStanfordUSA
  2. 2.Adobe ResearchSan JoseUSA

Personalised recommendations