CAD-Deform: Deformable Fitting of CAD Models to 3D Scans

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)


Shape retrieval and alignment are a promising avenue towards turning 3D scans into lightweight CAD representations that can be used for content creation such as mobile or AR/VR gaming scenarios. Unfortunately, CAD model retrieval is limited by the availability of models in standard 3D shape collections (e.g., ShapeNet). In this work, we address this shortcoming by introducing CAD-Deform (The code for the project:, a method which obtains more accurate CAD-to-scan fits by non-rigidly deforming retrieved CAD models. Our key contribution is a new non-rigid deformation model incorporating smooth transformations and preservation of sharp features, that simultaneously achieves very tight fits from CAD models to the 3D scan and maintains the clean, high-quality surface properties of hand-modeled CAD objects. A series of thorough experiments demonstrate that our method achieves significantly tighter scan-to-CAD fits, allowing a more accurate digital replica of the scanned real-world environment while preserving important geometric features present in synthetic CAD environments.


Scene reconstruction Mesh deformation 



The authors acknowledge the usage of the Skoltech CDISE HPC cluster Zhores for obtaining the results presented in this paper. The work was partially supported by the Russian Science Foundation under Grant 19-41-04109.

Supplementary material


  1. 1.
    Achenbach, J., Zell, E., Botsch, M.: Accurate face reconstruction through anisotropic fitting and eye correction. In: VMV, pp. 1–8 (2015)Google Scholar
  2. 2.
    Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and generative models for 3D point clouds. In: International Conference on Machine Learning, pp. 40–49 (2018)Google Scholar
  3. 3.
    Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid ICP algorithms for surface registration. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)Google Scholar
  4. 4.
    Aubry, M., Maturana, D., Efros, A., Russell, B., Sivic, J.: Seeing 3D chairs: exemplar part-based 2D–3D alignment using a large dataset of cad models. In: CVPR (2014)Google Scholar
  5. 5.
    Avetisyan, A., Dahnert, M., Dai, A., Savva, M., Chang, A.X., Nießner, M.: Scan2CAD: learning CAD model alignment in RGB-D scans. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2614–2623 (2019)Google Scholar
  6. 6.
    Avetisyan, A., Dai, A., Nießner, M.: End-to-end CAD model retrieval and 9DoF alignment in 3D scans. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2551–2560 (2019)Google Scholar
  7. 7.
    Botsch, M., Kobbelt, L.: An intuitive framework for real-time freeform modeling. ACM Trans. Graph. (TOG) 23(3), 630–634 (2004)CrossRefGoogle Scholar
  8. 8.
    Cagniart, C., Boyer, E., Ilic, S.: Iterative mesh deformation for dense surface tracking. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp. 1465–1472. IEEE (2009)Google Scholar
  9. 9.
    Chang, A.X., et al.: Shapenet: an information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
  10. 10.
    Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, pp. 303–312. Association for Computing Machinery, New York (1996).
  11. 11.
    Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: ScanNet: richly-annotated 3D reconstructions of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5828–5839 (2017)Google Scholar
  12. 12.
    Dai, A., Nießner, M., Zollöfer, M., Izadi, S., Theobalt, C.: Bundlefusion: real-time globally consistent 3D reconstruction using on-the-fly surface re-integration. In: ACM Transactions on Graphics 2017 (TOG) (2017)Google Scholar
  13. 13.
    Dai, A., Ruizhongtai Qi, C., Nießner, M.: Shape completion using 3D-encoder-predictor CNNs and shape synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5868–5877 (2017)Google Scholar
  14. 14.
    Deng, H., Birdal, T., Ilic, S.: 3D local features for direct pairwise registration. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019Google Scholar
  15. 15.
    Dey, T.K., Fu, B., Wang, H., Wang, L.: Automatic posing of a meshed human model using point clouds. Comput. Graph. 46, 14–24 (2015)CrossRefGoogle Scholar
  16. 16.
    Drost, B., Ilic, S.: 3D object detection and localization using multimodal point pair features. In: 3DIMPVT, pp. 9–16. IEEE Computer Society (2012)Google Scholar
  17. 17.
    Egiazarian, V., et al.: Latent-Space Laplacian Pyramids for Adversarial Representation Learning with 3D Point Clouds, December 2019Google Scholar
  18. 18.
    Firman, M., Mac Aodha, O., Julier, S., Brostow, G.J.: Structured prediction of unobserved voxels from a single depth image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5431–5440 (2016)Google Scholar
  19. 19.
    Fröhlich, S., Botsch, M.: Example-driven deformations based on discrete shells. Comput. Graph. Forum 30, 2246–2257 (2011). Scholar
  20. 20.
    Grinspun, E., Hirani, A.N., Desbrun, M., Schröder, P.: Discrete shells. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA 2003, pp. 62–67. Eurographics Association, Goslar, DEU (2003)Google Scholar
  21. 21.
    Guo, R., Zou, C., Hoiem, D.: Predicting complete 3D models of indoor scenes. arXiv preprint arXiv:1504.02437 (2015)
  22. 22.
    Gupta, S., Arbeláez, P., Girshick, R., Malik, J.: Aligning 3D models to RGB-D images of cluttered scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4731–4740 (2015)Google Scholar
  23. 23.
    He, L., Schaefer, S.: Mesh denoising via l0 minimization. In: Proceedings of ACM SIGGRAPH, pp. 64:1–64:8, January 2013Google Scholar
  24. 24.
    Huang, J., Su, H., Guibas, L.: Robust watertight manifold surface generation method for shapenet models. arXiv preprint arXiv:1802.01698 (2018)
  25. 25.
    Izadi, S., et al.: Kinectfusion: real-time 3D reconstruction and interaction using a moving depth camera. In: UIST 2011 Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 559–568. ACM (2011)Google Scholar
  26. 26.
    Jacobson, A., Tosun, E., Sorkine, O., Zorin, D.: Mixed finite elements for variational surface modeling. In: Computer Graphics Forum, vol. 29, pp. 1565–1574. Wiley Online Library (2010)Google Scholar
  27. 27.
    Koch, S., et al.: ABC: a big cad model dataset for geometric deep learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019Google Scholar
  28. 28.
    Li, Y., Dai, A., Guibas, L., Nießner, M.: Database-assisted object retrieval for real-time 3D reconstruction. Comput. Graph. Forum 34(2), 435–446 (2015)CrossRefGoogle Scholar
  29. 29.
    Liao, M., Zhang, Q., Wang, H., Yang, R., Gong, M.: Modeling Deformable Objects from a Single Depth Camera, pp. 167–174, November 2009.
  30. 30.
    Mattausch, O., Panozzo, D., Mura, C., Sorkine-Hornung, O., Pajarola, R.: Object detection and classification from large-scale cluttered indoor scans. In: Computer Graphics Forum, vol. 33, pp. 11–21. Wiley Online Library (2014)Google Scholar
  31. 31.
    Mo, K., et al.: PartNet: a large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 909–918 (2019)Google Scholar
  32. 32.
    Newcombe, R.A., et al.: Kinectfusion: real-time dense surface mapping and tracking. In: IEEE ISMAR. IEEE, October 2011Google Scholar
  33. 33.
    Nießner, M., Zollhöfer, M., Izadi, S., Stamminger, M.: Real-time 3D reconstruction at scale using voxel hashing. In: ACM Transactions on Graphics (TOG) (2013)Google Scholar
  34. 34.
    Park, S.I., Lim, S.J.: Template-based reconstruction of surface mesh animation from point cloud animation. ETRI J. 36(6), 1008–1015 (2014)CrossRefGoogle Scholar
  35. 35.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings Third International Conference on 3-D Digital Imaging and Modeling, pp. 145–152. IEEE (2001)Google Scholar
  36. 36.
    Salas-Moreno, R.F., Newcombe, R.A., Strasdat, H., Kelly, P.H.J., Davison, A.J.: Slam++: simultaneous localisation and mapping at the level of objects. In: CVPR, pp. 1352–1359. IEEE Computer Society (2013)Google Scholar
  37. 37.
    Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene completion from a single depth image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1746–1754 (2017)Google Scholar
  38. 38.
    Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Symposium on Geometry Processing, vol. 4, pp. 109–116 (2007)Google Scholar
  39. 39.
    Stoll, C., Karni, Z., Rössl, C., Yamauchi, H., Seidel, H.P.: Template deformation for point cloud fitting. In: SPBG, pp. 27–35 (2006)Google Scholar
  40. 40.
    Sungjoon Choi, Zhou, Q., Koltun, V.: Robust reconstruction of indoor scenes. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5556–5565, June 2015.
  41. 41.
    Váša, L., Rus, J.: Dihedral angle mesh error: a fast perception correlated distortion measure for fixed connectivity triangle meshes. Comput. Graph. Forum 31(5), 1715–1724 (2012).,
  42. 42.
    Whelan, T., Leutenegger, S., Salas-Moreno, R.F., Glocker, B., Davison, A.J.: ElasticFusion: dense SLAM without a pose graph. In: Robotics: Science and Systems (RSS), Rome, Italy, July 2015Google Scholar
  43. 43.
    Zhou, X., Karpur, A., Gan, C., Luo, L., Huang, Q.: Unsupervised domain adaptation for 3D keypoint estimation via view consistency. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11216, pp. 141–157. Springer, Cham (2018). Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Skolkovo Institute of Science and TechnologyMoscowRussia
  2. 2.Technical University of MunichMunichGermany
  3. 3.New York UniversityNew YorkUSA

Personalised recommendations