Skip to main content
SpringerLink
Log in

A Survey of Deep Learning-Based Mesh Processing

  • Published:
Communications in Mathematics and Statistics Aims and scope Submit manuscript

Abstract

In the past ten years, deep learning technology has achieved a great success in many fields, like computer vision and speech recognition. Recently, large-scale geometry data become more and more available, and the learned geometry priors have been successfully applied to 3D computer vision and computer graphics fields. Different from the regular representation of images, surface meshes have irregular structures with different vertex numbers and topologies. Therefore, the traditional convolution neural networks used for images cannot be directly used to handle surface meshes, and thus, many methods have been proposed to solve this problem. In this paper, we provide a comprehensive survey of existing geometric deep learning methods for mesh processing. We first introduce the relevant knowledge and theoretical background of geometric deep learning and some basic mesh data knowledge, including some commonly used mesh datasets. Then, we review various deep learning models for mesh data with two different types: graph-based methods and mesh structure-based methods. We also review the deep learning-based applications for mesh data. In the final, we give some potential research directions in this field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Cao, W., Yan, Z., He, Z., et al.: A comprehensive survey on geometric deep learning. IEEE Access 8, 35929–35949 (2020)

    Article  Google Scholar 

  2. Wu, Z., Pan, S., Chen, F., et al.: A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. (2020)

  3. Zhang Z, Cui P, Zhu W.: Deep learning on graphs: a survey. Preprint at arXiv:1812.04202 (2018)

  4. Shuman, D.I., et al.: The emerging field of signal processing on graphs: extending high-dimensional data analysis to networks and other irregular domains. IEEE Signal Process. Mag. 30(3), 83–98 (2013)

    Article  Google Scholar 

  5. Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 209–216 (1997)

  6. Ciampalini, A., Cignoni, P., Montani, C., et al.: Multiresolution decimation based on global error. Vis. Comput. 13(5), 228–246 (1997)

    Article  Google Scholar 

  7. Bruna, J., Zaremba, W., Szlam, A., et al.: Spectral networks and locally connected networks on graphs. Preprint at arXiv:1312.6203 (2013)

  8. Henaff, M., Bruna, J., LeCun, Y.: Deep convolutional networks on graph-structured data. Preprint at arXiv:1506.05163 (2015)

  9. Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in Neural Information Processing Systems, pp. 3844–3852 (2016)

  10. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. Preprint at arXiv:1609.02907 (2016)

  11. Levie, R., Monti, F., Bresson, X., et al.: Cayleynets: graph convolutional neural networks with complex rational spectral filters. IEEE Trans. Signal Process. 67(1), 97–109 (2018)

    Article  MathSciNet  Google Scholar 

  12. Monti, F., Otness, K., Bronstein, M.M., Motifnet: a motif-based graph convolutional network for directed graphs. In: 2018 IEEE Data Science Workshop (DSW), pp. 225–228. IEEE (2018)

  13. Benson, A.R., Gleich, D.F., Leskovec, J.: Higher-order organization of complex networks. Science 353(6295), 163–166 (2016)

    Article  Google Scholar 

  14. Yi, L., Su, H., Guo, X., et al.: SyncSpecCNN: synchronized spectral CNN for 3d shape segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2282–2290 (2017)

  15. Jaderberg, M., Simonyan, K., Zisserman, A.: Spatial transformer networks. In: Advances in Neural Information Processing Systems, pp. 2017–2025 (2015)

  16. Scarselli, F., Gori, M., Tsoi, A.C., et al.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2008)

    Article  Google Scholar 

  17. Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: Advances in Neural Information Processing Systems, pp. 1024–1034 (2017)

  18. Atwood, J., Towsley, D.: Diffusion-convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1993–2001 (2016)

  19. Niepert, M., Ahmed, M., Kutzkov, K.: Learning convolutional neural networks for graphs. In: International Conference on Machine Learning, pp. 2014–2023 (2016)

  20. Monti, F., Boscaini, D., Masci, J., et al.: Geometric deep learning on graphs and manifolds using mixture model CNNs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5115–5124 (2017)

  21. Veličković, P., Cucurull, G., Casanova, A., et al.: Graph attention networks. Preprint at arXiv:1710.10903 (2017)

  22. Wu, Z., Song, S., Khosla, A., et al.: 3d shapenets: A deep representation for volumetric shapes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1912–1920 (2015)

  23. Su, H., Maji, S., Kalogerakis, E, et al.: Multi-view convolutional neural networks for 3d shape recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 945–953 (2015)

  24. Masci, J., Boscaini, D., Bronstein, M., et al.: Geodesic convolutional neural networks on Riemannian manifolds. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 37–45 (2015)

  25. Poulenard, A., Ovsjanikov, M.: Multi-directional geodesic neural networks via equivariant convolution. ACM Trans. Graph. (TOG) 37(6), 1–14 (2018)

    Article  Google Scholar 

  26. Boscaini, D., Masci, J., Rodolà, E., et al.: Learning shape correspondence with anisotropic convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 3189–3197 (2016)

  27. Fey, M., Eric Lenssen, J., Weichert, F., et al.: Splinecnn: fast geometric deep learning with continuous b-spline kernels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 869-877 (2018)

  28. Boscaini, D., Masci, J., Melzi, S., et al.: Learning class-specific descriptors for deformable shapes using localized spectral convolutional networks. Comput. Graph. Forum. 34(5), 13–23 (2015)

    Article  Google Scholar 

  29. Bouritsas, G., Bokhnyak, S., Ploumpis, S., et al.: Neural 3d morphable models: spiral convolutional networks for 3d shape representation learning and generation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7213–7222 (2019)

  30. Ranjan, A., Bolkart, T., Sanyal, S., et al.: Generating 3D faces using convolutional mesh autoencoders. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 704–720 (2018)

  31. Gong, S., Chen, L., Bronstein M, et al.: Spiralnet++: a fast and highly efficient mesh convolution operator. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2019)

  32. Gao, Z., Zhai, G., Zhang, J., et al.: PAI-GCN: permutable anisotropic graph convolutional networks for 3D shape representation learning. Preprint at arXiv:2004.09995 (2020)

  33. Qi, C.R., Su, H., Mo, K., et al.: Pointnet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)

  34. Qi, C.R., Yi, L., Su, H., et al.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, pp. 5099–5108 (2017)

  35. Wang, Y., Sun, Y., Liu, Z., et al.: Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. (tog) 38(5), 1–12 (2019)

    Article  Google Scholar 

  36. Li, G., Muller, M., Thabet, A., et al.: Deepgcns: Can GCNs go as deep as CNNs? In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9267–9276 (2019)

  37. LeCun, Y., Bottou, L., Bengio, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  38. Hanocka, Rana, et al.: MeshCNN: a network with an edge. ACM Trans. Graph. 38, 90:1–90:12 (2019)

    Google Scholar 

  39. Litany, O., Remez, T., Rodola, E., et al.: Deep functional maps: Structured prediction for dense shape correspondence. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5659–5667 (2017)

  40. Li, Q., Liu, S., Hu, L., et al.: Shape correspondence using anisotropic Chebyshev spectral CNNs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14658–14667 (2020)

  41. Cheng, S., Bronstein, M., Zhou, Y., et al.: MeshGAN: non-linear 3D Morphable models of faces. Preprint atarXiv:1903.10384 (2019)

  42. Tan, Q., Gao, L., Lai, Y.-K., et al.: Variational autoencoders for deforming 3D mesh models. Preprint at arXiv:1709.04307 (2017)

  43. Mo, K., Zhu, S., Chang, A.X., et al.: PartNet: a Large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)

  44. Fu, H., Jia, R., Gao, L., et al.: 3d-future: 3d furniture shape with texture. Preprint at arXiv:2009.09633 (2020)

  45. Fu, H., Cai, B., Gao, L., et al.: 3D-FRONT: 3D furnished rooms with layOuts and semaNTics. Preprint at arXiv:2011.09127 (2020)

  46. Cao, C., Weng, Y., Zhou, S., et al.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph. 20(3), 413–25 (2014). https://doi.org/10.1109/TVCG.2013.249. PMID: 24434222

    Article  Google Scholar 

  47. Cheng, S., Kotsia, I., Pantic, M., et al.: 4dfab: a large scale 4d database for facial expression analysis and biometric applications. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5117–5126 (2018)

  48. Bogo, F., Romero, J., Loper, M., et al.: FAUST: dataset and evaluation for 3D mesh registration. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, 2014, pp. 3794–3801. https://doi.org/10.1109/CVPR.2014.491

  49. Bogo, F., Romero, J., Pons-Moll, G., et al.: Dynamic FAUST: registering human bodies in motion. 5573–5582 (2017). https://doi.org/10.1109/CVPR.2017.591

  50. Mahmood, N., Ghorbani, N., Troje, N.F., et al.: AMASS: archive of motion capture as surface shapes. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 5441–5450. https://doi.org/10.1109/ICCV.2019.00554

  51. Dai, A., Chang, A.X., Savva, M., et al.: ScanNet: richly-annotated 3D reconstructions of indoor scenes. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 2432–2443. https://doi.org/10.1109/CVPR.2017.261

  52. Armeni I., Sax, S., Zamir, A., et al.: Joint 2D-3D-semantic data for indoor scene understanding. Preprint at arXiv:1702.01105 (2017)

  53. Cao, Y.-P., Liu, Z.-N., Kuang, Z.-F., et al.: Learning to reconstruct high-quality 3D shapes with cascaded fully convolutional networks. In: 15th European Conference, Munich, Germany, September 8–14, 2018. Proceedings, Part IX (2018). https://doi.org/10.1007/978-3-030-01240-3_38

  54. Kalogerakis, E., Hertzmann, A., Singh, K.: Learning 3D Mesh segmentation and labeling. ACM Trans. Graph. 29(3) (2010)

  55. Qiao, Y., Gao, L., Yang, J., et al.: LaplacianNet: learning on 3D meshes with Laplacian encoding and pooling. In: IEEE Transactions on Visualization and Computer Graphics (2020)

  56. Chen, Z., Yin, K., Fisher, M., et al.: BAE-NET: branched autoencoder for shape co-segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8489–8498 (2019)

  57. Lévy, B., Zhang, H.: Spectral mesh processing. In: ACM SIGGRAPH 2010 Courses (SIGGRAPH’10). Association for Computing Machinery, New York, NY, USA, Article 8, pp. 1–312 (2010). https://doi.org/10.1145/1837101.1837109

  58. Zhu, C., Yi, R., Lira, W., et al.: Deformation-driven shape correspondence via shape recognition. ACM Trans. Graph. 36(4), Article 51, 12 pages (2017). https://doi.org/10.1145/3072959.3073613

  59. Dai, A., Qi, C.R., Nießner, M.: Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6545–6554 (2017)

  60. Wang, N., Zhang, Y., Li, Z., et al.: Pixel2Mesh: generating 3D mesh models from single RGB images. In: ECCV (2018)

  61. Wen, C., Zhang, Y., Li, Z., Fu, Y.: Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1042–1051 (2019)

  62. Yuan, Y., Lai, Y., Yang, J., et al.: Mesh variational autoencoders with edge contraction pooling. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1105–1112 (2020)

  63. Wu, F., Zhang, T., Souza, A., et al.: Simplifying graph convolutional networks. Preprint at arXiv:1902.07153

  64. Chen, M., Wei, Z., Huang, Z., et al.: Simple and deep graph convolutional networks. Preprint at arXiv:2007.02133

  65. Van Kaick, O., Zhang, H., Hamarneh, G., et al.: A survey on shape correspondence. In: Computer Graphics Forum, vol. 30, issue 6, pp. 1681–1707. Blackwell Publishing Ltd, Oxford (2011)

  66. Li, G., Xiong, C., Thabet, A., et al.: Deepergcn: all you need to train deeper GCNs. Preprint arXiv:2006.07739 (2020)

Download references

Author information

Authors and Affiliations

  1. School of Mathematical Sciences, University of Science and Technology of China, Hefei, 230026, People’s Republic of China

    He Wang & Juyong Zhang

Authors
  1. He Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juyong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to He Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, H., Zhang, J. A Survey of Deep Learning-Based Mesh Processing. Commun. Math. Stat. 10, 163–194 (2022). https://doi.org/10.1007/s40304-021-00246-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40304-021-00246-7

Keywords

Mathematics Subject Classification

Search

Navigation