Skip to main content
SpringerLink
Log in

MSGCN: a multiscale spatio graph convolution network for 3D point clouds

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We propose a multiscale spatio graph neural network (MSGCN) for 3D point cloud. The core of MSGCN is a multiscale spatio graph(MSG) that explicitly models the relations at various spatial scales. Different from many previous hierarchical structures, the MSG is built in a data adaptive fashion. MSG supports multiscale analysis of point clouds in the scale space and can obtain the dimensional features of point cloud data at different scales. Because traditional convolutional neural networks are not applicable to graph data with irregular vertex neighborhoods, this paper presents an sef-adaptive graph convolution kernel that uses the Chebyshev polynomial to fit an irregular convolution filter based on the theory of optimal approximation. In experiments conducted on four widely used public datasets, The results show that the proposed model outperforms most state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Adams A, Baek J, Davis A (2010) Fast high-dimensional filtering using the permutohedral lattice. Eurographics 7:162–179. https://doi.org/10.1111/j.1467-8659.2009.01645.x

    Article  Google Scholar 

  2. Benson D, Davis J (2015) Octree textures. SIGGRAPH 3:785–790

    Google Scholar 

  3. Brock A, Lim T, Ritchie JM, Weston N (2016) Generative and discriminative voxel modeling with convolutional neural networks. 3, p 5648–5656 . arXiv:http://arxiv.org/abs/1608.04236

  4. Caesar H, Bankiti V, Lang AH, Vora S, Liong VE, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O (2020) nuScenes: A Multimodal Dataset for Autonomous Driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 11621–11631

  5. Cao D, Wang Y, Duan J, Zhang C, Zhu X, Huang C, Tong Y, Xu B, Bai J, Tong J et al (2021) Spectral temporal graph neural network for multivariate time-series forecasting

  6. Chen K, Franko K, Sang R (2021) Structured Model Pruning of Convolutional Networks on Tensor Processing Units

  7. Cheng XJ, Guo W, Li Q (2017) Joint classification method for terrestrial LiDAR point cloud based on intensity and color information, vol 44

  8. Feng Y, Zizhao Z, Zhao X, Ji R (2018) GVCNN: Group-View Convolutional Neural Networks for 3D Shape Recognition. Conference on Computer Vision and Pattern Recognition(CVPR) 7:57–70. https://doi.org/10.1109/CVPR.2018.00035

    Article  Google Scholar 

  9. Grover A, Leskovec J (2016) node2vec: Scalable feature learning for networks

  10. Gumhold S, Wang X, Macleod R (2001) Feature extraction from point clouds. Proc Ofimr 3:293–305

    Google Scholar 

  11. Henaff M, Bruna J, LeCun Y (2015) Deep Convolutional Networks on Graph-Structured Data. NIPS 7:305–312

    Google Scholar 

  12. He MY, Cheng YL, Liao XJ (2018) Building extraction algorithm by fusing spectral and geometrical feature. Laser Optoelectron Prog 55:28–35

    Google Scholar 

  13. Hsu SH, Lai JY (2009) Extraction of geodesic and feature lines on triangular meshes. Int J Adv Manuf Technol 42:940–954

    Article  Google Scholar 

  14. Jin W, Barzilay R, Jaakkola T (2018) Junction tree variational autoencoder for molecular graph generation

  15. Jun Wu (2013) Aerial LiDAR Data Classification Using Weighted Support Vector Machines. Geomat Inf Sci Wuhan Univ 8009(1):800926–800926. https://doi.org/10.1117/12.896198

    Article  Google Scholar 

  16. Kim SK (2013) Extraction of ridge and valley lines from unorganized points. Multimed Tools Appl 63:265–279

    Article  Google Scholar 

  17. Kingma DP, Ba J (2015) Adam: a Method for Stochastic Optimization

  18. Klokov R, Lempitsky V (2017) Escape From Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models. IEEE International Conference on Computer Vision (ICCV) 12:15–38. https://doi.org/10.1109/ICCV.2017.99

    Article  Google Scholar 

  19. Lu C (2018) PointSIFT: a SIFT-like network module for 3D point cloud semantic segmentation. CVPR 42:256–278

    Google Scholar 

  20. Manyun H, Yinglei C, Xiangjiang L (2018) Building Extraction Algorithm by Fusing Spectral and Geometrical Features, vol 4

  21. Maturana S (2015) VoxNet:A 3D Convolutional Neural Network for real-time object recognition. Int Conf Intell Robots Syst 7:922–928. https://doi.org/10.1109/IROS.2015.7353481

    Article  Google Scholar 

  22. Meng Q, Wang W (2021) Towards A Weakly Supervised Framework for 3D Point Cloud Object Detection and Annotation, vol PP

  23. Meng Q, Wang W, Zhou T, Shen J, Jia Y, Van Gool L (2021) Towards A Weakly Supervised Framework for 3D Point Cloud Object Detection and Annotation

  24. Nascimento ER, Oliveira GL, Campos MFM (2012) BRAND: a robust appearance and depth descriptor for RGB-D images. Intell Robots Syst (IROS) 7:1720–1726

    Google Scholar 

  25. Perozzi B, Al-Rfou R, Skiena S (2014) Deepwalk: Online learning of social representations

  26. Qi CR, Su H, Mo K, Guibas LJ (2017) PointNet: deep learning on point sets for 3D classification and segmentation. CVPR 8:298–301. https://doi.org/10.1109/CVPR.2017.16

    Article  Google Scholar 

  27. Qi S, Wang W, Jia B, Shen J, Zhu S-C (2018) Learning human-object interactions by graph parsing neural networks

  28. Qi CR, Yi L, Su H, Guibas LJ (2017) Deep hierarchical feature learning on point sets in a metric space. Adv Neural Inf Process Syst 3:5105–5114

    Google Scholar 

  29. Rusu RB, Blodow N, Beetz M (2009) Fast Point Feature Histograms (FPFH) for 3D registration. IEEE IEEE Int Conf Robot Autom 4:3212–3217

    Google Scholar 

  30. Rusu RB, Bradski G, Thibaux R (2010) Fast 3D recognition and pose using the Viewpoint Feature Histogram. Int Conf Intell Robot Syst 9:2155–2162

    Google Scholar 

  31. Sadeghi D, Shoeibi A, Ghassemi N, Moridian P, Khadem A, Alizadehsani R, Teshnehlab M, Górriz JM, Nahavandi S (2021) An Overview on Artificial Intelligence Techniques for Diagnosis of Schizophrenia Based on Magnetic Resonance Imaging Modalities: Methods, Challenges, and Future Works. CoRR arXiv:http://arxiv.org/abs/2103.03081

  32. Shoeibi A, Ghassemi N, Khodatars M (2021) Detection of epileptic seizures on EEG signals using ANFIS classifier, autoencoders and fuzzy entropies. CoRR arXiv:http://arxiv.org/abs/2105.14278

  33. Shoeibi A, Ghassemi N, Khodatars M, Jafari M, Moridian P, Alizadehsani R, Khadem A, Kong Y, Zare A, Górriz JM, Ramírez J, Panahiazar M, Khosravi A, Nahavandi S (2021) Applications of Epileptic Seizures Detection in Neuroimaging Modalities Using Deep Learning Techniques: Methods, Challenges, and Future Works. CoRR arXiv:http://arxiv.org/abs/2105.14278

  34. Shoeibi A, Khodatars M, Alizadehsani R, Ghassemi N, Jafari M, Moridian P, Khadem A, Sadeghi D, Hussain S, Zare A, Sani ZA, Bazeli J, Khozeimeh F, Khosravi A, Nahavandi S, Acharya UR, Shi P (2020) Automated Detection and Forecasting of COVID-19 using Deep Learning Techniques: A Review. CoRR arXiv:http://arxiv.org/abs/2007.10785

  35. Shoeibi A, Khodatars M, Jafari M, Moridian P, Rezaei M, Alizadehsani R, Khozeimeh F, Gorriz JM, Heras J, Panahiazar M, Nahavandi S et al (2021) Applications of deep learning techniques for automated multiple sclerosis detection using magnetic resonance imaging: A review. Computers in Biology and Medicine 136:104697. https://doi.org/10.1016/j.compbiomed.2021.104697

    Article  Google Scholar 

  36. Simonovsky M, Komodakis N (2017) Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), DOI https://doi.org/10.1109/CVPR.2017.11, (to appear in print)

  37. Su H, Maji S, Kalogerakis E, Learned-Miller E (2016) Multi-view convolutional neural networks for 3D shape recognition. ICCV 4:114–121. https://doi.org/10.1109/ICCV.2015.114

    Article  Google Scholar 

  38. Tang J, Qu M, Wang M, Zhang M, Yan J, Mei Q (2015) Line: largescale information network embedding. SIGKDD

  39. Tombari F, Salti S, Stefano LD (2011) A combined texture-shape descriptor for enhanced 3D feature matching. IEEE International Conference on Image Processing 4:809–812

    Google Scholar 

  40. Wohlkinger W, Vincze M (2011) Ensemble of shape functions for 3D object classification. IEEE Int Conf Robot Biomimet 3:2987–2992

    Google Scholar 

  41. Wu B, Liu Y, Lang B, Huang L (2017) DGCNN: Disordered Graph Convolutional Neural Network Based on the Gaussian Mixture Model. Neurocomputing 3:346–356

    Google Scholar 

  42. Yan S, Xiong Y, Lin D (2018a) Geometry-aware graph transforms for light field compact representation. IEEE Transactions on Image Process

  43. Yan S, Xiong Y, Lin D (2018b) Spatial temporal graph convolutional networks for skeleton-based action recognition. AAAI

  44. Yang Y, Feng C, Shen Y, Tian D (2018a) FoldingNet: Point Cloud Auto-encoder via Deep Grid Deformation. IEEE Conference on Computer Vision and Pattern Recognition(CVPR) 7:321–334

  45. Yang Y, Feng C, Shen Y, Tian D (2018b) PPFNet: Global Context Aware Local Features for Robust 3D Point Matching. CVPR 7:217–223. https://doi.org/10.1109/CVPR.2018.00028

  46. Yin J, Shen J, Gao X, Crandall D, Yang R (2021) Graph neural network and spatiotemporal transformer attention for 3d video object detection from point clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence

  47. Yin J, Shen J, Guan C, Zhou D, Yang R (2020) Lidar-based online 3d video object detection with graph-based message passing and spatiotemporal transformer attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 11495-11504

  48. Yin T, Zhou X, Krahenbuhl P (2021) Center-based 3d object detection and tracking, computer vision and pattern recognition. 11784-11793

  49. You J, Liu B, Ying R, Pande V, Leskovec J (2018) Graph convolutional policy network for goal-directed molecular graph generation. arXiv:http://arxiv.org/abs/1806.02473

  50. Yu T, Meng J, Yuan J (2018) Multi-view harmonized bilinear network for 3D object recognition. Conf Comput Vision Patt Recog (CVPR) 7:90–105. https://doi.org/10.1109/CVPR.2018.00027

    Article  Google Scholar 

  51. Zheng C, Pan L, Wu P (2020) Multimodal deep network embedding with integrated structure and attribute information. TNNL

Download references

Acknowledgements

This paper is supported by Opening Foundation of Key Laboratory of Computer Network and Information Integration(Southeast University), Ministry of Education (K93-9-2021-05).

Author information

Authors and Affiliations

  1. School of Software, Nanchang Hangkong University, Nanchang, 330063, China

    Bo Wu

  2. Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, Nanjing, 211189, China

    Bo Wu

  3. Beihang University, Beijing, 100083, China

    Bo Wu & Bo Lang

Authors
  1. Bo Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Lang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Wu.

Ethics declarations

Conflict of interests

The authors declare that there is no conflict of interests regarding the publication of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, B., Lang, B. MSGCN: a multiscale spatio graph convolution network for 3D point clouds. Multimed Tools Appl 82, 35949–35968 (2023). https://doi.org/10.1007/s11042-023-14639-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14639-z

Keywords

Search

Navigation