Skip to main content
SpringerLink
Log in

PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution

  • Regular Papers
  • Robot and Applications
  • Published:
International Journal of Control, Automation and Systems Aims and scope Submit manuscript

Abstract

In the field of point cloud scene segmentation with deep learning, the ability of the network to extract spatial structure information limits the performance of semantic segmentation. This work proposes a novel framework named PointMS, which handles the semantic segmentation of point cloud scene, to solve the problem of missing local feature information due to the lack of spatial structure information on the training stage. The structure of framework utilizes spatial structure information of point cloud and balances the extraction of global feature and subtle feature when processing point cloud data. Firstly, a multi-scale combination module (SIFT-MS) is used to extract local features of different scales for enhancing the perception of local structure information at each point. Secondly, the process of feature transmission often leads to the loss of information, so a feature supplement module (FSM) is proposed to complete the information lost after feature transformation through the effective combination of global feature and subtle feature. This module integrates the features of different locations to supplement the information lost in feature conversion. The experimental results demonstrate that the proposed framework is efficient for semantic segmentation of S3DIS dataset. SIFT-MS module and FSM module can effectively improve the performance of the semantic segmentation model of point cloud.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Y. Wang, W. L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger, “Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving,” Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8445–8453, 2019.

  2. C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. J. Guibas, “Volumetric and multi-view CNNs for object classification on 3D data,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5648–5656, 2016.

  3. M. Gadelha, R. Wang, and S. Maji, “Multiresolution tree networks for 3D point cloud processing,” Proc. of the European Conference on Computer Vision, pp. 103–118, 2018.

  4. H. Y. Meng, L. Gao, Y. K. Lai, and D. Manocha, “Vv-Net: Voxel VAE net with group convolutions for point cloud segmentation,” Proc. of the IEEE/CVF International Conference on Computer Vision, pp. 8500–8508, 2019.

  5. H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller, “Multi-view convolutional neural networks for 3D shape recognition,” Proc. of the IEEE International Conference on Computer Vision, pp. 945–953, 2015.

  6. M. Tatarchenko, J. Park, V. Koltun, and Q. Y. Zhou, “Tangent convolutions for dense prediction in 3D,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3887–3896, 2018.

  7. Z. Han, H. Lu, Z. Liu, C.-M. Vong, Y.-S. Liu, M. Zwicker, J. Han, and C. L. P. Chen, “3D2SeqViews: Aggregating sequential views for 3D global feature learning by CNN with hierarchical attention aggregation,” IEEE Transactions on Image Processing, vol. 28, no. 8, pp. 3986–3999, August 2019.

    Article  MathSciNet  Google Scholar 

  8. C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep learning on point sets for 3D classification and segmentation,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660, July 2017.

  9. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep hierarchical feature learning on point sets in a metric space,” Advances in Neural Information Processing Systems, pp. 5099–5108, June 2017.

  10. Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph CNN for learning on point clouds,” ACM Transactions On Graphics, vol. 38, no. 5, pp. 1–12, 2019.

    Article  Google Scholar 

  11. H. Zhao, L. Jiang, C. W. Fu, and J. Jia, “PointWeb: Enhancing local neighborhood features for point cloud processing,” Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5565–5573, 2019.

  12. M. Jiang, Y. Wu, and C. Lu, “PointSIFT: A siftlike network module for 3D point cloud semantic segmentation,” arXiv:1807.00652, 2018. [Online]. Available: https://arxiv.org/abs/1807.00652

  13. S. Arshad, Q. Riaz, and M. M. Fraz, “DPRNet: Deep 3D point based residual network for semantic segmentation and classification of 3D point clouds,” IEEE Access, vol. 7, pp. 68892–68904, 2019.

    Article  Google Scholar 

  14. X.-T. Chen, Y. Li, H.-H. Fan, and R. Wang, “RGAM: A novel network architecture for 3D point cloud semantic segmentation in indoor scenes,” Information Sciences, vol. 571, pp. 87–103, 2021.

    Article  MathSciNet  Google Scholar 

  15. H. Woo, E. Kang, S. Wang, and K. H. Lee, “A new segmentation method for point cloud data,” International Journal of Machine Tools and Manufacture, vol. 42, no. 2, pp. 167–178, 2002.

    Article  Google Scholar 

  16. P. J. Besl and R. C. Jain, “Segmentation through variableorder surface fitting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 10, no. 2, pp. 167–192, 1988.

    Article  Google Scholar 

  17. J. Chen and B. Chen, “Architectural modeling from sparsely scanned range data,” International Journal of Computer Vision, vol. 78, no. 2–3, pp. 223–236, 2008.

    Article  Google Scholar 

  18. D. Maturana and S. Scherer, “Voxnet: A 3D convolutional neural network for real-time object recognition,” Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922–928, 2015.

  19. W. Wang, R. Yu, Q. Huang, and U. Neumann, “SGPN: Similarity group proposal network for 3D point cloud instance segmentation,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2569–2578, 2018.

  20. M. A. Uy, and G. H. Lee, “PointNetVLAD: Deep point cloud based retrieval for large-scale place recognition,” Proc. of the IEEE Connference on Computer Vision and Pattern Recognition, pp. 4470–4479, 2018.

  21. R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “NetVLAD: CNN architecture for weakly supervised place recognition,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5297–5307, 2016.

  22. L. Yu, X. Li, C.-W. Fu, D. Cohen-Or, and P.-A. Heng, “PU-Net: Point cloud upsampling network,” Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790–2799, 2016.

  23. Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen, “PointCNN: Convolution on x-transformed points,” Advances in Neural Information Processing Systems, pp. 820–830, 2018.

  24. F. Engelmann, T. Kontogianni, A. Hermans, and B. Leibe, “Exploring spatial context for 3D semantic segmentation of point clouds,” Proc. of the sIEEE International Conference on Computer Vision Workshops, pp. 716–724, 2017.

  25. A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. Villena-Martinez, and J. Garcia-Rodriguez, “A review on deep learning techniques applied to semantic segmentation,” arXiv preprint arXiv:1412.6980, 2014. [Online]. Available: https://arxiv.org/abs/1412.6980

  26. S. Qiu, S. Anwar, and H. Barnes, “Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion,” Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1757–1767, 2021.

  27. H. Zhao, L. Jiang, J. Jia, P. Torr, and V. Kolitun, “Point transformer,” arXiv preprint arXiv:2012.09164, 2020. [Online]. Available: https://arxiv.org/abs/2012.09164

  28. B. Yang, J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni, “Learning object bounding boxes for 3D instance segmentation on point clouds,” arXiv preprint arXiv:1906.01140, 2019. [Online]. Available: https://arxiv.org/abs/1906.01140

  29. L. Zhao and W. Tao, “JSNet: Joint instance and semantic segmentation of 3D point clouds,” Proc. of the AAAI Conference on Artificial Intelligence, vol. 34, no. 7, pp. 12951–12958, 2020.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Automation Engineering, Shanghai University of Electric Power, Shanghai, China

    Hui Chen

  2. Shanghai University of Electric Power, Shanghai, China

    Wanlou Chen

  3. State Grid Shijiazhuang Electric Power Supply Company, Hebei Province, 050051, China

    Yipeng Zuo

  4. Nari Research Instute, Nanjing, China

    Peng Xu

  5. College of Electrical Engineering, Qingdao University, Qingdao, 266071, China

    Zhonghua Hao

Authors
  1. Hui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wanlou Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yipeng Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhonghua Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhonghua Hao.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is supported by the National Natural Science Foundation of China (Grant No. 51705304), Natural Science Foundation of Shanghai (Grant No. 20ZR1421300), and sponsored by Shanghai Pujiang Program (Grant no.21PJD025), Project of the State Administration of foreign experts of the Ministry of science and technology (Grant No.DL2022013007L), and Shanghai Science and Technology Commission Program (Grant No.21DZ1207300).

Hui Chen received her B.Sc. degree in control and instrument specialty from Jiangsu University, Zhenjiang, China, in 2006, and her M.Sc. and Ph.D. degrees in control science and engineering from Shanghai University, Shanghai, China, in 2009 and 2014, respectively. She was a joint Ph.D. student with the Computation Department, Jacobs University Bremen, Bremen, Germany, from December 2011 to December 2012. She is currently an Associate Professor with the College of Automation Engineering, Shanghai University of Electric Power, Shanghai. Her research interests include pattern recognition, computer vision, and deep learning.

Wanlou Chen received his B.Sc. degree in building electricity and intelligentization from Anhui Jianzhu University, Hefei, China, in 2019. He is currently pursuing an M.Sc. degree in control engineering with Shanghai University of Electric Power, Shanghai, China. His research interests include deep learning and point cloud semantic segmentation.

Yipeng Zuo received his B.E. degree in electrical engineering and automation from Agricultural University of Hebei, Baoding, China, in 2018. He received a Master’s degree in electrical engineering from Shanghai University of Power, Shanghai, China, in 2021. He currently works at State Grid Shijiazhuang Electric Power Supply Company, Shijiazhuang, China. His research interests include deep learning and point cloud reconstruction.

Peng Xu received his B.Sc. degree in electrical engineering and automation from North China University of Water Resources and Electric Power, Zhengzhou, China, in 2017. He received a Master’s degree in electrical engineering from Shanghai University of Power, Shanghai, China, in 2020. He currently works at Nari Research Instute, Nanjing, China. His research interests include deep learning and point cloud semantic segmentation.

Zhonghua Hao received his B.Sc. degree in automation from Three gorges University, China in 2007. He received an M.Sc. degree in control theory and control engineering from Kunming University of Science and Technology, China, in 2010. And he received a Ph.D. degree in control theory and control engineering from Shanghai University, China, in 2016. Now he is an assistant professor in Qingdao University, China. His current research interests include digital image processing, data mining, machine learning, and pattern recognition.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, H., Chen, W., Zuo, Y. et al. PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution. Int. J. Control Autom. Syst. 20, 3321–3334 (2022). https://doi.org/10.1007/s12555-020-0571-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12555-020-0571-x

Keywords

Search

Navigation