Abstract
In recent years, exploring integrated heterogeneous features architecture has become one of the hot spots in 3D point cloud understanding. However, the efficacy of end-to-end training in enhancing the precision of multi-view fusion for point cloud segmentation and its flexibility remain limited. Furthermore, it is worth highlighting that prior studies have consistently employed encoder-decoder architectures, predominantly emphasizing the refinement of encoder designs, while relatively neglecting the significance of decoders which ultimately leads to increased computing costs. In this study, we present our novel TSTD model, which exhibits remarkable efficacy and efficiency, addressing the constraints encountered in prior research. Diverging from existing approaches that exclusively employ either geometry or RGB data for semantic segmentation, our proposed methodology incorporates both modalities within a unified, two-stage network architecture. This integrative approach enables the effective fusion of heterogeneous data features, leading to notable enhancements in semantic segmentation outcomes. Moreover, we have devised an innovative and efficient decoder utilizing a lightweight transformer module. This novel design further enhances the decoding process, resulting in improved performance and effectiveness. The performance of our model, TSTD, demonstrates strong results with an mIoU of 72.5\(\%\) on the ScanNet v2 validation set and 67.6\(\%\) on the test set. Notably, TSTD outperforms the current leading cross-modal point cloud semantic segmentation method CMX by a significant margin of 6.3\(\%\) mIoU. It also reduces Flops by 31.4\(\%\) compared to Point Transformer. Extensive experiments confirm that TSTD achieves state-of-the-art performance in cross-model point cloud semantic segmentation.
This work was supported in part by the Open Fund of Hubei Luojia Laboratory under Grant 220100053 and in part by the Science and Technology Major Project of Hubei Province under Grant 2021AAA010.
Supplementary Information. The supplementary materials are available at https://github.com/Whu-gaozhao/TSTD-Supp.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chiang, H.Y., et al.: A unified point-based framework for 3d segmentation. In: 2019 International Conference on 3D Vision (3DV), pp. 155–163. IEEE (2019)
Choy, C., et al.: 4d spatio-temporal convnets: minkowski convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3075–3084 (2019)
Dai, A., et al.: Scannet: richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5828–5839 (2017)
Dai, A., Nießner, M.: 3dmv: joint 3d-multi-view prediction for 3d semantic scene segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 452–468 (2018)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Graham, B., et al.: 3d semantic segmentation with submanifold sparse convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9224–9232 (2018)
Guo, M.H., et al.: Pct: point cloud transformer. Comput. Visual Media 7, 187–199 (2021)
Hazirbas, C., Ma, L., Domokos, C., Cremers, D.: FuseNet: incorporating depth into semantic segmentation via fusion-based CNN architecture. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10111, pp. 213–228. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54181-5_14
Hu, Q., et al.: Randla-net: efficient semantic segmentation of large-scale point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11108–11117 (2020)
Lei, H., et al.: Seggcn: efficient 3d point cloud segmentation with fuzzy spherical kernel. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11611–11620 (2020)
Li, Y., et al.: Pointcnn: convolution on x-transformed points. In: Advances in neural information processing systems 31 (2018)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Menini, D., Kumar, S., et al.: A real-time online learning framework for joint 3d reconstruction and semantic segmentation of indoor scenes. IEEE Robot. Autom. Lett. 7(2), 1332–1339 (2021)
Narita, G., et al.: Panopticfusion: online volumetric semantic mapping at the level of stuff and things. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4205–4212. IEEE (2019)
Qi, C.R., et al.: Pointnet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)
Qi, C.R., et al.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems 30 (2017)
Qian, Y., Deng, L., et al.: Gated-residual block for semantic segmentation using rgb-d data. IEEE Trans. Intell. Transp. Syst. 23(8), 11836–11844 (2021)
Shi, B., et al.: A transformer-based decoder for semantic segmentation with multi-level context mining. In: ECCV 2022, Part XXVIII, pp. 624–639. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19815-1_36
Shi, W., Xu, J., et al.: Rgb-d semantic segmentation and label-oriented voxelgrid fusion for accurate 3d semantic mapping. IEEE Trans. Circuits Syst. Video Technol. 32(1), 183–197 (2021)
Shi, W., Zhu, D., et al.: Multilevel cross-aware rgbd indoor semantic segmentation for bionic binocular robot. IEEE Trans. Med. Robot. Bionics 2(3), 382–390 (2020)
Thomas, H., et al.: Kpconv: flexible and deformable convolution for point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6411–6420 (2019)
Valada, A., et al.: Self-supervised model adaptation for multimodal semantic segmentation. Int. J. Comput. Vision 128(5), 1239–1285 (2020)
Wang, J., Sun, B., Lu, Y.: Mvpnet: multi-view point regression networks for 3d object reconstruction from a single image. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8949–8956 (2019)
Wu, W., et al.: Pointconv: deep convolutional networks on 3d point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9621–9630 (2019)
Xie, E., et al.: Segformer: simple and efficient design for semantic segmentation with transformers. Adv. Neural. Inf. Process. Syst. 34, 12077–12090 (2021)
Yan, L., et al.: Efficient depth fusion transformer for aerial image semantic segmentation. Remote Sens. 14(5), 1294 (2022)
Yan, X., et al.: Pointasnl: robust point clouds processing using nonlocal neural networks with adaptive sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5589–5598 (2020)
Zhang, J., et al.: Cmx: cross-modal fusion for rgb-x semantic segmentation with transformers. arXiv preprint arXiv:2203.04838 (2022)
Zhao, H., et al.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)
Zhao, H., et al.: Point transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259–16268 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Gao, Z., Yan, L., Xie, H., Wei, P., Wu, H., Wang, J. (2024). TSTD:A Cross-modal Two Stages Network with New Trans-decoder for Point Cloud Semantic Segmentation. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14432. Springer, Singapore. https://doi.org/10.1007/978-981-99-8543-2_11
Download citation
DOI: https://doi.org/10.1007/978-981-99-8543-2_11
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8542-5
Online ISBN: 978-981-99-8543-2
eBook Packages: Computer ScienceComputer Science (R0)