Skip to main content
SpringerLink
Log in

Semantic segmentation of 3D LiDAR data using deep learning: a review of projection-based methods

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

LiDAR sensor is an active remote sensing sensor that is increasingly used to capture 3D information of real-world objects. Real-time decision-making applications such as autonomous driving heavily rely on 3D information to navigate an urban environment. LiDAR data processing is, however, very complex and resource-intensive. Deep learning on point cloud is a recent advancement that is aimed to extract 3D information. Deep learning implementations include procedures where raw points are fed to neural networks and converted to 3D voxels. Individual voxels are fed to 3D convolutional layers and techniques that transform the 3D points into 2D images and utilize the well-established 2D CNNs. Of these, the two former methods are majorly reviewed, while the projection-based methods are less reviewed although the technique is widely used in numerous applications. To fill the gap, this paper examines the existing literature on projection-based methods by detailing the recent progress made. Identifying the state-of-the-art methodology and summarizing the important interventions are among the significant tasks covered in this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Qi CR, Su H, Mo K, Guibas LJ (2017) PointNet: deep learning on point sets for 3D classification and segmentation. Proceedings - 30th IEEE conference on computer vision and pattern recognition, CVPR 2017 2017-Janua: 77–85. https://doi.org/10.1109/CVPR.2017.16

  2. Hänsch R, Weber T, Hellwich O (2014) comparison of 3D interest point detectors and descriptors for point cloud fusion. ISPRS annals of the photogrammetry, remote sensing and spatial Information Sciences II–3 (September): 57–64. https://doi.org/10.5194/isprsannals-ii-3-57-2014

  3. Liu W, Sun J, Li W, Hu T, Wang P (2019) Deep learning on point clouds and its application: a survey. Sensors (Switzerland) 19(19):1–22. https://doi.org/10.3390/s19194188

    Article  Google Scholar 

  4. Guo Y, Wang H, Hu Q, Liu H, Liu L, Bennamoun M (2019) Deep learning for 3D point clouds: a survey. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 43(12):4338–4364. https://doi.org/10.1109/tpami.2020.3005434

  5. Jiang D, Li G, Tan C, Huang L, Sun Y, Kong J (2021) Semantic segmentation for multiscale target based on object recognition using the improved faster-RCNN model. Futur Gener Comput Syst 123:94–104. https://doi.org/10.1016/j.future.2021.04.019

    Article  Google Scholar 

  6. Uy MA, Pham QH, Hua BS, Nguyen T, Yeung SK (2019) Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data. Proceedings of the IEEE International Conference on Computer Vision 2019-Octob: 1588–97. https://doi.org/10.1109/ICCV.2019.00167

  7. Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3D ShapeNets: a deep representation for volumetric shapes. Proceedings of the IEEE computer society conference on computer vision and pattern recognition 07-12-June: 1912–20. https://doi.org/10.1109/CVPR.2015.7298801

  8. Chang AX, Funkhouser T, Guibas L, Hanrahan P, Huang Q, Li Z, Savarese S et al (2015) ShapeNet: An Information-Rich 3D Model Repository. http://arxiv.org/abs/1512.03012. Accessed 18 Aug 2021

  9. Mo K, Zhu S, Chang AX, Yi L, Tripathi S, Guibas LJ, Su H. (2019) Partnet: a large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. Proceedings of the IEEE computer society conference on computer vision and pattern recognition 2019June: 909–18. https://doi.org/10.1109/CVPR.2019.00100

  10. Dai A, Chang AX, Savva M, Halber M, Funkhouser T, Nießner M (2017) ScanNet: richly-annotated 3D reconstructions of indoor scenes. Proceedings - 30th IEEE conference on computer vision and pattern recognition, CVPR 2017 2017-Janua: 2432–43. https://doi.org/10.1109/CVPR.2017.261

  11. Hackel T, Savinov N, Ladicky L, Wegner JD, Schindler K, Pollefeys M (2017) Semantic3D.Net: a new large-scale point cloud classification benchmark. ISPRS annals of the photogrammetry, remote sensing and spatial information sciences 4 (1W1): 91–98. https://doi.org/10.5194/isprs-annals-IV-1W1-91-2017

  12. Behley J, Garbade M (n.d.) SemanticKITTI : a dataset for semantic scene understanding of LiDAR sequences. no. iii. Accessed 18 Aug 2021

  13. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 3354–3361. https://doi.org/10.1109/CVPR.2012.6248074

  14. Munoz D, Bagnell JA, Vandapel N, Hebert M (2009) Contextual Classification with Functional Max-Margin Markov Networks. 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2009 2009 IEEE: 975–82. https://doi.org/10.1109/CVPRW.2009.5206590

  15. Rottensteiner F, Sohn G, Jung J, Gerke M, Baillard C, Benitez S, Breitkopf U (2012) THE ISPRS BENCHMARK on URBAN OBJECT CLASSIFICATION and 3D BUILDING RECONSTRUCTION. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 1 (September): 293–98. https://doi.org/10.5194/isprsannals-I-3-293-2012

  16. Serna A, Marcotegui B, Goulette F, Deschaud JE (2014) Paris-Rue-Madame Database: A 3D Mobile Laser Scanner Dataset for Benchmarking Urban Detection, Segmentation and Classification Methods. ICPRAM 2014 - Proceedings of the 3rd International Conference on Pattern Recognition Applications and Methods, 819–24. https://doi.org/10.5220/0004934808190824

  17. Vallet B, Brédif M, Serna A, Marcotegui B, Vallet B, Brédif M, Serna A, Marcotegui B, Terramo NP (2015) TerraMobilita / IQmulus Urban Point Cloud Analysis Benchmark To Cite This Version : HAL Id : Hal-01167995 TerraMobilita / IQmulus Urban Point Cloud Analysis

  18. Armeni I, Sener O, Zamir AR, Jiang H, Brilakis I, Fischer M, Savarese S (2016) 3D semantic parsing of large-scale indoor spaces. Proceedings of the IEEE computer society conference on computer vision and pattern recognition 2016-Decem: 1534–43. https://doi.org/10.1109/CVPR.2016.170

  19. Roynard X, Deschaud J-E, Goulette F, Roynard X (2018) Paris-Lille-3D : a large and HighQuality ground truth urban point cloud dataset for automatic segmentation and classification to cite this version : HAL id : Hal-01695873 Paris-Lille-3D : a large and high-quality ground truth urban point cloud dataset Fo

  20. Tan W, Qin N, Ma L, Li Y, Du J, Cai G, Yang K, Li J (2020) Toronto-3D: a large-scale Mobile LiDAR dataset for semantic segmentation of urban Roadways2211. IEEE computer society conference on computer vision and pattern recognition workshops 2020-June: 797–806. https://doi.org/10.1109/CVPRW50498.2020.00109

  21. Varney N, Asari VK, Graehling Q (2020) DALES: a large-scale aerial LiDAR data set for semantic segmentation. IEEE computer society conference on computer vision and pattern recognition workshops 2020-June: 717–26. https://doi.org/10.1109/CVPRW50498.2020.00101

  22. Bello SA, Yu S, Cheng W, Adam JM, Li J (2020) Review: deep learning on 3D point clouds. Remote Sens 12(11):1–34. https://doi.org/10.3390/rs12111729

    Article  Google Scholar 

  23. Yan X, Zheng C, Li Z, Wang S, Cui S (2020) PointasNL: robust point clouds processing using nonlocal neural networks with adaptive sampling. Proceedings of the IEEE computer society conference on computer vision and pattern recognition, 5588–97. https://doi.org/10.1109/CVPR42600.2020.00563

  24. Yang Z, Sun Y, Liu S, Jia J (2020) 3DSSD: point-based 3d single stage object detector. Proceedings of the IEEE computer society conference on computer vision and pattern recognition, 11037–45. https://doi.org/10.1109/CVPR42600.2020.01105

  25. Haoming Lu, Rey HS (2020) Deep learning for 3D point cloud understanding: a survey. ArXiv Preprint ArXiv

  26. Bello SA, Yu S, Wang C (2020) Review: deep learning on 3D point clouds. ArXiv

  27. Huang Q, Wang W, Neumann U (2018) Recurrent slice networks for 3D segmentation of point clouds. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, no 1: 2626–35. https://doi.org/10.1109/CVPR.2018.00278

  28. Li Y, Bu R, Di X (2018) PointCNN : convolution on X transformed points. no. NeurIPS

  29. Liu J, Ni B, Li C, Yang J, Tian Q (2019) Dynamic points agglomeration for hierarchical point sets learning. Proceedings of the IEEE international conference on computer vision 2019-Octob: 7545–54. https://doi.org/10.1109/ICCV.2019.00764

  30. Qi CR, Yi L, Su H, Guibas LJ (2017) PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Advances in Neural Information Processing Systems 2017-Decem: 5100–5109

  31. Wang Y, Sun Y, Liu Z, Sarma SE, Bronstein MM, Solomon JM (2018) Dynamic graph CNN for learning on point clouds," January. http://arxiv.org/abs/1801.07829

  32. Xie S, Liu S, Chen Z, Tu Z (2018) Attentional ShapeContextNet for Point Cloud Recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 4606–15. https://doi.org/10.1109/CVPR.2018.00484

  33. Zhao H, Jiang L, Fu CW, Jia J. (2019) Pointweb: Enhancing Local Neighborhood Features for Point Cloud Processing. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June: 5560–68. https://doi.org/10.1109/CVPR.2019.00571

  34. Jiang M, Wu Y, Zhao T, Zhao Z, Lu C (2018) Pointsift: A sift-like network module for 3d point cloud semantic segmentation. arXiv preprint arXiv:1807.00652. http://arxiv.org/abs/1807.00652

  35. Choy C, Gwak J, Savarese S (2019) 4D Spatio-temporal Convnets: Minkowski convolutional neural networks. Proceedings of the IEEE computer society conference on computer vision and pattern recognition 2019 June: 3070–79. https://doi.org/10.1109/CVPR.2019.00319

  36. Graham B, Engelcke M, Van Der Maaten L (2018) 3D semantic segmentation with submanifold sparse convolutional networks. Proceedings of the IEEE computer society conference on computer vision and pattern recognition, 9224–32. https://doi.org/10.1109/CVPR.2018.00961

  37. Meng HY, Lin G, Lai YK, Manocha D (2019) VVNet: Voxel vae net with group convolutions for point cloud segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 8500–8508. https://doi.org/10.1109/ICCV.2019.00859

  38. Tchapmi L, Choy C, Armeni I, Gwak J, Savarese S (2018) SEGCloud: semantic segmentation of 3D point clouds. proceedings - 2017 international conference on 3D vision, 3DV 2017, 537–47. https://doi.org/10.1109/3DV.2017.00067

  39. Zhou Y, Tuzel O (2018) VoxelNet: end-to-end learning for point cloud based 3D object detection. Proceedings of the IEEE computer society conference on computer vision and pattern recognition, 4490–99. https://doi.org/10.1109/CVPR.2018.00472

  40. Milioto A, Vizzo I, Behley J, Stachniss C (2019) RangeNet ++: fast and accurate LiDAR semantic segmentation. IEEE international conference on intelligent robots and systems, no. i: 4213–20. https://doi.org/10.1109/IROS40897.2019.8967762

  41. Radi H, Ali W (2019) VolMap: A Real-Time Model for Semantic Segmentation of a LiDAR Surrounding View. http://arxiv.org/abs/1906.11873

  42. Wu B, Wan A, Yue X, Keutzer K (2018) SqueezeSeg: convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D LiDAR point cloud. proceedings - IEEE international conference on robotics and automation, 1887–93. https://doi.org/10.1109/ICRA.2018.8462926

  43. Wu B, Zhou X, Zhao S, Yue X, Keutzer K (2018) SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud. 4376–82. http://arxiv.org/abs/1809.08495

  44. Zhang Y, Zhou Z, David P, Yue X, Xi Z, Gong B (2020) PolarNet : an improved grid representation for online LiDAR point clouds semantic segmentation

  45. Kuffer M, Pfeffer K, Sliuzas R, Baud I (2016) Extraction of slum areas from VHR imagery using GLCM variance. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 9(5):1830–1840. https://doi.org/10.1109/JSTARS.2016.2538563

    Article  Google Scholar 

  46. He T, Huang H, Yi L, Zhou Y, Wu C, Wang J, Soatto S (2019) Geonet: Deep geodesic networks for point cloud analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6888–6897. https://doi.org/10.1109/CVPR.2019.00705

  47. Zamorski M, Zięba M, Klukowski P, Nowak R, Kurach K, Stokowiec W, Trzciński T (2020) Adversarial autoencoders for compact representations of 3D point clouds. Comput Vis Image Underst 193:102921. https://doi.org/10.1016/j.cviu.2020.102921

    Article  Google Scholar 

  48. Remelli E, Baque P, Fua P (2019) NeuralSampler: Euclidean Point Cloud Auto-Encoder and Sampler. http://arxiv.org/abs/1901.09394

  49. Ku T, Veltkamp RC, Boom B, Duque-Arias D, VelascoForero S, Deschaud JE, Goulette F et al (2020) SHREC 2020: 3D point cloud semantic segmentation for street scenes. Computers and Graphics (Pergamon) 93:13–24. https://doi.org/10.1016/j.cag.2020.09.006

    Article  Google Scholar 

  50. Beltrán J, Guindel C, Moreno FM, Cruzado D, García F, De La Escalera A (2018) BirdNet: A 3D Object Detection Framework from LiDAR Information. IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC 2018-Novem: 3517–23. https://doi.org/10.1109/ITSC.2018.8569311

  51. Alnaggar A, Yara MA, Amer K, ElHelw M (2021) Multi projection fusion for real-time semantic segmentation of 3d lidar point clouds. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 1800–1809. https://doi.org/10.1109/wacv48630.2021.00184

  52. Maturana D, Scherer S (2015) VoxNet: a 3D convolutional neural network for real-time object recognition. IEEE international conference on intelligent robots and systems 2015-Decem: 922–28. https://doi.org/10.1109/IROS.2015.7353481

  53. Xiao A, Yang X, Lu S, Guan D, Huang J (2021) ISPRS journal of photogrammetry and remote sensing FPS-net : a convolutional fusion network for large-scale LiDAR point cloud segmentation. ISPRS J Photogramm Remote Sens 176 (September 2020):237–249. https://doi.org/10.1016/j.isprsjprs.2021.04.011

    Article  Google Scholar 

  54. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2017) 50 X FEWER PARAMETERS AND < 0. 5MB MODEL SIZE. 1–13

  55. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision Meets Robotics: The KITTI Dataset. The International Journal of Robotics Research, no. October: 1–6 32:1231–1237

    Article  Google Scholar 

  56. Geyer J, Ricou X, Chung AS, Maximilian M, Dorn S, Martin J, Sturm M, Oelker M, Ag A (n.d.) A2D2 : Audi Autonomous Driving Dataset

  57. Aksoy EE, Baci S, Cavdar S (2020) SalsaNet: Fast Road and Vehicle Segmentation in LiDAR Point Clouds for Autonomous Driving. IEEE Intelligent Vehicles Symposium, Proceedings, no. Iv: 926–32. https://doi.org/10.1109/IV47402.2020.9304694

  58. Redmon J, Farhadi A (2018) YOLOv3: An Incremental Improvement. http://arxiv.org/abs/1804.02767

  59. Wang Y (n.d.) PointSeg : real-time semantic segmentation based on 3D LiDAR point cloud

  60. He K, Sun J (2016) Deep residual learning for image recognition. https://doi.org/10.1109/CVPR.2016.90

  61. Teichmann M, Weber M, Marius Z, Cipolla R, Urtasun R. (n.d.) MultiNet : Real-Time Joint Semantic Reasoning for Autonomous Driving

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, University of Petroleum and Energy Studies, Dehradun, India

    Alok Jhaldiyal & Navendu Chaudhary

  2. Symbiosis Institute of Geoinformatics, Symbiosis International (Deemed University), Pune, India

    Navendu Chaudhary

Authors
  1. Alok Jhaldiyal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Navendu Chaudhary
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alok Jhaldiyal.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jhaldiyal, A., Chaudhary, N. Semantic segmentation of 3D LiDAR data using deep learning: a review of projection-based methods. Appl Intell 53, 6844–6855 (2023). https://doi.org/10.1007/s10489-022-03930-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03930-5

Keywords

Search

Navigation