Skip to main content
Book cover

European Conference on Computer Vision

ECCV 2022: Computer Vision – ECCV 2022 pp 318–334Cite as

Open-world Semantic Segmentation for LIDAR Point Clouds

  • 912 Accesses

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13698)

Abstract

Current methods for LIDAR semantic segmentation are not robust enough for real-world applications, e.g., autonomous driving, since it is closed-set and static. The closed-set assumption makes the network only able to output labels of trained classes, even for objects never seen before, while a static network cannot update its knowledge base according to what it has seen. Therefore, in this work, we propose the open-world semantic segmentation task for LIDAR point clouds, which aims to 1) identify both old and novel classes using open-set semantic segmentation, and 2) gradually incorporate novel objects into the existing knowledge base using incremental learning without forgetting old classes. For this purpose, we propose a REdundAncy cLassifier (REAL) framework to provide a general architecture for both the open-set semantic segmentation and incremental learning problems. The experimental results show that REAL can simultaneously achieves state-of-the-art performance in the open-set semantic segmentation task on the SemanticKITTI and nuScenes datasets, and alleviate the catastrophic forgetting problem with a large margin during incremental learning.

Keywords

  • Open-world semantic segmentation
  • LIDAR point clouds
  • Open-set semantic segmentation
  • Incremental learning

code is available at: https://github.com/Jun-CEN/Open_world_3D_semantic_segmentation

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-19839-7_19
  • Chapter length: 17 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-031-19839-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

References

  1. Baur, C., Wiestler, B., Albarqouni, S., Navab, N.: Deep autoencoding models for unsupervised anomaly segmentation in brain MR images. In: International MICCAI Brainlesion Workshop (2018)

    Google Scholar 

  2. Behley, J., et al.: Towards 3D LiDAR-based semantic scene understanding of 3D point cloud sequences: The SemanticKITTI Dataset. In: The International Journal on Robotics Research (2021)

    Google Scholar 

  3. Behley, J., et al.: A Dataset for Semantic Scene Understanding of LiDAR Sequences. In: ICCV (2019)

    Google Scholar 

  4. Bendale, A., Boult, T.: Towards open world recognition. In: CVPR (2015)

    Google Scholar 

  5. Bozhinoski, D., Di Ruscio, D., Malavolta, I., Pelliccione, P., Crnkovic, I.: Safety for mobile robotic systems: A systematic mapping study from a software engineering perspective. J. Syst. Softw. 151 150–179 (2019)

    Google Scholar 

  6. Caesar, H., et al.: nuscenes: A multimodal dataset for autonomous driving. In: CVPR (2020)

    Google Scholar 

  7. Cen, J., Yun, P., Cai, J., Wang, M.Y., Liu, M.: Deep metric learning for open world semantic segmentation. In: ICCV (2021)

    Google Scholar 

  8. Cermelli, F., Mancini, M., Bulo, S.R., Ricci, E., Caputo, B.: Modeling the background for incremental learning in semantic segmentation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 9230–9239 (2020). https://doi.org/10.1109/CVPR42600.2020.00925

  9. Cermelli, F., Mancini, M., Bulo, S.R., Ricci, E., Caputo, B.: Modeling the background for incremental learning in semantic segmentation. In: CVPR (2020)

    Google Scholar 

  10. Cheng, R., Razani, R., Taghavi, E., Li, E., Liu, B.: 2–s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In: CVPR (2021)

    Google Scholar 

  11. Delange, M., et al.: A continual learning survey: Defying forgetting in classification tasks. IEEE Trans. Pattern Analysis Mach. Intell. 44 3366–3385 (2021)

    Google Scholar 

  12. Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: ICML (2016)

    Google Scholar 

  13. Geiger, A., Lenz, P., Urtasun, R.: Are We Ready for Autonomous Driving? CVPR, The KITTI Vision Benchmark Suite. In (2012)

    Google Scholar 

  14. Hendrycks, D., Basart, S., Mazeika, M., Mostajabi, M., Steinhardt, J., Song, D.: Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132 (2019)

  15. Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: ICLR (2017)

    Google Scholar 

  16. Hu, Q., et al.: Learning semantic segmentation of large-scale point clouds with random sampling. IEEE Trans. Pattern Analysis Mach. Intell. textbf44(11), 8338–8354 (2021)

    Google Scholar 

  17. Joseph, K.J., Khan, S., Khan, F.S., Balasubramanian, V.N.: Towards open world object detection. In: CVPR (2021)

    Google Scholar 

  18. Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In: NeurIPS (2017)

    Google Scholar 

  19. Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: NeurIPS (2017)

    Google Scholar 

  20. Li, D., Tasci, S., Ghosh, S., Zhu, J., Zhang, J.T., Heck, L.: Rilod: near real-time incremental learning for object detection at the edge. In: Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pp. 113–126 (2019)

    Google Scholar 

  21. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Analysis Mach. Intell. 40(12), 2935–2947 (2018)

    Google Scholar 

  22. Lis, K., Nakka, K., Fua, P., Salzmann, M.: Detecting the unexpected via image resynthesis. In: ICCV (2019)

    Google Scholar 

  23. Liu, L., Kuang, Z., Chen, Y., Xue, J., Yang, W., Zhang, W.: Incdet: In defense of elastic weight consolidation for incremental object detection. IEEE Transactions on Neural Networks and Learning Systems, pp. 1–14 (2020)

    Google Scholar 

  24. MacKay, D.J.: Bayesian neural networks and density networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment (1995)

    Google Scholar 

  25. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of learning and motivation (1989)

    Google Scholar 

  26. Milioto, A., Vizzo, I., Behley, J., Stachniss, C.: Rangenet++: Fast and accurate lidar semantic segmentation. In: IROS (2019)

    Google Scholar 

  27. Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR (2019)

    Google Scholar 

  28. Peng, C., Zhao, K., Lovell, B.: Faster ilod: Incremental learning for object detectors based on faster rcnn. arXiv preprint arXiv:2003.03901 (2020)

  29. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: CVPR (2017)

    Google Scholar 

  30. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413 (2017)

  31. Rannen, A., Aljundi, R., Blaschko, M.B., Tuytelaars, T.: Encoder based lifelong learning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1320–1328 (2017)

    Google Scholar 

  32. Shmelkov, K., Schmid, C., Alahari, K.: Incremental learning of object detectors without catastrophic forgetting. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3420–3429 (2017)

    Google Scholar 

  33. Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: Flexible and deformable convolution for point clouds. In: ICCV (2019)

    Google Scholar 

  34. Wang, Y., Li, B., Che, T., Zhou, K., Liu, Z., Li, D.: Energy-based open-world uncertainty modeling for confidence calibration. In: ICCV (2021)

    Google Scholar 

  35. Wang, Y., Shi, T., Yun, P., Tai, L., Liu, M.: Pointseg: Real-time semantic segmentation based on 3d lidar point cloud. arXiv preprint arXiv:1807.06288 (2018)

  36. Welling, M., Teh, Y.W.: Bayesian learning via stochastic gradient langevin dynamics. In: ICML, pp. 681–688 (2011)

    Google Scholar 

  37. Wu, B., Zhou, X., Zhao, S., Yue, X., Keutzer, K.: Squeezesegv 2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud. In: ICRA (2019)

    Google Scholar 

  38. Wu, W., Qi, Z., Fuxin, L.: Pointconv: Deep convolutional networks on 3d point clouds. In: CVPR (2019)

    Google Scholar 

  39. Xia, Y., Zhang, Y., Liu, F., Shen, W., Yuille, A.L.: Synthesize then compare: detecting failures and anomalies for semantic segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 145–161. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_9

    CrossRef  Google Scholar 

  40. Yun, P., Cen, J., Liu, M.: Conflicts between likelihood and knowledge distillation in task incremental learning for 3d object detection. In: 3DV (2021)

    Google Scholar 

  41. Yun, P., Liu, Y., Liu, M.: In defense of knowledge distillation for task incremental learning and its application in 3D object detection. IEEE Robot. Autom. Lett. 6(2), 2012–2019 (2021). https://doi.org/10.1109/LRA.2021.3060417

    CrossRef  MathSciNet  Google Scholar 

  42. Zhang, Y., et al.: Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In: CVPR (2020)

    Google Scholar 

  43. Zhou, D.W., Ye, H.J., Zhan, D.C.: Learning placeholders for open-set recognition. In: CVPR (2021)

    Google Scholar 

  44. Zhu, X., et al.: Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In: CVPR (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The Hong Kong University of Science and Technology, Hong Kong, China

    Jun Cen, Peng Yun, Junhao Cai, Di Luan, Ming Liu & Michael Yu Wang

  2. Alibaba Group, Hangzhou, China

    Shiwei Zhang & Mingqian Tang

Authors
  1. Jun Cen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Yun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shiwei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junhao Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Di Luan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mingqian Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Michael Yu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiwei Zhang .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Cen, J. et al. (2022). Open-world Semantic Segmentation for LIDAR Point Clouds. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13698. Springer, Cham. https://doi.org/10.1007/978-3-031-19839-7_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19839-7_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19838-0

  • Online ISBN: 978-3-031-19839-7

  • eBook Packages: Computer ScienceComputer Science (R0)