Advertisement

EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection

Conference paper
  • 709 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12360)

Abstract

In this paper, we aim at addressing two critical issues in the 3D detection task, including the exploitation of multiple sensors (namely LiDAR point cloud and camera image), as well as the inconsistency between the localization and classification confidence. To this end, we propose a novel fusion module to enhance the point features with semantic image features in a point-wise manner without any image annotations. Besides, a consistency enforcing loss is employed to explicitly encourage the consistency of both the localization and classification confidence. We design an end-to-end learnable framework named EPNet to integrate these two components. Extensive experiments on the KITTI and SUN-RGBD datasets demonstrate the superiority of EPNet over the state-of-the-art methods. Codes and models are available at: https://github.com/happinesslz/EPNet.

Keywords

3D object detection Point cloud Multiple sensors 

Notes

Acknowledgement

This work was supported by National Key R&D Program of China (No. 2018YFB 1004600), Xiang Bai was supported by the National Program for Support of Top-notch Young Professionals and the Program for HUST Academic Frontier Youth Team 2017QYTD08.

Supplementary material

504470_1_En_3_MOESM1_ESM.pdf (2.2 mb)
Supplementary material 1 (pdf 2207 KB)

References

  1. 1.
    Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., Urtasun, R.: Monocular 3D object detection for autonomous driving. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  2. 2.
    Chen, X., Kundu, K., Zhu, Y., Ma, H., Fidler, S., Urtasun, R.: 3D object proposals using stereo imagery for accurate object class detection. IEEE Trans. Pattern Anal. Mach. Intell. 40(5), 1259–1272 (2017)CrossRefGoogle Scholar
  3. 3.
    Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3D object detection network for autonomous driving. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  4. 4.
    Chen, Y., Liu, S., Shen, X., Jia, J.: Fast point R-CNN. In: Proceedings of IEEE International Conference on Computer Vision (2019)Google Scholar
  5. 5.
    Du, X., Ang, M.H., Karaman, S., Rus, D.: A general pipeline for 3D detection of vehicles. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3194–3200 (2018).  https://doi.org/10.1109/ICRA.2018.8461232
  6. 6.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving. In: Proceedings of IEEE International Conference on Computer Vision and Pattern RecognitionGoogle Scholar
  7. 7.
    Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)Google Scholar
  8. 8.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of International Conference on Machine Learning (2015)Google Scholar
  9. 9.
    Jiang, B., Luo, R., Mao, J., Xiao, T., Jiang, Y.: Acquisition of localization confidence for accurate object detection. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 816–832. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01264-9_48CrossRefGoogle Scholar
  10. 10.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2014)Google Scholar
  11. 11.
    Ku, J., Mozifian, M., Lee, J., Harakeh, A., Waslander, S.L.: Joint 3D proposal generation and object detection from view aggregation. In: IROS, pp. 1–8. IEEE (2018)Google Scholar
  12. 12.
    Ku, J., Pon, A.D., Waslander, S.L.: Monocular 3D object detection leveraging accurate proposals and shape reconstruction. In: CVPR (2019)Google Scholar
  13. 13.
    Lahoud, J., Ghanem, B.: 2D-driven 3D object detection in RGB-D images. In: Proceedings of IEEE International Conference on Computer Vision (2017)Google Scholar
  14. 14.
    Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: fast encoders for object detection from point clouds. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  15. 15.
    Li, B., Ouyang, W., Sheng, L., Zeng, X., Wang, X.: GS3D: an efficient 3D object detection framework for autonomous driving. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  16. 16.
    Li, P., Chen, X., Shen, S.: Stereo R-CNN based 3D object detection for autonomous driving. In: CVPR (2019)Google Scholar
  17. 17.
    Liang, M., Yang, B., Chen, Y., Hu, R., Urtasun, R.: Multi-task multi-sensor fusion for 3D object detection. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  18. 18.
    Liang, M., Yang, B., Wang, S., Urtasun, R.: Deep continuous fusion for multi-sensor 3D object detection. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 663–678. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01270-0_39CrossRefGoogle Scholar
  19. 19.
    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of IEEE International Conference on Computer Vision (2017)Google Scholar
  20. 20.
    Liu, L., Lu, J., Xu, C., Tian, Q., Zhou, J.: Deep fitting degree scoring network for monocular 3D object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1057–1066 (2019)Google Scholar
  21. 21.
    Liu, Z., Zhao, X., Huang, T., Hu, R., Zhou, Y., Bai, X.: Tanet: robust 3D object detection from point clouds with triple attention. In: AAAI, pp. 11677–11684 (2020)Google Scholar
  22. 22.
    Luo, W., Yang, B., Urtasun, R.: Fast and furious: real time end-to-end 3D detection, tracking and motion forecasting with a single convolutional net. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  23. 23.
    Ma, X., Wang, Z., Li, H., Zhang, P., Ouyang, W., Fan, X.: Accurate monocular object detection via color-embedded 3D reconstruction for autonomous driving. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  24. 24.
    Meyer, G.P., Laddha, A., Kee, E., Vallespi-Gonzalez, C., Wellington, C.K.: LaserNet: an efficient probabilistic 3D object detector for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  25. 25.
    Mousavian, A., Anguelov, D., Flynn, J., Kosecka, J.: 3D bounding box estimation using deep learning and geometry. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  26. 26.
    Qi, C.R., Litany, O., He, K., Guibas, L.J.: Deep hough voting for 3D object detection in point clouds. In: Proceedings of IEEE International Conference on Computer Vision (2019)Google Scholar
  27. 27.
    Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum PointNets for 3D object detection from RGB-D data. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  28. 28.
    Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, pp. 5099–5108 (2017)Google Scholar
  29. 29.
    Qin, Z., Wang, J., Lu, Y.: MonoGRNet: a geometric reasoning network for 3D object localization. In: The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19) (2019)Google Scholar
  30. 30.
    Ren, Z., Sudderth, E.B.: Three-dimensional object detection and layout prediction using clouds of oriented gradients. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  31. 31.
    Shi, S., Wang, X., Li, H.: PointRCNN: 3D object proposal generation and detection from point cloud. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  32. 32.
    Simonelli, A., Bulò, S.R.R., Porzi, L., López-Antequera, M., Kontschieder, P.: Disentangling monocular 3D object detection. arXiv preprint arXiv:1905.12365 (2019)
  33. 33.
    Song, S., Lichtenberg, S.P., Xiao, J.: Sun RGB-D: a RGB-D scene understanding benchmark suite. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  34. 34.
    Song, S., Xiao, J.: Deep sliding shapes for amodal 3D object detection in RGB-D images. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  35. 35.
    Wang, Y., Chao, W.L., Garg, D., Hariharan, B., Campbell, M., Weinberger, K.: Pseudo-LIDAR from visual depth estimation: bridging the gap in 3D object detection for autonomous driving. In: CVPR (2019)Google Scholar
  36. 36.
    Xu, B., Chen, Z.: Multi-level fusion based 3D object detection from monocular images. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  37. 37.
    Xu, D., Anguelov, D., Jain, A.: PointFusion: deep sensor fusion for 3D bounding box estimation. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  38. 38.
    Yan, Y., Mao, Y., Li, B.: Second: sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)CrossRefGoogle Scholar
  39. 39.
    Yang, B., Luo, W., Urtasun, R.: PIXOR: real-time 3D object detection from point clouds. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  40. 40.
    Yang, Z., Sun, Y., Liu, S., Shen, X., Jia, J.: STD: sparse-to-dense 3D object detector for point cloud. In: ICCV (2019). http://arxiv.org/abs/1907.10471
  41. 41.
    Yu, J., Jiang, Y., Wang, Z., Cao, Z., Huang, T.: UnitBox: an advanced object detection network. In: Proceedings of the 24th ACM International Conference on Multimedia (2016)Google Scholar
  42. 42.
    Zhao, X., Liu, Z., Hu, R., Huang, K.: 3D object detection using scale invariant and feature reweighting networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 9267–9274 (2019)Google Scholar
  43. 43.
    Zhou, Y., Tuzel, O.: VoxelNet: end-to-end learning for point cloud based 3D object detection. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Huazhong University of Science and TechnologyWuhanChina

Personalised recommendations