Advertisement

Feature Map Transformation for Multi-sensor Fusion in Object Detection Networks for Autonomous Driving

Conference paper
First Online:
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 944)

Abstract

We present a general framework for fusing pre-trained object detection networks for multiple sensor modalities in autonomous cars at an intermediate stage. The key innovation is an autoencoder-inspired Transformer module which transforms perspective as well as feature activation characteristics from one sensor modality to another. Transformed feature maps can be combined with those of a modality-native feature extractor to enhance performance and reliability through a simple fusion scheme. Our approach is not limited to specific object detection network types. Compared to other methods, our framework allows fusion of pre-trained object detection networks and fuses sensor modalities at a single stage, resulting in a modular and traceable architecture. We show effectiveness of the proposed scheme by fusing camera and Lidar information to detect objects using our own as well as the KITTI dataset.

Keywords

Autonomous driving Perception Sensor fusion Object detection Lidar 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3D object detection network for autonomous driving. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6526–6534 (2017).  https://doi.org/10.1109/CVPR.2017.691. http://doi.ieeecomputersociety.org/10.1109/CVPR.2017.691
  2. 2.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010).  https://doi.org/10.1007/s11263-009-0275-4, http://link.springer.com/10.1007/s11263-009-0275-4CrossRefGoogle Scholar
  3. 3.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The kitti vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354–3361. IEEE (2012). http://ieeexplore.ieee.org/abstract/document/6248074/
  4. 4.
    Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015). http://www.cv-foundation.org/openaccess/content_iccv_2015/html/Girshick_Fast_R-CNN_ICCV_2015_paper.html
  5. 5.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016). https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html
  6. 6.
    Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., Murphy, K.: Speed/accuracy trade-offs for modern convolutional object detectors. arXiv:1611.10012 [cs] (2016)
  7. 7.
    Ku, J., Mozifian, M., Lee, J., Harakeh, A., Waslander, S.: Joint 3D Proposal Generation and Object Detection from View Aggregation. arXiv:1712.02294 [cs] (2017)
  8. 8.
    Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum PointNets for 3D object detection from RGB-D data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 918–927 (2018). http://openaccess.thecvf.com/content_cvpr_2018/html/Qi_Frustum_PointNets_for_CVPR_2018_paper.html
  9. 9.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. arXiv:1506.01497 [cs] (2015)
  10. 10.
    Simon, M., Milz, S., Amende, K., Gross, H.M.: Complex-YOLO: Real-time 3D Object Detection on Point Clouds. arXiv:1803.06199 [cs] (2018)
  11. 11.
    Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 [cs] (2014)
  12. 12.
    Wang, Z., Zhan, W., Tomizuka, M.: Fusing Bird View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection (2017). https://arxiv.org/abs/1711.06703
  13. 13.
    Yan, X., Yang, J., Yumer, E., Guo, Y., Lee, H.: Perspective transformer nets: learning single-view 3D object reconstruction without 3D supervision. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 1696–1704. Curran Associates, Inc. (2016)Google Scholar
  14. 14.
    Zhou, Y., Tuzel, O.: VoxelNet: end-to-end learning for point cloud based 3D object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4490–4499 (2018). http://openaccess.thecvf.com/content_cvpr_2018/html/Zhou_VoxelNet_End-to-End_Learning_CVPR_2018_paper.html

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Development Sensor Data Fusion and LocalisationAUDI AGIngolstadtGermany
  2. 2.Department of Computer SciencesTechnical University ChemnitzChemnitzGermany

Personalised recommendations

Cite paper

Buy options