Skip to main content
SpringerLink
Log in

Visual image and radio signal fusion identification based on convolutional neural networks

  • Research Article
  • Published:
Journal of Optics Aims and scope Submit manuscript

Abstract

Target detection and identification based on heterogeneous data fusion is significant when performance is restricted by a sensor. Due to UAV with the characteristic of small size, identification is difficult by visual image when it is far away. Hence, we fused the information of radio signal and image for the recognition when it was too far to distinguish the type of UAV. To validate the effectiveness of data fusion, we used the deep learning framework of faster RCNN with different feature extraction network in this study. First, we constructed three datasets with various distance to verify the shortage of object recognition based on visual image. Subsequently, visual image and radio signal fusion identification based on faster RCNN is proposed. Finally, the improvement of performance was confirmed by contrast experiments. The proposed method can enhance the accuracy of identification and has faster inference with similar accuracy compared with deeper feature extraction network, which promotes the practical development of target detection and identification based on deep learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. E.F. Nakamura, A.A.F. Loureiro, A.C. Frery, Information fusion for wireless sensor networks: methods, models, and classifications. ACM Comput. Surv. (CSUR) 39(3), 9 (2007)

    Article  Google Scholar 

  2. S. Sun, A survey of multi-view machine learning. Neural Comput. Appl. 23(7–8), 2031–2038 (2013)

    Article  Google Scholar 

  3. C.M. Bishop, Pattern Recognition and Machine Learning (Springer, Berlin, 2006)

    MATH  Google Scholar 

  4. R. Kiros, R. Salakhutdinov, R. Zemel, Multimodal neural language models, in International Conference on Machine Learning (2014), pp. 595–603

  5. J. Ngiam, A. Khosla, M. Kim et al., Multimodal Deep Learning (ICML, IN, 2011)

    Google Scholar 

  6. J. Redmon, A. Farhadi, YOLO9000: better, faster, stronger, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 7263–7271

  7. A. Asvadi, L. Garrote, C. Premebida et al., Multimodal vehicle detection: fusing 3D-LIDAR and color camera data. Pattern Recognit. Lett. 115, 20–29 (2018)

    Article  ADS  Google Scholar 

  8. R. Niu, P. Zulch, M. Distasio, et al., Joint sparsity based heterogeneous data-level fusion for target detection and estimation, in Sensors and Systems for Space Applications X. International Society for Optics and Photonics (2017), p. 10196

  9. Z. Liu, W. Zhang, S. Lin et al., Heterogeneous sensor data fusion by deep multimodal encoding. IEEE J. Sel. Top. Signal Process. 11(3), 479–491 (2017)

    Article  ADS  Google Scholar 

  10. S. Ren, K. He, R. Girshick, et al., Faster r-cnn: towards real-time object detection with region proposal networks, in Advances in Neural Information Processing Systems (2015), pp. 91–99

  11. J. Ma, Y. Ma, C. Li, Infrared and visual image fusion methods and applications: a survey. Inf. Fus. 45, 153–178 (2019)

    Article  Google Scholar 

  12. D. Liu, D. Zhou, R. Nie et al., Infrared and visible image fusion based on convolutional neural network model and saliency detection via hybrid l0–l1 layer decomposition. J. Electron. Imag. 27(6), 063036 (2018)

    Article  Google Scholar 

  13. Y. Zhang, L. Zhang, X. Bai et al., Infrared and visual image fusion through infrared feature extraction and visual information preservation. Infra. Phys. Technol. 83, 227–237 (2017)

    Article  ADS  Google Scholar 

  14. H. Li, X.J. Wu, J. Kittler, Infrared and visual image fusion using a deep learning framework, in 2018 24th International Conference on Pattern Recognition (ICPR). IEEE (2018), pp. 2705–2710

  15. J. Redmon, S. Divvala, R. Girshick, et al., You only look once: unified, real-time object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 779–788

  16. J. Redmon, A. Farhadi, Yolov3: An Incremental Improvement. arXiv:1804.02767 (2018)

  17. W. Liu, D. Anguelov, D. Erhan, et al., Ssd: single shot multibox detector, in European Conference on Computer Vision (Springer, Cham, 2016), pp. 21–37

  18. R. Girshick. Fast r-cnn, in Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 1440–1448

  19. K. He, G. Gkioxari, P. Dollár, et al., Mask r-cnn, in Proceedings of the IEEE International Conference on Computer Vision (2017), pp. 2961–2969

  20. T.Y. Lin, P. Dollár, R. Girshick, et al., Feature pyramid networks for object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 2117–2125

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Pulsed Power Laser Technology, National University of Defense Technology, Hefei, 230037, China

    Yang Zhang, Bin Zhu, Bo Xie, Yi Chen & Hongwei Zhang

Authors
  1. Yang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongwei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Zhu, B., Xie, B. et al. Visual image and radio signal fusion identification based on convolutional neural networks. J Opt 50, 237–244 (2021). https://doi.org/10.1007/s12596-020-00672-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12596-020-00672-w

Keywords

Search

Navigation