Skip to main content
SpringerLink
Log in

A multi-scale mixed convolutional network for infrared image super-resolution reconstruction

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Infrared image is widely used in military, medical, monitoring security and other fields. Due to the limitation of hardware devices, infrared image has the problems of low signal-to-noise ratio, blurred edge and low contrast. In view of the above problems, In this paper, a super-resolution reconstruction method of infrared image based on mixed convolution multi-scale residual network is proposed. Through the multi-scale residual network to improve the utilization of features, the mixed convolution is introduced into the multi-scale residual network, which can increase the receptive field without changing the size of the feature map and eliminate the blind spots. The extracted features are fused by recursive fusion to improve the utilization of features. Through experiments and tests on multiple infrared image data sets, Through the test on the infrared image data set show that the proposed method can improve the infrared image edge information, fully extract the texture details from the infrared image, and suppress noise. The objective index of the reconstructed infrared image is mainly better than that of the contrast method, and can still achieve a better reconstruction effect in the real scene.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The data that support the findings of this study are available on request from the corresponding author upon reasonable request.

References

  1. Alain H, Ziou D (2010) Image quality metrics: PSNR vs. SSIM, 2010 20th international conference on pattern recognition. Istanbul. https://doi.org/10.1109/ICPR.2010.579

  2. Alejandro G, Fang Z, Yainuvis S et al (2016) Pedestrian detection at day/night time with visible and FIR cameras: a comparison. Sensors 16(6):820. https://doi.org/10.3390/s16060820

    Article  Google Scholar 

  3. Andreas NS (1997) Space-based infrared system (SBIRS) system of systems. IEEE Aerospace Conf:429–438

  4. Anwar S, Khan S, Barnes N (2020) A deep journey into super-resolution: a survey. ACM Comput Surv 53(3):1–34. https://doi.org/10.1145/3390462

    Article  Google Scholar 

  5. Baker S, Kanade T (2002) Limits on super-resolution and how to break them. IEEE Trans Pattern Anal Mach Intell 24(9):1167–1183. https://doi.org/10.1109/cvpr.2000.854852

    Article  Google Scholar 

  6. Baker S, Kanade T (2002) Limits on super-resolution and how to break them. IEEE Trans Pattern Anal Mach Intell 24(9):1167–1183. https://doi.org/10.1109/cvpr.2000.854852

    Article  Google Scholar 

  7. Dai T, Cai J, Zhang Y, Xia S, Zhang L (2019) Second-Order Attention Network for Single Image Super-Resolution, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.01132

  8. Dong C, Loy C, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. Eur Conf Comp Vision (ECCV) 8692:184–199. https://doi.org/10.1007/978-3-319-10593-2_13

    Article  Google Scholar 

  9. Elad M, Feuer A (1997) Restoration of a single super-resolution image from several blurred, noisy, and under-sampled measured images. IEEE Trans Image Process 6(12):1646–1658. https://doi.org/10.1109/83.650118

    Article  Google Scholar 

  10. Elad M, Feuer A (2002) Restoration of a single super-resolution image from several blurred, Noisy, and under sampled measured images. IEEE Trans Image Process 6(12):1646–1658

    Article  Google Scholar 

  11. Goodfellow I, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial networks. Proceed 27th Int Conf Neural Inform Process Syst 2(14):2672–2680

    Google Scholar 

  12. He Z, Tang S, Yang J, Cao Y, Ying Yang M, Cao Y (2019) Cascaded deep networks with multiple receptive fields for infrared image super-resolution. IEEE Trans Circuits Syst Video Technol 29(8):2310–2322. https://doi.org/10.1109/tcsvt.2018.2864777

    Article  Google Scholar 

  13. Justin J, Alexandre A, Li F (2016) Perceptual losses for real-time style transfer and super-resolution. Proceed Eur Conf Comp Vision 694–711. https://doi.org/10.1007/978-3-319-46475-6_43

  14. Kermany D, Goldbaum M, Cai W et al (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(5):1122–1131. https://doi.org/10.1016/j.cell.2018.02.010

    Article  Google Scholar 

  15. Kim K, Kwon Y (2010) Single-image super-resolution using sparse regression and natural image prior. IEEE Trans Pattern Anal Mach Intell 32(6):1127–1133. https://doi.org/10.1109/tpami.2010.25

    Article  Google Scholar 

  16. Kim J, Lee J, Lee K (2016) Deeply-recursive convolutional network for image super-resolution. IEEE Conf Comput Vision Patt Recog (CVPR) 1637-1645. https://doi.org/10.1109/cvpr.2016.181

  17. Kingma D, Ba J (2014) Adam: a method for stochastic optimization, computer science. arXiv preprint arXiv:1412.6980

  18. Kuang X, Sui X, Liu Y, Chen Q, Gu G (2019) Single infrared image enhancement using a deep convolutional neural network. Neurocomputing 332(7):119–128. https://doi.org/10.1016/j.neucom.2018.11.081

    Article  Google Scholar 

  19. Lahiri BB, Bagavathiappan S, Jayakumar T et al (2012) Medical applications of infrared thermography: a review. Infrared Phys Technol 55(4):221–235

    Article  Google Scholar 

  20. Lai W, Huang J, Ahuja N, Yang M (2017) Deep Laplacian pyramid networks for fast and accurate super-resolution. IEEE Conf Comp Vision Patt Recogn (CVPR) 5835–5843. https://doi.org/10.1109/CVPR.2017.618

  21. Ledig C, Theis L, Huszar F, Caballero J (2017) Photo-realistic single image super-resolution using a generative adversarial network. IEEE Conf Comp Vision Patt Recogn (CVPR) 105-114. https://doi.org/10.1109/cvpr.2017.19

  22. Li J, Fang F, Mei K, G. (2018) Zhang, multi-scale residual network for image super-resolution. Eur Conf Comput Vision(ECCV) 11212:527–542. https://doi.org/10.1007/978-3-030-01237-3_32

    Article  Google Scholar 

  23. Liu F, Han P, Wang Y, Li X, Bai L, Shao X (2016) Super resolution reconstruction of infrared images based on classified dictionary learning. Infrared Phys Technol 76:139–147. https://doi.org/10.1016/j.infrared.2018.03.008

    Article  Google Scholar 

  24. Ma J, Ma J, Ma Y, Li C (2019) Infrared and visible image fusion methods and applications: a survey. Inform Fusion 45:153–178. https://doi.org/10.1016/j.inffus.2018.02.004

    Article  Google Scholar 

  25. Ma J, Liang Z, Zhang L (2022) A text attention network for spatial deformation robust scene text image super-resolution. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, pp 5901–5910. https://doi.org/10.1109/CVPR52688.2022.00582

  26. Michal I, Shmuel P (1991) Improving resolution by image registration, CVGIP. Graph Models Image Process 53(3):231–239. https://doi.org/10.1016/1049-9652(91)90045-l

    Article  Google Scholar 

  27. Pei Y, Huang Y, Zou Q et al (2021) Effects of image degradation and degradation removal to CNN-based image classification. IEEE Trans Pattern Anal Mach Intell 43(4):1239–1253

    Article  Google Scholar 

  28. Qiu Y, Wang R, Tao D, Cheng J (2019) Embedded block residual network: a recursive restoration model for single-image super-resolution. IEEE/CVF Int Conf Comput Vision (ICCV) 4179-4188. https://doi.org/10.1109/iccv.2019.00428

  29. Qu Z, Jiang P, Zhang W (2020) Development and application of infrared thermography non-destructive testing techniques. Sensors 20(14):3851

    Article  Google Scholar 

  30. Sheikh H, Bovik A (2005) An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans Image Process 14(12):2117–2128. https://doi.org/10.1109/tip.2005.859389

    Article  Google Scholar 

  31. Stark H (1989) High-resolution image recovery from image-plane arrays, using convex projections. J Opt Soc Am A 6(11):1715–1726. https://doi.org/10.1364/josaa.6.001715

    Article  Google Scholar 

  32. Stark H, Oskoui P (1989) High-resolution image recovery from image-plane arrays, using convex projection. J Opt Soc Am A 6(11):1715–1726. https://doi.org/10.1364/josaa.6.001715

    Article  Google Scholar 

  33. Timofte R, De S, Van G (2015) A+: adjusted anchored neighborhood regression for fast super-resolution. Asian Conf Comput Vision 9006:111–126. https://doi.org/10.1007/978-3-319-16817-3_8

    Article  Google Scholar 

  34. Wang Z, Bovik A, Sheikh H et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/tip.2003.819861

    Article  Google Scholar 

  35. Yang T, Tang Y, Lv X, Wang Q, Ke H, Deng Y et al (2017) Size-dependent Ag2S Nan dots for second near-infrared fluorescence/photo acoustics imaging and simultaneous photo thermal therapy. ACS Nano 11(2):1848–1857. https://doi.org/10.1021/acsnano.6b07866.s001

    Article  Google Scholar 

  36. Yang X, Wu W, Liu K, Chen W, Zhou Z (2018) Multiple dictionary pairs learning and sparse representation-based infrared image super-resolution with improved fuzzy clustering. Soft Comput 22:1385–1398. https://doi.org/10.1007/s00500-017-2812-3

    Article  Google Scholar 

  37. Yang F, Yang H, Fu J et al (2020) Learning texture transformer network for image super-resolution, 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/cvpr42600.2020.00583

  38. Yu F, Koltun V (2016) Multi-scale context aggregation by dilated convolutions, ICLR, arXiv preprint arXiv:1511.07122v3

  39. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238. https://doi.org/10.1109/tip.2006.877407

    Article  Google Scholar 

  40. Zhang K, Gao X, Tao D, Li X (2012) Single image super-resolution with non-local means and steering kernel regression. IEEE Trans Image Process 21(11):4544–4556. https://doi.org/10.1109/tip.2012.2208977

    Article  MathSciNet  MATH  Google Scholar 

  41. Zhang Z, Wang X, Jung C (2019) DCSR: dilated convolutions for single image super-resolution. IEEE Trans Image Process 28(4):1625–1635. https://doi.org/10.1109/tip.2018.2877483

    Article  MathSciNet  Google Scholar 

Download references

Funding

The authors are grateful for collaborative funding support from the Humanity and Social Science Foundation of Ministry of Education, China (21YJAZH077).

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, China

    Yan-Bin Du, Hong-Mei Sun, Zhe Cui & Rui-Sheng Jia

  2. Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590, China

    Hong-Mei Sun & Rui-Sheng Jia

  3. Graduate School of Shandong University of Science and Technology, Qingdao, 266590, China

    Bin Zhang

Authors
  1. Yan-Bin Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong-Mei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhe Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rui-Sheng Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hong-Mei Sun or Rui-Sheng Jia.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Du, YB., Sun, HM., Zhang, B. et al. A multi-scale mixed convolutional network for infrared image super-resolution reconstruction. Multimed Tools Appl 82, 41895–41911 (2023). https://doi.org/10.1007/s11042-023-15359-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15359-0

Keywords

Search

Navigation