Skip to main content
SpringerLink
Log in

Infrared blind-pixel compensation algorithm based on generative adversarial networks and Poisson image blending

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Infrared focal plane array is affected by nonuniformity, which leads to the generation of infrared image blind pixels and reduces the image quality. This paper puts forward an infrared blind-pixel compensation algorithm based on generative adversarial networks and Poisson image blending (GAN–PIB), which implements the greyscale prediction of the original image blind pixels through the combination of the pre-trained adversarial network and the blind-pixel compensation function and in the form of generating new images, breaking through the traditional thinking of the existing compensation algorithm based on interpolation and filtering. Firstly, the blind-pixel compensation network is constructed based on the generative adversarial networks. Through training, the model learns the image features of infrared blind pixels and achieves a better compensation effect for the blind-pixel image data sets which are used for training. Secondly, the blind-pixel detection of the test blind-pixel images is carried out to generate a binary mask matrix, and the constructed blind-pixel compensation loss function is combined to create the fake image. Finally, by fitting the original blind-pixel image and the generated image with the Poisson image blending algorithm, the compensation precision is improved through iteration, and finally, the compensation for the infrared blind pixels is completed. The experimental results show that GAN–PIB algorithm has strong adaptability to isolated blind pixels and clustered blind pixels, and compared with traditional algorithms, the compensated images have higher intelligibility and texture details.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Mooney, J.M., et al.: Responsivity nonuniformity limited performance of infrared staring cameras. Opt. Eng. 28(11), 1151–1161 (1989)

    Article  Google Scholar 

  2. Faraklioti, M., Petrou, M.: Multiresolution versus single resolution horizon picking in 2D seismic images. Proc. SPIE Int. Soc. Opt. Eng. 5238(6), 1165–1172 (2004)

    Google Scholar 

  3. Wei, Y., Shi, Z.: An automatic target detection algorithm based on wavelet analysis for infrared image small target in background of sea and sky. Proc. SPIE Int. Soc. Opt. Eng. 2003, 123–131 (2003)

    Google Scholar 

  4. Liu, N., Xie, J.: Interframe phase-correlated registration scene-based nonuniformity correction technology. Infrared Phys. Technol. 69, 198–205 (2015)

    Article  Google Scholar 

  5. Goma, S., Aleksic, M.: Bad pixel location algorithm for cell phone cameras. Proc SPIE 6502, 65020H-65020H-10 (2007)

    Google Scholar 

  6. Iwamoto, M., Ueyama, D.: Autonomously detecting the defective pixels in an imaging sensor array using a robust statistical technique. Proc. SPIE Int. Soc. Opt. Eng. 6808, 680813-680813-12 (2008)

    Google Scholar 

  7. Dierickx, B., Meynants, G.: Missing pixel correction algorithm for image sensors. Proc. SPIE Int. Soc. Opt. Eng. 3410, 200–203 (1999)

    Google Scholar 

  8. Nguyen, C.T., Mould, N., Regens, J.L.: Dead pixel correction techniques for dual-band infrared imagery. Infrared Phys. Technol. 71, 227–235 (2015)

    Article  Google Scholar 

  9. Cao, Y., et al.: Scene-based bad pixel dynamic correction and evaluation for IRFPA device. In: Advances in Optoelectronics and Micro/Nano-Optics (AOM), 2010 OSA-IEEE-COS (2010), pp. 1–4

  10. Chen, S., Meng, H., Pei, T., et al.: An adaptive regression method for infrared blind-pixel compensation. Infrared Phys. Technol. 85, 443–449 (2017)

    Article  Google Scholar 

  11. Ledig, C., Theis, L., Huszar, F., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR (2017)

  12. Yeh, R.A., Chen, C., Lim, T.Y., Schwing, A.G., Hasegawa-Johnson, M., Do, M.N.: Semantic image inpainting with deep generative models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5485–5493 (2017)

  13. Divakar, N., Babu, R.V.: Image denoising via CNNs: an adversarial approach. In: Computer Vision and Pattern Recognition Workshops IEEE, pp. 1076–1083 (2017)

  14. Harmeling, S., Schuler, C.J., Burger, H.C.: Image denoising: can plain neural networks compete with BM3D? In: 2012 IEEE Conference on Computer Vision and Pattern Recognition IEEE Computer Society (2012)

  15. Goodfellow, I.J., et al.: Generative adversarial nets. In: International Conference on Neural Information Processing Systems, pp. 2672–2680. MIT Press (2014)

  16. Ratliff, L.J., Burden, S.A., Sastry, S.S.: Characterization and computation of local Nash equilibria in continuous games. In: Communication, Control, and Computing, pp. 917–924. IEEE (2006)

  17. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning (ICML), pp. 214–223M (2017)

  18. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.: Improved training of Wasserstein GANs. In: Advances in Neural Information Processing Systems (NIPS) (2017)

  19. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)

  20. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

  21. Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: International Conference on Machine Learning, vol. 30 (2013)

  22. Xu, B., et al.: Empirical evaluation of rectified activations in convolutional network (2015). arXiv preprint arXiv:1505.00853

  23. Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)

    Article  MathSciNet  Google Scholar 

  24. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  Google Scholar 

  25. Pathak, D., et al.: Context encoders: feature learning by inpainting. In: IEEE conference on computer vision and pattern recognition, pp. 2536–2544. IEEE Computer Society (2016)

  26. Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2014)

    Article  Google Scholar 

  27. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: International Conference on Learning Representations (2016)

  28. Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017). arXiv:1611.07004

  29. Rez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), 313–318 (2003)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (61705019), and Natural Science Foundation of the Jiangsu Higher Education Institutions of China (12KJA510001).

Author information

Authors and Affiliations

  1. Jiangsu Key Laboratory of Meteorological Observation and Information Processing, Nanjing University of Information Science and Technology, Nanjing, 210044, China

    Suting Chen & Meng Jin

  2. Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology (CICAEET), Nanjing University of Information Science and Technology, Nanjing, 210044, China

    Suting Chen, Yanyan Zhang & Chuang Zhang

Authors
  1. Suting Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meng Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanyan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chuang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SC curated the data; MJ proposed the methodology; SC, CZ and YZ supervised the study; MJ wrote the original draft; SC and MJ reviewed and edited the manuscript.

Corresponding author

Correspondence to Suting Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, S., Jin, M., Zhang, Y. et al. Infrared blind-pixel compensation algorithm based on generative adversarial networks and Poisson image blending. SIViP 14, 77–85 (2020). https://doi.org/10.1007/s11760-019-01526-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-019-01526-0

Keywords

Search

Navigation