Skip to main content
SpringerLink
Log in

Remote Sensing Image Fusion with Convolutional Neural Network

  • Original Paper
  • Published:
Sensing and Imaging Aims and scope Submit manuscript

Abstract

Remote sensing image fusion (RSIF) is referenced as restoring the high-resolution multispectral image from its corresponding low-resolution multispectral (LMS) image aided by the panchromatic (PAN) image. Most RSIF methods assume that the missing spatial details of the LMS image can be obtained from the high resolution PAN image. However, the distortions would be produced due to the much difference between the structural component of LMS image and that of PAN image. Actually, the LMS image can fully utilize its spatial details to improve the resolution. In this paper, a novel two-stage RSIF algorithm is proposed, which makes full use of both spatial details and spectral information of the LMS image itself. In the first stage, the convolutional neural network based super-resolution is used to increase the spatial resolution of the LMS image. In the second stage, Gram–Schmidt transform is employed to fuse the enhanced MS and the PAN images for further improvement the resolution of MS image. Since the spatial resolution enhancement in the first stage, the spectral distortions in the fused image would be decreased in evidence. Moreover, the spatial details can be preserved to construct the fused images. The QuickBird satellite source images are used to test the performances of the proposed method. The experimental results demonstrate that the proposed method can achieve better spatial details and spectral information simultaneously compared with other well-known methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Thomas, C., Wald, L., Ranchin, T., & Chanussot, J. (2008). Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Transactions on Geoscience and Remote Sensing, 46(5), 1301–1312. doi:10.1109/TGRS.2007.912448.

    Article  Google Scholar 

  2. Lu, T., Li, S. T., & Fu, W. (2014). Fusion based seamless mosaic for remote sensing images. Sensing and Imaging. doi:10.1007/s11220-014-0101-0.

    Google Scholar 

  3. Kang, X. D., Li, S. T., Fang, L. Y., & Benediktsson, J. A. (2014). Pansharpening based on intrinsic image decomposition. Sensing and Imaging. doi:10.1007/s11220-014-0094-8.

    Google Scholar 

  4. Ragheb, A. M., Osman, H., Abbas, A. M., & Elkaffas, S. M. (2012). Simultaneous fusion and denoising of panchromatic and multispectral satellite Images. Sensing and Imaging, 13(3), 119–141. doi:10.1007/s11220-013-0075-3.

    Article  Google Scholar 

  5. Choi, M. (2006). A new intensity-hue-saturation fusion approach to image fusion with a tradeoff parameter. IEEE Transactions on Geoscience and Remote Sensing, 44(6), 1672–1682. doi:10.1109/TGRS.2006.869923.

    Article  Google Scholar 

  6. Tu, T. M., Huang, P. S., Hung, C. L., & Chang, C. P. (2004). A fast intensity–hue–saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geoscience and Remote Sensing Letters, 1(4), 309–312. doi:10.1109/LGRS.2004.834804.

    Article  Google Scholar 

  7. Laben, C. A., & Brower, B. V. (2000). Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S. Patent 6 011 875, Tech. Rep., Eastman Kodak Company.

  8. Huang, F. H., & Yan, L. M. (2013). Study on the hyperspectral image fusion based on the Gram Schmidt improved algorithm. Information Technology Journal, 12(22), 6694–6701. doi:10.3923/itj.2013.6694.6701.

    Article  Google Scholar 

  9. Chavez, P. S., Sides, S. C., & Anderson, J. A. (1991). Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic. Photogrammetric Engineering and Remote Sensing, 57(3), 295–303.

    Google Scholar 

  10. Ibrahim, S., & Wirth, M. (2009). Multiresolution region-based image fusion using the Contourlet transform. In Science and technology for humanity (TIC-STH), 2009 IEEE Toronto international conference, 26–27 Sept 2009 (pp. 421–426). doi: 10.1109/TIC-STH.2009.5444464.

  11. Pradhan, P. S., King, R. L., & Younan, N. H. (2006). Estimation of the number of decomposition levels for a wavelet-based multiresolution multisensor image fusion. IEEE Transactions on Geoscience and Remote Sensing, 44(12), 3674–3686. doi:10.1109/TGRS.2006.881758.

    Article  Google Scholar 

  12. Li, S. (2008). Multisensor remote sensing image fusion using stationary wavelet transform: effects of basis and decomposition level. International Journal of Wavelets, Multiresolution and Information Processing, 6(1), 37–50. doi:10.1142/S0219691308002203.

    Article  MathSciNet  MATH  Google Scholar 

  13. Núñez, J., Otazu, X., Fors, O., Prades, A., Palà, V., & Arbiol, R. (1999). Multiresolution-based image fusion with additive wavelet decomposition. IEEE Transactions on Geoscience and Remote Sensing, 37(3), 1204–1211. doi:10.1109/36.763274.

    Article  Google Scholar 

  14. Zheng, S., Shi, W., & Liu, J. (2008). Remote sensing image fusion using multiscale mapped LS-SVM. IEEE Transactions on Geoscience and Remote Sensing, 46(5), 1313–1322. doi:10.1109/TGRS.2007.912737.

    Article  Google Scholar 

  15. Wu, Y. Q., & Tao, F. X. (2015). Multispectral and panchromatic image fusion based on improved projected gradient NMF in NSST domain. Acta Optica Sinica, 35(4), 0410005. doi:10.3788/aos201535.0410005.

    Article  Google Scholar 

  16. Otazu, X., González-Audícana, M., & Fors, O. (2005). Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing, 43(10), 2376–2385. doi:10.1109/TGRS.2005.856106.

    Article  Google Scholar 

  17. Shah, V. P., Younan, N. H., & King, R. L. (2008). An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Transactions on Geoscience and Remote Sensing, 46(5), 1323–1335. doi:10.1109/TGRS.2005.856106.

    Article  Google Scholar 

  18. Moonon, A. U., Hu, J. W., & Li, S. T. (2015). Remote sensing image fusion method based on nonsubsampled shearlet transform and sparse representation. Sensing and Imaging. doi:10.1007/s11220-015-0125-0.

    Google Scholar 

  19. Liu, B., Li, K., & Liu, W. (2013). Construction method of three-channel non-separable symmetric wavelets with arbitrary dilation matrices and its applications in multispectral image fusion. Image Processing, IET, 7(7), 679–685. doi:10.1049/iet-ipr.2013.0085.

    Article  Google Scholar 

  20. Dong, C., Loy, C. C., He, K.M., & Tang, X. O. (2014). Learning a deep convolutional network for image super-resolution. In Proceedings of the 13th European conference on computer vision, 6–12 Sept 2014 (Vol. 8692, pp. 184–199). doi: 10.1007/978-3-319-10593-2_13.

  21. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. doi:10.1109/5.726791.

    Article  Google Scholar 

  22. Zhou, X. R., Liu, J., Liu, S. G., Cao, L., Zhou, Q. M., & Huang, H. W. (2014). A GIHS-based spectral preservation fusion method for remote sensing images using edge restored spectral modulation. ISPRS Journal of Photogrammetry and Remote Sensing., 88, 16–27. doi:10.1016/j.isprsjprs.2013.11.011.

    Article  Google Scholar 

Download references

Acknowledgments

This paper is supported by the National Natural Science Foundation of China (Nos. 61102108), Scientific Research Fund of Hunan Provincial Education Department (Nos. YB2013B039), Young talents program of the University of South China, and the construct program of key disciplines in USC (No. NHXK04).

Author information

Authors and Affiliations

  1. College of Electric Engineering, University of South China, Hengyang, 421001, China

    Jinying Zhong, Bin Yang, Guoyu Huang, Fei Zhong & Zhongze Chen

Authors
  1. Jinying Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guoyu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fei Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhongze Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bin Yang.

Additional information

This article is part of the Topical Collection on Hyperspectral Imaging and Image Processing.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhong, J., Yang, B., Huang, G. et al. Remote Sensing Image Fusion with Convolutional Neural Network. Sens Imaging 17, 10 (2016). https://doi.org/10.1007/s11220-016-0135-6

Download citation

  • Received:

  • Revised:

  • Published:

  • DOI: https://doi.org/10.1007/s11220-016-0135-6

Keywords

Search

Navigation