Abstract
Remote sensing image fusion (RSIF) is referenced as restoring the high-resolution multispectral image from its corresponding low-resolution multispectral (LMS) image aided by the panchromatic (PAN) image. Most RSIF methods assume that the missing spatial details of the LMS image can be obtained from the high resolution PAN image. However, the distortions would be produced due to the much difference between the structural component of LMS image and that of PAN image. Actually, the LMS image can fully utilize its spatial details to improve the resolution. In this paper, a novel two-stage RSIF algorithm is proposed, which makes full use of both spatial details and spectral information of the LMS image itself. In the first stage, the convolutional neural network based super-resolution is used to increase the spatial resolution of the LMS image. In the second stage, Gram–Schmidt transform is employed to fuse the enhanced MS and the PAN images for further improvement the resolution of MS image. Since the spatial resolution enhancement in the first stage, the spectral distortions in the fused image would be decreased in evidence. Moreover, the spatial details can be preserved to construct the fused images. The QuickBird satellite source images are used to test the performances of the proposed method. The experimental results demonstrate that the proposed method can achieve better spatial details and spectral information simultaneously compared with other well-known methods.
Similar content being viewed by others
References
Thomas, C., Wald, L., Ranchin, T., & Chanussot, J. (2008). Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Transactions on Geoscience and Remote Sensing, 46(5), 1301–1312. doi:10.1109/TGRS.2007.912448.
Lu, T., Li, S. T., & Fu, W. (2014). Fusion based seamless mosaic for remote sensing images. Sensing and Imaging. doi:10.1007/s11220-014-0101-0.
Kang, X. D., Li, S. T., Fang, L. Y., & Benediktsson, J. A. (2014). Pansharpening based on intrinsic image decomposition. Sensing and Imaging. doi:10.1007/s11220-014-0094-8.
Ragheb, A. M., Osman, H., Abbas, A. M., & Elkaffas, S. M. (2012). Simultaneous fusion and denoising of panchromatic and multispectral satellite Images. Sensing and Imaging, 13(3), 119–141. doi:10.1007/s11220-013-0075-3.
Choi, M. (2006). A new intensity-hue-saturation fusion approach to image fusion with a tradeoff parameter. IEEE Transactions on Geoscience and Remote Sensing, 44(6), 1672–1682. doi:10.1109/TGRS.2006.869923.
Tu, T. M., Huang, P. S., Hung, C. L., & Chang, C. P. (2004). A fast intensity–hue–saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geoscience and Remote Sensing Letters, 1(4), 309–312. doi:10.1109/LGRS.2004.834804.
Laben, C. A., & Brower, B. V. (2000). Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S. Patent 6 011 875, Tech. Rep., Eastman Kodak Company.
Huang, F. H., & Yan, L. M. (2013). Study on the hyperspectral image fusion based on the Gram Schmidt improved algorithm. Information Technology Journal, 12(22), 6694–6701. doi:10.3923/itj.2013.6694.6701.
Chavez, P. S., Sides, S. C., & Anderson, J. A. (1991). Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic. Photogrammetric Engineering and Remote Sensing, 57(3), 295–303.
Ibrahim, S., & Wirth, M. (2009). Multiresolution region-based image fusion using the Contourlet transform. In Science and technology for humanity (TIC-STH), 2009 IEEE Toronto international conference, 26–27 Sept 2009 (pp. 421–426). doi: 10.1109/TIC-STH.2009.5444464.
Pradhan, P. S., King, R. L., & Younan, N. H. (2006). Estimation of the number of decomposition levels for a wavelet-based multiresolution multisensor image fusion. IEEE Transactions on Geoscience and Remote Sensing, 44(12), 3674–3686. doi:10.1109/TGRS.2006.881758.
Li, S. (2008). Multisensor remote sensing image fusion using stationary wavelet transform: effects of basis and decomposition level. International Journal of Wavelets, Multiresolution and Information Processing, 6(1), 37–50. doi:10.1142/S0219691308002203.
Núñez, J., Otazu, X., Fors, O., Prades, A., Palà, V., & Arbiol, R. (1999). Multiresolution-based image fusion with additive wavelet decomposition. IEEE Transactions on Geoscience and Remote Sensing, 37(3), 1204–1211. doi:10.1109/36.763274.
Zheng, S., Shi, W., & Liu, J. (2008). Remote sensing image fusion using multiscale mapped LS-SVM. IEEE Transactions on Geoscience and Remote Sensing, 46(5), 1313–1322. doi:10.1109/TGRS.2007.912737.
Wu, Y. Q., & Tao, F. X. (2015). Multispectral and panchromatic image fusion based on improved projected gradient NMF in NSST domain. Acta Optica Sinica, 35(4), 0410005. doi:10.3788/aos201535.0410005.
Otazu, X., González-Audícana, M., & Fors, O. (2005). Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing, 43(10), 2376–2385. doi:10.1109/TGRS.2005.856106.
Shah, V. P., Younan, N. H., & King, R. L. (2008). An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Transactions on Geoscience and Remote Sensing, 46(5), 1323–1335. doi:10.1109/TGRS.2005.856106.
Moonon, A. U., Hu, J. W., & Li, S. T. (2015). Remote sensing image fusion method based on nonsubsampled shearlet transform and sparse representation. Sensing and Imaging. doi:10.1007/s11220-015-0125-0.
Liu, B., Li, K., & Liu, W. (2013). Construction method of three-channel non-separable symmetric wavelets with arbitrary dilation matrices and its applications in multispectral image fusion. Image Processing, IET, 7(7), 679–685. doi:10.1049/iet-ipr.2013.0085.
Dong, C., Loy, C. C., He, K.M., & Tang, X. O. (2014). Learning a deep convolutional network for image super-resolution. In Proceedings of the 13th European conference on computer vision, 6–12 Sept 2014 (Vol. 8692, pp. 184–199). doi: 10.1007/978-3-319-10593-2_13.
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. doi:10.1109/5.726791.
Zhou, X. R., Liu, J., Liu, S. G., Cao, L., Zhou, Q. M., & Huang, H. W. (2014). A GIHS-based spectral preservation fusion method for remote sensing images using edge restored spectral modulation. ISPRS Journal of Photogrammetry and Remote Sensing., 88, 16–27. doi:10.1016/j.isprsjprs.2013.11.011.
Acknowledgments
This paper is supported by the National Natural Science Foundation of China (Nos. 61102108), Scientific Research Fund of Hunan Provincial Education Department (Nos. YB2013B039), Young talents program of the University of South China, and the construct program of key disciplines in USC (No. NHXK04).
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is part of the Topical Collection on Hyperspectral Imaging and Image Processing.
Rights and permissions
About this article
Cite this article
Zhong, J., Yang, B., Huang, G. et al. Remote Sensing Image Fusion with Convolutional Neural Network. Sens Imaging 17, 10 (2016). https://doi.org/10.1007/s11220-016-0135-6
Received:
Revised:
Published:
DOI: https://doi.org/10.1007/s11220-016-0135-6