Skip to main content
SpringerLink
Log in

Remote Sensing Image Fusion Using Multi-Scale Convolutional Neural Network

  • Research Article
  • Published:
Journal of the Indian Society of Remote Sensing Aims and scope Submit manuscript

Abstract

In this paper, a novel remote sensing (RS) image fusion algorithm based on Multi-scale convolutional neural network is proposed. The most important innovation is that the proposed remote sensing image fusion method utilizes a set of convolutional neural networks (CNN) to perform multi-scale image analysis on each band of a multispectral image in order to extract the typical characteristics of different band of multispectral images. In addition, to prevent losing the information of the original image, the max-pooling layer of the traditional CNN is replaced with a standard convolutional layer, and the standard convolutional layer has one step size of 2. The RS image fusion results presented in this paper demonstrate that the proposed method is not only competitive with the most advanced methods, but also superior to other classical methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. For PCA, GS, AWLP, GLP and CNMF methods, the source codes can be download from: http://openremotesensing.net/knowledgebase/a-critical-comparison-among-pansharpening-algorithms/.

  2. http://glcf.umd.edu/data/ikonos/.

  3. http://www.digitalglobe.com/product-samples.

References

  • Aiazzi, B., Alparone, L., Baronti, S., & Garzelli, A. (2002). Context-driven fusionof high spatial and spectral resolution images based on oversampled multiresolutionanalysis. IEEE Transactions on Geoscience and Remote Sensing, 40(10), 2300–2312.

    Article  Google Scholar 

  • Aiazzi, B., Alparone, L., Baronti, S., & Garzelli, A. (2002). Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remote Sensing, 40, 2300–2312.

    Article  Google Scholar 

  • Alparone, L., Wald, L., Chanussot, J., Thomas, C., Gamba, P., & Bruce, L. M. (2007). Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S datafusion contest. IEEE Transactions on Geoscience and Remote Sensing, 45(10), 3012–3021.

    Article  Google Scholar 

  • Chavez, P. S., Jr., & Kwarteng, A. Y. (1989). Extracting spectral contrast in landsat thematic map- per image data using selective principal component analysis. Photogramm Engineering and Remote Sensing, 55(3), 339–348.

    Google Scholar 

  • Chavez, P. S., & Kwarteng, A. Y. (1989). Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis. Photogramm Engineering and Remote Sensing, 55, 339–348.

    Google Scholar 

  • Choi, J., Yu, K., & Kim, Y. (2011). A new adaptive component substitutionbased satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing, 49(1), 295–309.

    Article  Google Scholar 

  • Deng, L.-J., Feng, M., & Tai, X.-C. (2019). The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior. Information Fusion, 52, 76–89.

    Article  Google Scholar 

  • Dian, R., Li, S., Fang, L., & Wei, Qi. (2019). Multispectral and hyperspectral image fusion with spatial-spectral sparse representation. Information Fusion., 49, 262–270.

    Article  Google Scholar 

  • Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.

    Article  Google Scholar 

  • Farfade, S., Saberian, M. Li, L (2015) Multi-view face detection using deep convolutional neural networks, in: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 643–650.

  • Fasbender, D., Radoux, J., & Bogaert, P. (2008). Bayesian data fusion for adaptable image pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 46(6), 1847–1857.

    Article  Google Scholar 

  • Garzelli, A., Nencini, F., & Capobianco, L. (2008). Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Transactions on Geoscience and Remote Sensing, 46(1), 228–236.

    Article  Google Scholar 

  • Ghahremani, M., & Ghassemian, H. (2016). Nonlinear IHS: a Promising method for pansharpening. IEEE Geoscience and Remote Sensing Letters, 13, 1606–1610.

    Article  Google Scholar 

  • Gillespie, A. R., Kahle, A. B., & Walker, R. E. (1987). Color enhancement of highly correlated images. 2. channel ratio and chromaticity transformation techniques. Remote Sensing Environment, 22, 345–365.

    Article  Google Scholar 

  • Glorot, X. Bengio, Y. (2010) Understanding the difficulty of training deep feed forward neural networks, in: International Conference on Artificial Intelligence and Statistics

  • He, K., Zhang, X., Ren, S., & Sun, J. (2015). Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9), 1904–1916.

    Article  Google Scholar 

  • https://en.wikipedia.org/wiki/Deep _ learning . 01-Jan-2017

  • Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T., Caffe: convolutional architecture for fast feature embedding, In: Proceedings of the ACM International Conference on Multimedia, 2014, pp. 675–678.

  • Kim, J., Lee, J. K., and LEE, K.M. (2016) Accurate image super-resolution using very deep convolutional networks. In Las Vegas, NV, USA, 2016, pp. 1646–1654.

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90.

    Article  Google Scholar 

  • Laben, C.A., Brower, B.V. (2000) Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening,. US Patent 6011875.

  • Laben, C. A., Brower, B. V.(2000) Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. Google Patents US 6011875 A.

  • Liu, J. G. (2000). “Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing, 21, 3461–3472.

    Article  Google Scholar 

  • Long, J., Shelhamer, E., Darrell T. (2015) Fully convolutional networks for semantic segmentation, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440

  • Matikainen, L., Karila, K., Hyypp¨a, J., Litkey, P., Puttonen, E., & Ahokas, E. (2017). Object-based analysis ofmultispectral airborne laser scanner data for landcover classification and map updating. ISPRS Journal of Photogrammetry and Remote Sensing, 128, 298–313.

    Article  Google Scholar 

  • Nencini, F., Garzelli, A., Baronti, S., & Alparone, L. (2007). Remote sensing image fusion using the curvelet transform. Information Fusion, 8, 143–156.

    Article  Google Scholar 

  • Otazu, X., González-Audícana, M., Fors, O., & Núñez, J. (2005). Introduction of sensor spectral response into image fusion methods. IEEE Transactions on Geoscience and Remote Sensing, 43, 2376–2385.

    Article  Google Scholar 

  • Ouyang, W. et al. (2015) Deepid-net: Deformable deep convolutional neural networks for object detection. In Boston, MA, USA, 2015, pp. 2403–2412.

  • Palsson, F., Sveinsson, J. R., Ulfarsson, M. O., & Benediktsson, J. A. (2016). MTF-based deblurring using a wiener filter for CS and MRA pansharpeningmethods. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9, 2255–2269.

    Article  Google Scholar 

  • Plowright, A. A., Coops, N. C., Chance, C. M., Sheppard, S. R. J., & Aven, N. W. (2017). Multi-scale analysis of relationship between imperviousness and urban tree height using airborne remote sensing. Remote Sensing of Environment, 194, 391–400.

    Article  Google Scholar 

  • Posselt, R., Mueller, R. W., St¨ockli, R., & Trentmann, J. (2012). Remote sensingof solar surface radiation for climate monitoring-the CM-SAF retrieval ininternational comparison. Remote Sensing of Environment, 118(6), 186–198.

    Article  Google Scholar 

  • Shao, Z., & Cai, J. (2018). Remote sensing image fusion with deep convolutional neural network. IEEE Journal Of Selected Topics In Applied Earth Observations And Remote Sensing, 11(5), 1656–1669.

    Article  Google Scholar 

  • Shen, H., Meng, X., & Zhang, L. (2016). An integrated framework for the spatio–temporal–spectral fusion of remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 54(12), 7135–7148.

    Article  Google Scholar 

  • Tu, T. M., Huang, P. S., Hung, C. L., & Chang, C. P. (2004). A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geoscience and Remote Sensing Letters, 1(4), 309–312.

    Article  Google Scholar 

  • Wald, L. (2000) Quality of high resolution synthesised images: Is there a simple criterion?. In Procceedings 3rd conference fusion earth data: Merging point meas. raster maps remotely sensed images, Sophia-Antipolis, France, pp. 99–103.

  • Wald, L., Ranchin, T., & Mangolini, M. (1997). Fusion of satellite images of different spatial resolution: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63, 691–699.

    Google Scholar 

  • Wang, Z., & Bovik, A. C. (2002). A universal image quality index. IEEE Signal Processing Letters, 9(3), 81–84.

    Article  Google Scholar 

  • Yokoya, N., Yairi, T., & Iwasaki, A. (2012). Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 50, 528–537.

    Article  Google Scholar 

  • Yokoya, N., Grohnfeldt, C., & Chanussot, J. (2017). Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geoscience and Remote Sensing Magazine, 5, 29–56.

    Article  Google Scholar 

  • Zhang, L., Shen, H., Gong, W., & Zhang, H. (2012). Adjustable model-based fusion method for multispectral and panchromatic images. IEEE Transactions on Systems, Man, and Cybernetics Part B (Cybernetics), 42(6), 1693–1704.

    Article  Google Scholar 

  • Zhang, P., Gong, M., Su, L., Liu, J., & Li, Z. (2016). Change detection based on deep feature representation and mapping transformation for multi-spatial-resolution remote sensing images. ISPRS Journal of Photogrammetry and Remote Sensing, 116, 24–41.

    Article  Google Scholar 

  • Zhong, J., Yang, B., Huang, G., et al. (2016). Remote sensingimage fusion with convolutional neural network. Sensing Imaging, 17(1), 339–348.

    Article  Google Scholar 

  • Zhu, Z., Yin, H., Chai, Y., Li, Y., & Qi, G. (2018). A novel multi-modality image fusion method based on image decomposition and sparse representation. Information Science, 432, 516–529.

    Article  Google Scholar 

Download references

Acknowledgement

The work of this paper was supported by the National Natural Science Foundation of China (Project Number: 41904028).

Author information

Authors and Affiliations

  1. College of Geography and Environment Sciences, Northwest Normal University, Lanzhou, 730070, China

    Wei Shi

  2. School of Automation, Northwestern Polytechnical University, Xi’an 710129, China

    ChaoBen Du & BingBing Gao

  3. School of Computer Science, China University of Geosciences, Wuhan, 430074, China

    JiNing Yan

Authors
  1. Wei Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. ChaoBen Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. BingBing Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. JiNing Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Shi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, W., Du, C., Gao, B. et al. Remote Sensing Image Fusion Using Multi-Scale Convolutional Neural Network. J Indian Soc Remote Sens 49, 1677–1687 (2021). https://doi.org/10.1007/s12524-021-01353-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12524-021-01353-2

Keywords

Search

Navigation