Abstract
In the last few years, Earth Observation sensors received a large development, offering, therefore, various types of data with different temporal, spatial, spectral, and radiometric resolutions. However, due to physical and budget limitations, the acquisition of images with the best characteristics is not feasible. Image fusion becomes a valuable technique to deal with some specific applications. In particular, the vegetation area, which needs a high spectral resolution and frequent coverage. In this paper, we present a novel fusion technique based on Convolutional Neural Networks (CNN) to combine two kinds of remote sensing data with different but complement spectral and temporal characteristics, to produce one high spectral and temporal resolution product. To the best of our knowledge, this is the first attempt to deal with the spectral-temporal fusion problem. The feasibility of the proposed method is evaluated via Sentinel-2 data. The experimental results show that the proposed technique can achieve substantial gains in terms of fusion performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A., Selva, M.: Twenty-five years of pansharpening: a critical review and new developments. In: Signal and Image Processing for Remote Sensing, pp. 552–599. CRC Press (2012)
Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A.: Remote Sensing Image Fusion. CRC Press, Boca Raton (2015)
Benzenati, T., Kallel, A., Kessentini, Y.: Two stages pan-sharpening details injection approach based on very deep residual networks. IEEE Trans. Geosci. Remote Sens. (2020)
Benzenati, T., Kessentini, Y., Kallel, A., Hallabia, H.: Generalized Laplacian pyramid pan-sharpening gain injection prediction based on CNN. IEEE Geosci. Remote Sens. Lett. 17(4), 651–655 (2019)
Bovolo, F., Bruzzone, L., Capobianco, L., Garzelli, A., Marchesi, S., Nencini, F.: Analysis of the effects of pansharpening in change detection on VHR images. IEEE Geosci. Remote Sens. Lett. 7(1), 53–57 (2009)
Chang, N.B., Bai, K.: Multisensor Data Fusion and Machine Learning for Environmental Remote Sensing. CRC Press, Boca Raton (2018)
Clevers, J.G., Kooistra, L., Van den Brande, M.M.: Using Sentinel-2 data for retrieving LAI and leaf and canopy chlorophyll content of a potato crop. Remote Sens. 9(5), 405 (2017)
Daudt, R.C., Le Saux, B., Boulch, A., Gousseau, Y.: Urban change detection for multispectral earth observation using convolutional neural networks. In: IGARSS 2018–2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 2115–2118. IEEE (2018)
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)
Frampton, W.J., Dash, J., Watmough, G., Milton, E.J.: Evaluating the capabilities of Sentinel-2 for quantitative estimation of biophysical variables in vegetation. ISPRS J. Photogramm. Remote Sens. 82, 83–92 (2013)
Gao, F., Masek, J., Schwaller, M., Hall, F.: On the blending of the Landsat and MODIS surface reflectance: predicting daily Landsat surface reflectance. IEEE Trans. Geosci. Remote Sens. 44(8), 2207–2218 (2006)
Gilbertson, J.K., Kemp, J., Van Niekerk, A.: Effect of pan-sharpening multi-temporal Landsat 8 imagery for crop type differentiation using different classification techniques. Comput. Electron. Agric. 134, 151–159 (2017)
Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Liu, X., Deng, C., Chanussot, J., Hong, D., Zhao, B.: StfNet: a two-stream convolutional neural network for spatiotemporal image fusion. IEEE Trans. Geosci. Remote Sens. 57(9), 6552–6564 (2019)
Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G.: Pansharpening by convolutional neural networks. Remote Sens. 8(7), 594 (2016)
Planet: Satellite Imagery and Archive. https://planet.com/products/planet-imagery/. Accessed 01 July 2020
Song, H., Liu, Q., Wang, G., Hang, R., Huang, B.: Spatiotemporal satellite image fusion using deep convolutional neural networks. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 11(3), 821–829 (2018)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Wu, H., Zhang, H., Zhang, J., Xu, F.: Typical target detection in satellite images based on convolutional neural networks. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics, pp. 2956–2961. IEEE (2015)
Yokoya, N., Chan, J.C.W., Segl, K.: Potential of resolution-enhanced hyperspectral data for mineral mapping using simulated EnMAP and Sentinel-2 images. Remote Sens. 8(3), 172 (2016)
Yuhas, R.H., Goetz, A.F., Boardman, J.W.: Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm (1992)
Acknowledgement
We gratefully acknowledge the support of NVIDIA Corporation with the contribution of the Titan Xp GPU used for this research.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Benzenati, T., Kallel, A., Kessentini, Y. (2021). End-to-End Spectral-Temporal Fusion Using Convolutional Neural Network. In: Djeddi, C., Kessentini, Y., Siddiqi, I., Jmaiel, M. (eds) Pattern Recognition and Artificial Intelligence. MedPRAI 2020. Communications in Computer and Information Science, vol 1322. Springer, Cham. https://doi.org/10.1007/978-3-030-71804-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-71804-6_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-71803-9
Online ISBN: 978-3-030-71804-6
eBook Packages: Computer ScienceComputer Science (R0)