European Space Aqency (ESA)’s PROBA-V Earth observation (EO) satellite enables us to monitor our planet at a large scale to study the interaction between vegetation and climate, and provides guidance for important decisions on our common global future. However, the interval at which high-resolution images are recorded spans over several days, in contrast to the availability of lower-resolution images which is often daily. We collect an extensive dataset of both high- and low-resolution images taken by PROBA-V instruments during monthly periods to investigate Multi Image Super-resolution, a technique to merge several low-resolution images into one image of higher quality. We propose a convolutional neural network (CNN) that is able to cope with changes in illumination, cloud coverage, and landscape features which are introduced by the fact that the different images are taken over successive satellite passages at the same region. Given a bicubic upscaling of low resolution images taken under optimal conditions, we find the Peak Signal to Noise Ratio of the reconstructed image of the network to be higher for a large majority of different scenes. This shows that applied machine learning has the potential to enhance large amounts of previously collected EO data during multiple satellite passes.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Park, S. C., Park, M. K., Kang, M. G. Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine, 2003, 20(3): 21–36.
Nasrollahi, K., Moeslund, T. B. Super-resolution: a comprehensive survey. Machine Vision and Applications, 2014, 25(6): 1423–1468.
Latry, C., Delvit, J. M. Staggered arrays for high resolution earth observing systems. In: Proceedings of SPIE 7452, Earth Observing Systems XIV, 2009: 74520O.
Li, L., Wang, W., Luo, H., Ying, S. Super-resolution reconstruction of high-resolution satellite ZY-3 TLC images. Sensors, 2017, 17(5): 1062.
Dierckx, W., Sterckx, S., Benhadj, I., Livens, S., Duhoux, G., van Achteren, T., Francois, M., Mellab, K., Saint, G. PROBA-V mission for global vegetation monitoring: Standard products and image quality. International Journal of Remote Sensing, 2014, 35(7): 2589–2614.
Yang, C. Y., Ma, C., Yang, M. H. Single-image super-resolution: a benchmark. In: Computer Vision — ECCV 2014. Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T., Eds. Cham: Springer International Publishing, 2014: 372–386.
Yang, J. C., Wright, J., Huang, T. S., Ma, Y. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861–2873.
Dong, C., Loy, C. C., He, K. M., Tang, X. O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307.
Kim, J., Lee, J. K., Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646–1654.
Dong, C., Loy, C. C., Tang, X. O. Accelerating the super-resolution convolutional neural network. In: Computer Vision — ECCV 2016. Leibe, B., Matas, J., Sebe, N., Welling, M., Eds. Cham: Springer International Publishing, 2016: 391–407.
Shi, W. Z., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., Wang, Z. H. Realtime single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z. H. et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., Hasegawa-Johnson, M., Do, M. N. Semantic image inpainting with deep generative models. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5485–5493.
Timofte, R., Gu, S. H., van Gool, L., Zhang, L., Yang, M. H. NTIRE 2018 challenge on single image super-resolution: Methods and results. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2018: 852–863.
Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M., et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of Computer Vision and Pattern Recognition Workshops, 2017: 1110–1121.
He, K. M., Zhang, X. Y., Ren, S. Q., Sun, J. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778.
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K. Q. Densely connected convolutional networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700–4708.
Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612.
Schultz, R. R., Stevenson, R. L. Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing, 1996, 5(6): 996–1011.
Faramarzi, E., Rajan, D., Fernandes, F. C. A., Christensen, M. P. Blind super resolution of real-life video sequences. IEEE Transactions on Image Processing, 2016, 25(4): 1544–1555.
Takeda, H., Milanfar, P., Protter, M., Elad, M. Superresolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing, 2009, 18(9): 1958–1975.
Mitzel, D., Pock, T., Schoenemann, T., Cremers, D. Video super resolution using duality based TV-L1 optical flow. In: Lecture Notes in Computer Science. Denzler, J., Notni, G., Süße, H., Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009: 432–441.
Brodu, N. Super-resolving multiresolution images with band-independent geometry of multispectral pixels. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(8): 4610–4617.
Gillespie, A. R., Kahle, A. B., Walker, R. E. Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques. Remote Sensing of Environment, 1987, 22(3): 343–365.
Thomas, C., Ranchin, T., Wald, L., Chanussot, J. Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(5): 1301–1312.
Murthy, K., Shearn, M., Smiley, B. D., Chau, A. H., Levine, J., Robinson, M. D. SkySat-1: very high-resolution imagery from a small satellite. In: Proceedings of SPIE 9241,Sensors, Systems, and Next-Generation Satellites XVIII, 2014: 92411E.
Wolters, E., Dierckx, W., Iordache, M. D., Swinnen, E. PROBA-V Products User Manual, 2014.
Carlson, T. N., Ripley, D. A. On the relation between NDVI, fractional vegetation cover, and leaf area index. Remote Sensing of Environment, 1997, 62(3): 241–252.
Pettorelli, N., Vik, J. O., Mysterud, A., Gaillard, J. M., Tucker, C. J., Stenseth, N. C. Using the satellite-derived NDVI to assess ecological responses to environmental change. Trends in Ecology & Evolution, 2005, 20(9): 503–510.
Wilson, A. M., Jetz, W. Remotely sensed high-resolution global cloud dynamics for predicting ecosystem and biodiversity distributions. PLoS Biology, 2016, 14(3): e1002415.
Sheikh, H. R., Bovik, A. C. Image information and visual quality. IEEE Transactions on Image Processing, 2006, 15(2):430–444.
The authors would like to thank the anonymous reviewers for their extremely detailed comments and critical questions, which helped to make the presentation of these results clearer and increased the overall quality of this work.
Marcus Märtens graduated from the University of Paderborn (Germany) with a master degree in computer science. He joined ESA as a young graduate trainee in artificial intelligence where he worked on multi-objective optimization of spacecraft trajectories. He was part of the winning team of the 8th edition of the Global Trajectory Optimization Competition (GTOC) and received a HUMIES gold medal for developing algorithms achieving human competitive results in trajectory design. The Delft University of Technology awarded him a Ph.D. degree for his thesis on information propagation in complex networks. After working at the network architectures and services group in Delft (the Netherlands), Marcus rejoined ESA, where he now works as a research follow in the Advanced Concepts Team. While his main focus is on applied artificial intelligence and evolutionary optimization, Marcus has been working with experts from different fields and authored works related to neuroscience, cyber-security and gaming.
Dario Izzo graduated as a doctor of aeronautical engineering from the University Sapienza of Rome (Italy). He then took his second master degree in satellite platforms at the University of Cranfield in the United Kingdom, and received his Ph.D. degree in mathematical modeling at the University Sapienza of Rome where he lectured classical mechanics and space flight mechanics. He later joined the ESA and became the scientific coordinator of its Advanced Concepts Team. He devised and managed the Global Trajectory Optimization Competitions events, the ESA’s Summer of Code in Space and the Kelvins innovation and competition platform for space problems. He published more than 170 papers in international journals and conferences, making key contributions to the understanding of flight mechanics and spacecraft control and pioneering techniques based on evolutionary and machine learning approaches. He received the HUMIES Gold Medal and led the team winning the 8th edition of the Global Trajectory Optimization Competition.
Andrej Krzic graduated in physics from the University of Ljubljana (Slovenia), and received his M.Sc. degree in optics from the Friedrich Schiller University Jena (Germany). While in Jena, he was working at Carl Zeiss Microscopy GmbH, where he investigated the applications of adaptive optics in microscopic imaging. After studying, Andrej spent two years as a young graduate trainee at the ESA in the Netherlands. He was working at the Advanced Concepts Team, where he was active in several areas of research, including super-resolution imaging, quantum metrology, and optical communication. Andrej later returned to Jena, where he is now working on adaptive optics assisted quantum communication at the Fraunhofer Institute for Applied Optics and Precision Engineering.
Daniël Cox graduated as a Bachelor of science in applied physics at the University of Twente (the Netherlands) and is currently working on his master degree in applied physics with a focus on optics and image processing. He was briefly working at the Advanced Concepts Team of ESA during his internship.
About this article
Cite this article
Märtens, M., Izzo, D., Krzic, A. et al. Super-resolution of PROBA-V images using convolutional neural networks. Astrodyn 3, 387–402 (2019). https://doi.org/10.1007/s42064-019-0059-8
- deep learning
- convolutional neural network (CNN)
- super-resolution imaging
- remote sensing
- Earth observation (EO)