Advertisement

Astrodynamics

, Volume 3, Issue 4, pp 387–402 | Cite as

Super-resolution of PROBA-V images using convolutional neural networks

  • Marcus MärtensEmail author
  • Dario Izzo
  • Andrej Krzic
  • Daniël Cox
Research Article
  • 170 Downloads

Abstract

European Space Aqency (ESA)’s PROBA-V Earth observation (EO) satellite enables us to monitor our planet at a large scale to study the interaction between vegetation and climate, and provides guidance for important decisions on our common global future. However, the interval at which high-resolution images are recorded spans over several days, in contrast to the availability of lower-resolution images which is often daily. We collect an extensive dataset of both high- and low-resolution images taken by PROBA-V instruments during monthly periods to investigate Multi Image Super-resolution, a technique to merge several low-resolution images into one image of higher quality. We propose a convolutional neural network (CNN) that is able to cope with changes in illumination, cloud coverage, and landscape features which are introduced by the fact that the different images are taken over successive satellite passages at the same region. Given a bicubic upscaling of low resolution images taken under optimal conditions, we find the Peak Signal to Noise Ratio of the reconstructed image of the network to be higher for a large majority of different scenes. This shows that applied machine learning has the potential to enhance large amounts of previously collected EO data during multiple satellite passes.

Keywords

deep learning convolutional neural network (CNN) super-resolution imaging remote sensing Earth observation (EO) 

Notes

Acknowledgements

The authors would like to thank the anonymous reviewers for their extremely detailed comments and critical questions, which helped to make the presentation of these results clearer and increased the overall quality of this work.

References

  1. [1]
    Park, S. C., Park, M. K., Kang, M. G. Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine, 2003, 20(3): 21–36.CrossRefGoogle Scholar
  2. [2]
    Nasrollahi, K., Moeslund, T. B. Super-resolution: a comprehensive survey. Machine Vision and Applications, 2014, 25(6): 1423–1468.CrossRefGoogle Scholar
  3. [3]
    Latry, C., Delvit, J. M. Staggered arrays for high resolution earth observing systems. In: Proceedings of SPIE 7452, Earth Observing Systems XIV, 2009: 74520O.CrossRefGoogle Scholar
  4. [4]
    Li, L., Wang, W., Luo, H., Ying, S. Super-resolution reconstruction of high-resolution satellite ZY-3 TLC images. Sensors, 2017, 17(5): 1062.CrossRefGoogle Scholar
  5. [5]
    Dierckx, W., Sterckx, S., Benhadj, I., Livens, S., Duhoux, G., van Achteren, T., Francois, M., Mellab, K., Saint, G. PROBA-V mission for global vegetation monitoring: Standard products and image quality. International Journal of Remote Sensing, 2014, 35(7): 2589–2614.CrossRefGoogle Scholar
  6. [6]
    Yang, C. Y., Ma, C., Yang, M. H. Single-image super-resolution: a benchmark. In: Computer Vision — ECCV 2014. Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T., Eds. Cham: Springer International Publishing, 2014: 372–386.CrossRefGoogle Scholar
  7. [7]
    Yang, J. C., Wright, J., Huang, T. S., Ma, Y. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861–2873.MathSciNetCrossRefGoogle Scholar
  8. [8]
    Dong, C., Loy, C. C., He, K. M., Tang, X. O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295–307.CrossRefGoogle Scholar
  9. [9]
    Kim, J., Lee, J. K., Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646–1654.CrossRefGoogle Scholar
  10. [10]
    Dong, C., Loy, C. C., Tang, X. O. Accelerating the super-resolution convolutional neural network. In: Computer Vision — ECCV 2016. Leibe, B., Matas, J., Sebe, N., Welling, M., Eds. Cham: Springer International Publishing, 2016: 391–407.CrossRefGoogle Scholar
  11. [11]
    Shi, W. Z., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., Wang, Z. H. Realtime single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.Google Scholar
  12. [12]
    Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z. H. et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017.Google Scholar
  13. [13]
    Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., Hasegawa-Johnson, M., Do, M. N. Semantic image inpainting with deep generative models. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5485–5493.Google Scholar
  14. [14]
    Timofte, R., Gu, S. H., van Gool, L., Zhang, L., Yang, M. H. NTIRE 2018 challenge on single image super-resolution: Methods and results. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2018: 852–863.Google Scholar
  15. [15]
    Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M., et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of Computer Vision and Pattern Recognition Workshops, 2017: 1110–1121.Google Scholar
  16. [16]
    He, K. M., Zhang, X. Y., Ren, S. Q., Sun, J. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778.CrossRefGoogle Scholar
  17. [17]
    Huang, G., Liu, Z., van der Maaten, L., Weinberger, K. Q. Densely connected convolutional networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700–4708.Google Scholar
  18. [18]
    Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612.CrossRefGoogle Scholar
  19. [19]
    Schultz, R. R., Stevenson, R. L. Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing, 1996, 5(6): 996–1011.CrossRefGoogle Scholar
  20. [20]
    Faramarzi, E., Rajan, D., Fernandes, F. C. A., Christensen, M. P. Blind super resolution of real-life video sequences. IEEE Transactions on Image Processing, 2016, 25(4): 1544–1555.MathSciNetCrossRefGoogle Scholar
  21. [21]
    Takeda, H., Milanfar, P., Protter, M., Elad, M. Superresolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing, 2009, 18(9): 1958–1975.MathSciNetCrossRefGoogle Scholar
  22. [22]
    Mitzel, D., Pock, T., Schoenemann, T., Cremers, D. Video super resolution using duality based TV-L1 optical flow. In: Lecture Notes in Computer Science. Denzler, J., Notni, G., Süße, H., Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009: 432–441.Google Scholar
  23. [23]
    Brodu, N. Super-resolving multiresolution images with band-independent geometry of multispectral pixels. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(8): 4610–4617.CrossRefGoogle Scholar
  24. [24]
    Gillespie, A. R., Kahle, A. B., Walker, R. E. Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques. Remote Sensing of Environment, 1987, 22(3): 343–365.CrossRefGoogle Scholar
  25. [25]
    Thomas, C., Ranchin, T., Wald, L., Chanussot, J. Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(5): 1301–1312.CrossRefGoogle Scholar
  26. [26]
    Murthy, K., Shearn, M., Smiley, B. D., Chau, A. H., Levine, J., Robinson, M. D. SkySat-1: very high-resolution imagery from a small satellite. In: Proceedings of SPIE 9241,Sensors, Systems, and Next-Generation Satellites XVIII, 2014: 92411E.Google Scholar
  27. [27]
    Wolters, E., Dierckx, W., Iordache, M. D., Swinnen, E. PROBA-V Products User Manual, 2014.Google Scholar
  28. [28]
    Carlson, T. N., Ripley, D. A. On the relation between NDVI, fractional vegetation cover, and leaf area index. Remote Sensing of Environment, 1997, 62(3): 241–252.CrossRefGoogle Scholar
  29. [29]
    Pettorelli, N., Vik, J. O., Mysterud, A., Gaillard, J. M., Tucker, C. J., Stenseth, N. C. Using the satellite-derived NDVI to assess ecological responses to environmental change. Trends in Ecology & Evolution, 2005, 20(9): 503–510.CrossRefGoogle Scholar
  30. [30]
    Wilson, A. M., Jetz, W. Remotely sensed high-resolution global cloud dynamics for predicting ecosystem and biodiversity distributions. PLoS Biology, 2016, 14(3): e1002415.CrossRefGoogle Scholar
  31. [31]
    Sheikh, H. R., Bovik, A. C. Image information and visual quality. IEEE Transactions on Image Processing, 2006, 15(2):430–444.CrossRefGoogle Scholar

Copyright information

© Tsinghua University Press 2019

Authors and Affiliations

  • Marcus Märtens
    • 1
    Email author
  • Dario Izzo
    • 1
  • Andrej Krzic
    • 1
  • Daniël Cox
    • 1
  1. 1.European Space AgencyNoordwijkthe Netherlands

Personalised recommendations