Skip to main content
SpringerLink
Log in

Guided Deep Decoder: Unsupervised Image Pair Fusion

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12351))

Included in the following conference series:

Abstract

The fusion of input and guidance images that have a tradeoff in their information (e.g., hyperspectral and RGB image fusion or pansharpening) can be interpreted as one general problem. However, previous studies applied a task-specific handcrafted prior and did not address the problems with a unified approach. To address this limitation, in this study, we propose a guided deep decoder network as a general prior. The proposed network is composed of an encoder-decoder network that exploits multi-scale features of a guidance image and a deep decoder network that generates an output image. The two networks are connected by feature refinement units to embed the multi-scale features of the guidance image into the deep decoder network. The proposed network allows the network parameters to be optimized in an unsupervised way without training data. Our results show that the proposed network can achieve state-of-the-art performance in various image fusion problems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www1.cs.columbia.edu/CAVE/databases/.

References

  1. Akhtar, N., Shafait, F., Mian, A.: Bayesian sparse representation for hyperspectral image super resolution. In: CVPR, pp. 3631–3640 (2015)

    Google Scholar 

  2. Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F., Selva, M.: Multispectral and panchromatic data fusion assessment without reference. Photogramm. Eng. Remote Sens. 74(2), 193–200 (2008)

    Article  Google Scholar 

  3. Alparone, L., Baronti, S., Garzelli, A., Nencini, F.: A global quality measurement of pan-sharpened multispectral imagery. IEEE Geosci. Remote Sens. Lett. 1(4), 313–317 (2004)

    Article  Google Scholar 

  4. Chakrabarty, P., Maji, S.: The spectral bias of the deep image prior. In: NeurIPS Workshops (2019)

    Google Scholar 

  5. Chen, C., Li, Y., Liu, W., Huang, Z.: SIRF: simultaneous satellite image registration and fusion in a unified framework. IEEE Trans. Image Process. 24(11), 4213–4224 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  6. Chen, C., Li, Y., Liu, W., Huang, J.: Image fusion with local spectral consistency and dynamic gradient sparsity. In: CVPR (2014)

    Google Scholar 

  7. Dian, R., Fang, L., Li, S.: Hyperspectral image super-resolution via non-local sparse tensor factorization. In: CVPR, pp. 3862–3871 (2017)

    Google Scholar 

  8. Dian, R., Li, S., Guo, A., Fang, L.: Deep hyperspectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5345–5355 (2018)

    Article  MathSciNet  Google Scholar 

  9. Dong, W., et al.: Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing 25(5) (2016)

    Google Scholar 

  10. Fu, X., Lin, Z., Huang, Y., Ding, X.: A variational pan-sharpening with local gradient constraints. In: CVPR (2019)

    Google Scholar 

  11. Fu, Y., Zhang, T., Zheng, Y., Zhang, D., Huang, H.: Hyperspectral image super-resolution with optimized rgb guidance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11661–11670 (2019)

    Google Scholar 

  12. Gandelsman, Y., Shocher, A., Irani, M.: “Double-Dip”: unsupervised image decomposition via coupled deep-image-priors. In: CVPR (2019)

    Google Scholar 

  13. Garzelli, A., Nencini, F., Capobianco, L.: Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Trans. Geosci. Remote Sens. 46(1), 228–236 (2007)

    Article  Google Scholar 

  14. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

    Google Scholar 

  15. Heckel, R., Hand, P.: Deep decoder: Concise image representations from untrained non-convolutional networks. In: ICLR (2019)

    Google Scholar 

  16. Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y., Ikeuchi, K.: High-resolution hyperspectral imaging via matrix factorization. In: CVPR, pp. 2329–2336 (2011)

    Google Scholar 

  17. Kwon, H., Tai, Y.W.: RGB-guided hyperspectral image upsampling. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 307–315 (2015)

    Google Scholar 

  18. Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: ICCV (2015)

    Google Scholar 

  19. Liu, P., Xiao, L., Li, T.: A variational pan-sharpening method based on spatial fractional-order geometry and spectral–spatial low-rank priors. IEEE Trans. Geosci. Remote Sens. 56, 1788–1802 (2018)

    Article  Google Scholar 

  20. Lutio, R.d., D’Aronco, S., Wegner, J.D., Schindler, K.: Guided super-resolution as pixel-to-pixel transformation. In: ICCV (2019)

    Google Scholar 

  21. Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G.: Pansharpening by convolutional neural networks. Remote Sensing 8(7), 594 (2016)

    Article  Google Scholar 

  22. Palsson, F., Sveinsson, J.R., Ulfarsson, M.O.: A new pansharpening algorithm based on total variation. IEEE Geosci. Remote Sens. Lett. 11, 318–322 (2014)

    Article  Google Scholar 

  23. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23(3), 664 (2004)

    Article  Google Scholar 

  24. Qu, Y., Qi, H., Kwan, C.: Unsupervised sparse Dirichlet-net for hyperspectral image super-resolution. In: CVPR (2018)

    Google Scholar 

  25. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  26. Scarpa, G., Vitale, S., Cozzolino, D.: Target-adaptive CNN-based pansharpening. IEEE Trans. Geosci. Remote Sens. 56(9), 5443–5457 (2018)

    Article  Google Scholar 

  27. Sidorov, O., Hardeberg, J.Y.: Deep hyperspectral prior: denoising, inpainting, super-resolution. In: ICIP (2019)

    Google Scholar 

  28. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: CVPR (2018)

    Google Scholar 

  29. Vivone, G., et al.: A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53(5), 2565–2586 (2014)

    Article  Google Scholar 

  30. Wald, L., Ranchin, T., Mangolini, M.: Fusion of satellite images of different spatial resoltuions: assessing the quality of resulting images. Photogrammetric engineering and remote sensing 63(6), 691–699 (1997)

    Google Scholar 

  31. Wald, L.: Quality of high resolution synthesised images: is there a simple criterion? In: Third Conference Fusion of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images, pp. 99–103. SEE/URISCA (2000)

    Google Scholar 

  32. Wang, Z., Bovik, A.C.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)

    Article  Google Scholar 

  33. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  34. Wei, Q., Dobigeon, N., Tourneret, J., Bioucas-Dias, J., Godsill, S.: R-FUSE: robust fast fusion of multiband images based on solving a Sylvester equation. IEEE Signal Process. Lett. 23(11), 1632–1636 (2016)

    Article  Google Scholar 

  35. Wei, Y., Yuan, Q., Shen, H., Zhang, L.: Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geosci. Remote Sens. Lett. 14(10), 1795–1799 (2017)

    Article  Google Scholar 

  36. Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., Xu, Z.: Multispectral and hyperspectral image fusion by MS/HS fusion net. In: CVPR (2019)

    Google Scholar 

  37. Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., Paisley, J.: PanNet: a deep network architecture for pan-sharpening. In: ICCV. pp. 1753–1761 (2017)

    Google Scholar 

  38. Yokota, T., Kawai, K., Sakata, M., Kimura, Y., Hontani, H.: Dynamic pet image reconstruction using nonnegative matrix factorization incorporated with deep image prior. In: ICCV (2019)

    Google Scholar 

  39. Yokoya, N., Grohnfeldt, C., Chanussot, J.: Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geosci. Remote Sens. Mag. 5(2), 29–56 (2017)

    Article  Google Scholar 

  40. Yokoya, N., Yairi, T., Iwasaki, A.: Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 50(2), 528–537 (2012)

    Article  Google Scholar 

  41. Zhou, J., Civco, D., Silander, J.: A wavelet transform method to merge landsat TM and SPOT panchromatic data. Int. J. Remote Sens. 19(4), 743–757 (1998)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. RIKEN AIP, Tokyo, Japan

    Tatsumi Uezato, Naoto Yokoya & Wei He

  2. German Aerospace Center, Wessling, Germany

    Danfeng Hong

  3. Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, Grenoble, France

    Danfeng Hong

  4. The University of Tokyo, Tokyo, Japan

    Naoto Yokoya

Authors
  1. Tatsumi Uezato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danfeng Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Naoto Yokoya
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tatsumi Uezato .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 16917 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Uezato, T., Hong, D., Yokoya, N., He, W. (2020). Guided Deep Decoder: Unsupervised Image Pair Fusion. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12351. Springer, Cham. https://doi.org/10.1007/978-3-030-58539-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58539-6_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58538-9

  • Online ISBN: 978-3-030-58539-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation