Advertisement

Guided Deep Decoder: Unsupervised Image Pair Fusion

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12351)

Abstract

The fusion of input and guidance images that have a tradeoff in their information (e.g., hyperspectral and RGB image fusion or pansharpening) can be interpreted as one general problem. However, previous studies applied a task-specific handcrafted prior and did not address the problems with a unified approach. To address this limitation, in this study, we propose a guided deep decoder network as a general prior. The proposed network is composed of an encoder-decoder network that exploits multi-scale features of a guidance image and a deep decoder network that generates an output image. The two networks are connected by feature refinement units to embed the multi-scale features of the guidance image into the deep decoder network. The proposed network allows the network parameters to be optimized in an unsupervised way without training data. Our results show that the proposed network can achieve state-of-the-art performance in various image fusion problems.

Keywords

Deep image prior Deep decoder Image fusion Hyperspectral image Super-resolution Pansharpening 

Supplementary material

504443_1_En_6_MOESM1_ESM.pdf (16.5 mb)
Supplementary material 1 (pdf 16917 KB)

References

  1. 1.
    Akhtar, N., Shafait, F., Mian, A.: Bayesian sparse representation for hyperspectral image super resolution. In: CVPR, pp. 3631–3640 (2015)Google Scholar
  2. 2.
    Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F., Selva, M.: Multispectral and panchromatic data fusion assessment without reference. Photogramm. Eng. Remote Sens. 74(2), 193–200 (2008)CrossRefGoogle Scholar
  3. 3.
    Alparone, L., Baronti, S., Garzelli, A., Nencini, F.: A global quality measurement of pan-sharpened multispectral imagery. IEEE Geosci. Remote Sens. Lett. 1(4), 313–317 (2004)CrossRefGoogle Scholar
  4. 4.
    Chakrabarty, P., Maji, S.: The spectral bias of the deep image prior. In: NeurIPS Workshops (2019)Google Scholar
  5. 5.
    Chen, C., Li, Y., Liu, W., Huang, Z.: SIRF: simultaneous satellite image registration and fusion in a unified framework. IEEE Trans. Image Process. 24(11), 4213–4224 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Chen, C., Li, Y., Liu, W., Huang, J.: Image fusion with local spectral consistency and dynamic gradient sparsity. In: CVPR (2014)Google Scholar
  7. 7.
    Dian, R., Fang, L., Li, S.: Hyperspectral image super-resolution via non-local sparse tensor factorization. In: CVPR, pp. 3862–3871 (2017)Google Scholar
  8. 8.
    Dian, R., Li, S., Guo, A., Fang, L.: Deep hyperspectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5345–5355 (2018)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Dong, W., et al.: Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing 25(5) (2016)Google Scholar
  10. 10.
    Fu, X., Lin, Z., Huang, Y., Ding, X.: A variational pan-sharpening with local gradient constraints. In: CVPR (2019)Google Scholar
  11. 11.
    Fu, Y., Zhang, T., Zheng, Y., Zhang, D., Huang, H.: Hyperspectral image super-resolution with optimized rgb guidance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11661–11670 (2019)Google Scholar
  12. 12.
    Gandelsman, Y., Shocher, A., Irani, M.: “Double-Dip”: unsupervised image decomposition via coupled deep-image-priors. In: CVPR (2019)Google Scholar
  13. 13.
    Garzelli, A., Nencini, F., Capobianco, L.: Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Trans. Geosci. Remote Sens. 46(1), 228–236 (2007)CrossRefGoogle Scholar
  14. 14.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)Google Scholar
  15. 15.
    Heckel, R., Hand, P.: Deep decoder: Concise image representations from untrained non-convolutional networks. In: ICLR (2019)Google Scholar
  16. 16.
    Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y., Ikeuchi, K.: High-resolution hyperspectral imaging via matrix factorization. In: CVPR, pp. 2329–2336 (2011)Google Scholar
  17. 17.
    Kwon, H., Tai, Y.W.: RGB-guided hyperspectral image upsampling. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 307–315 (2015)Google Scholar
  18. 18.
    Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: ICCV (2015)Google Scholar
  19. 19.
    Liu, P., Xiao, L., Li, T.: A variational pan-sharpening method based on spatial fractional-order geometry and spectral–spatial low-rank priors. IEEE Trans. Geosci. Remote Sens. 56, 1788–1802 (2018)CrossRefGoogle Scholar
  20. 20.
    Lutio, R.d., D’Aronco, S., Wegner, J.D., Schindler, K.: Guided super-resolution as pixel-to-pixel transformation. In: ICCV (2019)Google Scholar
  21. 21.
    Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G.: Pansharpening by convolutional neural networks. Remote Sensing 8(7), 594 (2016)CrossRefGoogle Scholar
  22. 22.
    Palsson, F., Sveinsson, J.R., Ulfarsson, M.O.: A new pansharpening algorithm based on total variation. IEEE Geosci. Remote Sens. Lett. 11, 318–322 (2014)CrossRefGoogle Scholar
  23. 23.
    Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23(3), 664 (2004)CrossRefGoogle Scholar
  24. 24.
    Qu, Y., Qi, H., Kwan, C.: Unsupervised sparse Dirichlet-net for hyperspectral image super-resolution. In: CVPR (2018)Google Scholar
  25. 25.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  26. 26.
    Scarpa, G., Vitale, S., Cozzolino, D.: Target-adaptive CNN-based pansharpening. IEEE Trans. Geosci. Remote Sens. 56(9), 5443–5457 (2018)CrossRefGoogle Scholar
  27. 27.
    Sidorov, O., Hardeberg, J.Y.: Deep hyperspectral prior: denoising, inpainting, super-resolution. In: ICIP (2019)Google Scholar
  28. 28.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: CVPR (2018)Google Scholar
  29. 29.
    Vivone, G., et al.: A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53(5), 2565–2586 (2014)CrossRefGoogle Scholar
  30. 30.
    Wald, L., Ranchin, T., Mangolini, M.: Fusion of satellite images of different spatial resoltuions: assessing the quality of resulting images. Photogrammetric engineering and remote sensing 63(6), 691–699 (1997)Google Scholar
  31. 31.
    Wald, L.: Quality of high resolution synthesised images: is there a simple criterion? In: Third Conference Fusion of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images, pp. 99–103. SEE/URISCA (2000)Google Scholar
  32. 32.
    Wang, Z., Bovik, A.C.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)CrossRefGoogle Scholar
  33. 33.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  34. 34.
    Wei, Q., Dobigeon, N., Tourneret, J., Bioucas-Dias, J., Godsill, S.: R-FUSE: robust fast fusion of multiband images based on solving a Sylvester equation. IEEE Signal Process. Lett. 23(11), 1632–1636 (2016)CrossRefGoogle Scholar
  35. 35.
    Wei, Y., Yuan, Q., Shen, H., Zhang, L.: Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geosci. Remote Sens. Lett. 14(10), 1795–1799 (2017)CrossRefGoogle Scholar
  36. 36.
    Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., Xu, Z.: Multispectral and hyperspectral image fusion by MS/HS fusion net. In: CVPR (2019)Google Scholar
  37. 37.
    Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., Paisley, J.: PanNet: a deep network architecture for pan-sharpening. In: ICCV. pp. 1753–1761 (2017)Google Scholar
  38. 38.
    Yokota, T., Kawai, K., Sakata, M., Kimura, Y., Hontani, H.: Dynamic pet image reconstruction using nonnegative matrix factorization incorporated with deep image prior. In: ICCV (2019)Google Scholar
  39. 39.
    Yokoya, N., Grohnfeldt, C., Chanussot, J.: Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geosci. Remote Sens. Mag. 5(2), 29–56 (2017)CrossRefGoogle Scholar
  40. 40.
    Yokoya, N., Yairi, T., Iwasaki, A.: Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 50(2), 528–537 (2012)CrossRefGoogle Scholar
  41. 41.
    Zhou, J., Civco, D., Silander, J.: A wavelet transform method to merge landsat TM and SPOT panchromatic data. Int. J. Remote Sens. 19(4), 743–757 (1998)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.RIKEN AIPTokyoJapan
  2. 2.German Aerospace CenterWesslingGermany
  3. 3.Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-LabGrenobleFrance
  4. 4.The University of TokyoTokyoJapan

Personalised recommendations