Skip to main content
SpringerLink
Log in

Learning the external and internal priors for multispectral and hyperspectral image fusion

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Recently, multispectral image (MSI) and hyperspectral image (HSI) fusion has been a popular topic in high-resolution HSI acquisition. This fusion leads to a challenging underdetermined problem, which image priors are used to regularize, aiming at improving fusion accuracy. To fully exploit HSI priors, this paper proposes two kinds of priors, i.e., external priors and internal priors, to regularize the fusion problem. An external prior represents the general image characteristics and is learned from abundant training data by using a Gaussian denoising convolutional neural network (CNN) trained in the additional gray images. An internal prior represents the unique characteristics of the HSI and MSI to be fused. To learn the external prior, we first segment the MSI into several superpixels and then enforce a low-rank constraint for each superpixel, which can well model local similarities in the HSI. In addition, to model a low-rank property in the spectral mode, the high-resolution HSI is decomposed into a low-rank spectral basis and abundances. Finally, we formulate the fusion as an external and internal prior-regularized optimization problem, which is efficiently tackled through the alternating direction method of multipliers. Experiments on simulated and real datasets demonstrate the superiority of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Gu Y, Liu T, Gao G, et al. Multimodal hyperspectral remote sensing: an overview and perspective. Sci China Inf Sci, 2021, 64: 121301

    Article  Google Scholar 

  2. Xu F, Hu C, Li J, et al. Special focus on deep learning in remote sensing image processing. Sci China Inf Sci, 2020, 63: 140300

    Article  Google Scholar 

  3. Mei S, Geng Y, Hou J, et al. Learning hyperspectral images from RGB images via a coarse-to-fine CNN. Sci China Inf Sci, 2022, 65: 152102

    Article  Google Scholar 

  4. Hong D, Gao L, Yokoya N, et al. More diverse means better: multimodal deep learning meets remote-sensing imagery classification. IEEE Trans Geosci Remote Sens, 2020, 59: 4340–4354

    Article  Google Scholar 

  5. Hong D, Yokoya N, Chanussot J, et al. CoSpace: common subspace learning from hyperspectral-multispectral correspondences. IEEE Trans Geosci Remote Sens, 2019, 57: 4349–4359

    Article  Google Scholar 

  6. Hou Z, Li W, Tao R, et al. Collaborative representation with background purification and saliency weight for hyperspectral anomaly detection. Sci China Inf Sci, 2022, 65: 112305

    Article  Google Scholar 

  7. Akbari H, Kosugi Y, Kojima K, et al. Detection and analysis of the intestinal ischemia using visible and invisible hyperspectral imaging. IEEE Trans Biomed Eng, 2010, 57: 2011–2017

    Article  Google Scholar 

  8. Pan Z H, Healey G, Prasad M, et al. Face recognition in hyperspectral images. IEEE Trans Pattern Anal Machine Intell, 2003, 25: 1552–1560

    Article  Google Scholar 

  9. Wang L, Xiong Z, Huang H, et al. High-speed hyperspectral video acquisition by combining nyquist and compressive sampling. IEEE Trans Pattern Anal Mach Intell, 2018, 41: 857–870

    Article  Google Scholar 

  10. Wang L, Xiong Z, Shi G, et al. Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging. IEEE Trans Pattern Anal Mach Intell, 2016, 39: 2104–2111

    Article  Google Scholar 

  11. Dian R, Fang L, and Li S. Hyperspectral image super-resolution via non-local sparse tensor factorization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017. 5344–5353

  12. Qu Y, Qi H, Ayhan B, et al. Does multispectral/hyperspectral pansharpening improve the performance of anomaly detection? In: Proceedings of IEEE International Geoscience and Remote Sensing Symposium, 2017. 6130–6133

  13. Gomez-Chova L, Tuia D, Moser G, et al. Multimodal classification of remote sensing images: a review and future directions. Proc IEEE, 2015, 103: 1560–1584

    Article  Google Scholar 

  14. Ferraris V, Dobigeon N, Wei Q, et al. Robust fusion of multiband images with different spatial and spectral resolutions for change detection. IEEE Trans Comput Imag, 2017, 3: 175–186

    Article  MathSciNet  Google Scholar 

  15. Dian R, Li S, Sun B, et al. Recent advances and new guidelines on hyperspectral and multispectral image fusion. Inf Fusion, 2021, 69: 40–51

    Article  Google Scholar 

  16. Deng L J, Feng M, Tai X C. The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior. Inf Fusion, 2019, 52: 76–89

    Article  Google Scholar 

  17. Ma J, Yu W, Chen C, et al. Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Inf Fusion, 2020, 62: 110–120

    Article  Google Scholar 

  18. Meng X, Xiong Y, Shao F, et al. A large-scale benchmark data set for evaluating pansharpening performance: overview and implementation. IEEE Geosci Remote Sens Mag, 2021, 9: 18–52

    Article  Google Scholar 

  19. Meng X, Shen H, Yuan Q, et al. Pansharpening for cloud-contaminated very high-resolution remote sensing images. IEEE Trans Geosci Remote Sens, 2018, 57: 2840–2854

    Article  Google Scholar 

  20. Zhou Z M, Ma N, Li Y X, et al. Variational PCA fusion for Pan-sharpening very high resolution imagery. Sci China Inf Sci, 2014, 57: 112107

    Article  Google Scholar 

  21. Chen Z, Pu H, Wang B, et al. Fusion of hyperspectral and multispectral images: a novel framework based on generalization of pan-sharpening methods. IEEE Geosci Remote Sens Lett, 2014, 11: 1418–1422

    Article  Google Scholar 

  22. Selva M, Aiazzi B, Butera F, et al. Hyper-sharpening: a first approach on SIM-GA data. IEEE J Sel Top Appl Earth Obs Remote Sens, 2015, 8: 3008–3024

    Article  Google Scholar 

  23. Dian R, Li S, Fang L, et al. Multispectral and hyperspectral image fusion with spatial-spectral sparse representation. Inf Fusion, 2019, 49: 262–270

    Article  Google Scholar 

  24. Dong W, Fu F, Shi G, et al. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans Image Process, 2016, 25: 2337–2352

    Article  MathSciNet  Google Scholar 

  25. Han X H, Shi B, Zheng Y. Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Trans Image Process, 2018, 27: 5625–5637

    Article  MathSciNet  Google Scholar 

  26. Simoẽs M, Bioucas-Dias J, Almeida L B, et al. A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Trans Geosci Remote Sens, 2015, 53: 3373–3388

    Article  Google Scholar 

  27. Nascimento J M P, Bioucasdias J M B. Vertex component analysis: a fast algorithm to unmix hyperspectral data. IEEE Trans Geosci Remote Sens, 2005, 43: 898–910

    Article  Google Scholar 

  28. Zhou Y, Feng L, Hou C, et al. Hyperspectral and multispectral image fusion based on local low rank and coupled spectral unmixing. IEEE Trans Geosci Remote Sens, 2017, 55: 5997–6009

    Article  Google Scholar 

  29. Liu J, Wu Z, Xiao L, et al. A truncated matrix decomposition for hyperspectral image super-resolution. IEEE Trans Image Process, 2020, 29: 8028–8042

    Article  MathSciNet  Google Scholar 

  30. Wu R, Ma W K, Fu X, et al. Hyperspectral super-resolution via global-local low-rank matrix estimation. IEEE Trans Geosci Remote Sens, 2020, 58: 7125–7140

    Article  Google Scholar 

  31. Ren K, Sun W, Meng X, et al. A locally optimized model for hyperspectral and multispectral images fusion. IEEE Trans Geosci Remote Sens, 2022, 60: 1–15

    Google Scholar 

  32. Dian R, Li S, Fang L, et al. Nonlocal sparse tensor factorization for semiblind hyperspectral and multispectral image fusion. IEEE Trans Cybern, 2019, 50: 4469–4480

    Article  Google Scholar 

  33. Li S, Dian R, Fang L, et al. Fusing hyperspectral and multispectral images via coupled sparse tensor factorization. IEEE Trans Image Process, 2018, 27: 4118–4130

    Article  MathSciNet  Google Scholar 

  34. Chang Y, Yan L, Zhao X L, et al. Weighted low-rank tensor recovery for hyperspectral image restoration. IEEE Trans Cybern, 2020, 50: 4558–4572

    Article  Google Scholar 

  35. Xu Y, Wu Z, Chanussot J, et al. Nonlocal coupled tensor CP decomposition for hyperspectral and multispectral image fusion. IEEE Trans Geosci Remote Sens, 2019, 58: 348–362

    Article  Google Scholar 

  36. Liu N, Li L, Li W, et al. Hyperspectral restoration and fusion with multispectral imagery via low-rank tensor-approximation. IEEE Trans Geosci Remote Sens, 2021, 59: 7817–7830

    Article  Google Scholar 

  37. He W, Chen Y, Yokoya N, et al. Hyperspectral super-resolution via coupled tensor ring factorization. 2021. ArXiv:2001.01547

  38. Chen Y, Zeng J, He W, et al. Hyperspectral and multispectral image fusion using factor smoothed tensor ring decomposition. IEEE Trans Geosci Remote Sens, 2022, 60: 1–17

    Google Scholar 

  39. Fu X, Wang W, Huang Y, et al. Deep multiscale detail networks for multiband spectral image sharpening. IEEE Trans Neural Netw Learn Syst, 2021, 32: 2090–2104

    Article  Google Scholar 

  40. Dong W, Zhang T, Qu J, et al. Laplacian pyramid dense network for hyperspectral pansharpening. IEEE Trans Geosci Remote Sens, 2022, 60: 1–13

    Google Scholar 

  41. Hu J F, Huang T Z, Deng L J, et al. Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks. IEEE Trans Neural Netw Learn Syst, 2022, 33: 7251–7265

    Article  Google Scholar 

  42. Wang Z, Chen B, Lu R, et al. FusionNet: an unsupervised convolutional variational network for hyperspectral and multispectral image fusion. IEEE Trans Image Process, 2020, 29: 7565–7577

    Article  Google Scholar 

  43. Wang W, Fu X, Zeng W, et al. Enhanced deep blind hyperspectral image fusion. IEEE Trans Neural Netw Learn Syst, 2021. doi: https://doi.org/10.1109/TNNLS.2021.3105543

  44. Wang W, Zeng W, Huang Y, et al. Deep blind hyperspectral image fusion. In: Proceedings of IEEE/CVF International Conference on Computer Vision, 2019. 4149–4158

  45. Dian R, Li S, Guo A, et al. Deep hyperspectral image sharpening. IEEE Trans Neural Netw Learn Syst, 2018, 29: 5345–5355

    Article  MathSciNet  Google Scholar 

  46. Zheng Y, Li J, Li Y, et al. Edge-conditioned feature transform network for hyperspectral and multispectral image fusion. IEEE Trans Geosci Remote Sens, 2022, 60: 1–15

    Article  Google Scholar 

  47. Dong W, Zhou C, Wu F, et al. Model-guided deep hyperspectral image super-resolution. IEEE Trans Image Process, 2021, 30: 5754–5768

    Article  Google Scholar 

  48. Xie Q, Zhou M, Zhao Q, et al. MHF-Net: an interpretable deep network for multispectral and hyperspectral image fusion. IEEE Trans Pattern Anal Mach Intell, 2022, 44: 1457–1473

    Article  Google Scholar 

  49. Xie Q, Zhou M, Zhao Q, et al. Multispectral and hyperspectral image fusion by MS/HS fusion net. In: Proceedings of Conference on Computer Vision and Pattern Recognition, 2019. 1585–1594

  50. Yang J, Zhao Y Q, Chan J. Hyperspectral and multispectral image fusion via deep two-branches convolutional neural network. Remote Sens, 2018, 10: 800

    Article  Google Scholar 

  51. Dian R, Li S, Kang X. Regularizing hyperspectral and multispectral image fusion by CNN denoiser. IEEE Trans Neural Netw Learn Syst, 2021, 32: 1124–1135

    Article  Google Scholar 

  52. Yang J, Xiao L, Zhao Y Q, et al. Variational regularization network with attentive deep prior for hyperspectral-multispectral image fusion. IEEE Trans Geosci Remote Sens, 2022, 60: 1–17

    Google Scholar 

  53. Zheng K, Gao L, Hong D, et al. NonRegSRNet: a nonrigid registration hyperspectral super-resolution network. IEEE Trans Geosci Remote Sens, 2022, 60: 1–16

    Article  Google Scholar 

  54. Huang T, Dong W, Wu J, et al. Deep hyperspectral image fusion network with iterative spatio-spectral regularization. IEEE Trans Comput Imag, 2022, 8: 201–214

    Article  Google Scholar 

  55. Dian R, Li S. Hyperspectral image super-resolution via subspace-based low tensor multi-rank regularization. IEEE Trans Image Process, 2019, 28: 5135–5146

    Article  MathSciNet  Google Scholar 

  56. Zhuang L, Bioucas-Dias J M. Fast hyperspectral image denoising and inpainting based on low-rank and sparse representations. IEEE J Sel Top Appl Earth Obs Remote Sens, 2018, 11: 730–742

    Article  Google Scholar 

  57. Zhuang L, Ng M K, Fu X, et al. Hy-demosaicing: hyperspectral blind reconstruction from spectral subsampling. IEEE Trans Geosci Remote Sens, 2022, 60: 1–15

    Google Scholar 

  58. Boyd S. Distributed optimization and statistical learning via the alternating direction method of multipliers. FNT Machine Learn, 2011, 3: 1–122

    Article  Google Scholar 

  59. Dian R, Li S, Fang L. Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Trans Neural Netw Learn Syst, 2019, 30: 2672–2683

    Article  MathSciNet  Google Scholar 

  60. Wei Q, Dobigeon N, Tourneret J Y. Fast fusion of multi-band images based on solving a sylvester equation. IEEE Trans Image Process, 2015, 24: 4109–4121

    Article  MathSciNet  Google Scholar 

  61. Liu M, Tuzel O, Ramalingam S, et al. Entropy rate superpixel segmentation. In: Proceedings of Conference on Computer Vision and Pattern Recognition, 2011. 2097–2104

  62. Zhang K, Zuo W, Zhang L. FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans Image Process, 2018, 27: 4608–4622

    Article  MathSciNet  Google Scholar 

  63. Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015. 448–456

  64. Nair V, Hinton G E. Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, 2010. 807–814

  65. Dell’Acqua F, Gamba P, Ferrari A, et al. Exploiting spectral and spatial information in hyperspectral urban data with high resolution. IEEE Geosci Remote Sens Lett, 2004, 1: 322–326

    Article  Google Scholar 

  66. Green R O, Eastwood M L, Sarture C M, et al. Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (AVIRIS). Remote Sens Environ, 1998, 65: 227–248

    Article  Google Scholar 

  67. Aiazzi B, Baronti S, Selva M. Improving component substitution pansharpening through multivariate regression of MS +Pan data. IEEE Trans Geosci Remote Sens, 2007, 45: 3230–3239

    Article  Google Scholar 

  68. Yokoya N, Yairi T, Iwasaki A. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans Geosci Remote Sens, 2012, 50: 528–537

    Article  Google Scholar 

  69. Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process, 2004, 13: 600–612

    Article  Google Scholar 

  70. Wald L. Quality of high resolution synthesised images: is there a simple criterion? In: Proceedings of the 3rd Conference Fusion of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images, 2000. 99–103

  71. Wang Z, Bovik A C. A universal image quality index. IEEE Signal Process Lett, 2002, 9: 81–84

    Article  Google Scholar 

  72. Yokoya N, Grohnfeldt C, Chanussot J. Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geosci Remote Sens Mag, 2017, 5: 29–56

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Key Research and Development Program of China (Grant No. 2021YFA0715203), National Natural Science Foundation of China (Grant Nos. 62201205, 62221002, 61890962), Key Laboratory of Visual Perception and Artificial Intelligence Fund of Hunan Province (Grant No. 2018TP1013), and Changsha Natural Science Foundation of China (Grant No. kq2202170).

Author information

Authors and Affiliations

  1. College of Electrical and Information Engineering, Hunan University, Changsha, 410082, China

    Shutao Li

  2. School of Robotics, Hunan University, Changsha, 410082, China

    Renwei Dian & Haibo Liu

Authors
  1. Shutao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Renwei Dian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haibo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Renwei Dian.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Dian, R. & Liu, H. Learning the external and internal priors for multispectral and hyperspectral image fusion. Sci. China Inf. Sci. 66, 140303 (2023). https://doi.org/10.1007/s11432-022-3610-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-022-3610-5

Keywords

Search

Navigation