Skip to main content
SpringerLink
Log in

Joint image pansharpening and registration via structure tensor total variation regularization

  • Original Paper
  • Published:
Aerospace Systems Aims and scope Submit manuscript

Abstract

Pansharpening has been extensively studied in recent years. However, one drawback of the known image fusion methods is that the fusion performance is degraded by the registration error. We develops a variational framework for joint pansharpening and registration with structure tensor total variation regularization method. The proposed framework can fully capture the target image’s first-order information around a local neighborhood and align image gradient domain. An efficient optimization method based on the scheme of fast iterative shrinkage-thresholding algorithm (FISTA) is proposed to solve the objective fusion model. This framework consists of two key steps: (i) pansharpening with structure tensor total variation regularization; (ii) image registration in image gradient domain. Extensive experiments demonstrate the effectiveness of our method compared with the existing state-of-the-art fusion models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability Statement

The data used to support the findings of this study are available from the corresponding author upon request.

References

  1. Zhu XX, Grohnfeldt C, Bamler R (2013) Collaborative sparse image fusion with application to pan-sharpening. In: 2013 18th International conference on digital signal processing (DSP), pp 1–6

  2. Zhu XX, Spiridonova S, Bamler R (2012) A pan-sharpening algorithm based on joint sparsity. In: Tyrrhenian workshop on advances in radar and remote sensing (TyWRRS), pp 177–184

  3. Vivone G, Alparone L, Chanussot J, Dalla Mura M, Garzelli A, Licciardi GA, Restaino R, Wald L (2014) A critical comparison among pansharpening algorithms. IEEE Trans Geosci Remote Sens 53(5):2565–2586. https://doi.org/10.1109/TGRS.2014.2361734

    Article  Google Scholar 

  4. Leung H, Mukhopadhyay SC (2015) Intelligent environmental sensing, vol 13. Springer, Berlin

    Book  Google Scholar 

  5. Zhongliang J, Han P, Yuankai L, Peng D (2018) Non-cooperative target tracking, fusion and control—algorithms and advances. Springer International Publishing, Berlin

    Google Scholar 

  6. Stathaki T (2011) Image fusion: algorithms and applications. Academic Press, Cambridge

    Google Scholar 

  7. Wei Q, Dobigeon N, Tourneret J-Y (2015) Fast fusion of multi-band images based on solving a Sylvester equation. IEEE Trans Image Process 24(11):4109–4121

    Article  MathSciNet  Google Scholar 

  8. Pan H, Jing Z, Qiao L, Li M (2018) Visible and infrared image fusion using 0-generalized total variation model. Sci China Inf Sci 61(4):049103

    Article  MathSciNet  Google Scholar 

  9. Pan H, Jing Z, Leung H and Li M (2021) Hyperspectral Image Fusion and Multitemporal Image Fusion by Joint Sparsity. In: IEEE Transactions on Geoscience and Remote Sensing, vol 59, no 9, pp 7887–7900

  10. Garzelli A, Capobianco L, Nencini F (2009) On the effects of pansharpening to target detection. In: 2009 IEEE international geoscience and remote sensing symposium, vol 2, pp II-136–II-139

  11. Ying Y, Han P, Zhongliang J (2019) Fusion object detection of satellite imagery with arbitrary-oriented region convolutional neural network. Aerospace Systems. 2. https://doi.org/10.1007/s42401-019-00033-x

  12. Bovolo F, Bruzzone L, Capobianco L, Garzelli A, Marchesi S, Nencini F (2010) Analysis of the effects of pansharpening in change detection on VHR images. IEEE Geosci Remote Sens Lett 7(1):53-57

    Article  Google Scholar 

  13. Palsson F, Sveinsson JR, Benediktsson JA, Aanaes H (2012) Classification of pansharpened urban satellite images. IEEE J Sel Top Appl Earth Obs Remote Sens 5(1):281–297

    Article  Google Scholar 

  14. Zhu X, Grohnfeldt C, Bamler R (2016) Exploiting joint sparsity for pansharpening: the J-SparseFI algorithm. IEEE Trans Geosci Remote Sens 54(5):2664–2681

    Article  Google Scholar 

  15. Vogel CR (2011) Computational methods for inverse problems. Society for Industrial and Applied Mathematics, Philadelphia

    Google Scholar 

  16. Shah VP, Younan NH, King RL (2008) An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Trans Geosci Remote Sens 46(5):1323–1335

    Article  Google Scholar 

  17. Rahmani S, Strait M, Merkurjev D, Moeller M, Wittman T (2010) An adaptive IHS pan-sharpening method. IEEE Geosci Remote Sens Lett 7(4):746–750

    Article  Google Scholar 

  18. Nunez J, Otazu X, Fors O, Prades A, Pala V, Arbiol R (1999) Multiresolution-based image fusion with additive wavelet decomposition. IEEE Trans Geosci Remote Sens 37(3):1204–1211

    Article  Google Scholar 

  19. Otazu X, Gonzalez-Audicana M, Fors O, Nunez J (2005) Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Trans Geosci Remote Sens 43(10):2376–2385

    Article  Google Scholar 

  20. C C, Y L, W L, Sirf HJ (2015) Simultaneous satellite image registration and fusion in a unified framework. IEEE Trans Image Process 24(11):4213–4224

    Article  MathSciNet  Google Scholar 

  21. Chen C, Li Y, Liu W, Huang J (2014) Image fusion with local spectral consistency and dynamic gradient sparsity. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 2760–2765

  22. Jun L, Han P, Shuqing C, Zhongliang J (2021) Multiband image fusion using total generalized variation regularization. Aerospace Systems 4:261–267

    Article  Google Scholar 

  23. Parisotto S, Lellmann J, Masnou S, Schonlieb C-B (2020) Higher-order total directional variation: imaging applications. SIAM J Imaging Sci 13(4):2063–2104

    Article  MathSciNet  Google Scholar 

  24. Lefkimmiatis S, Roussos A, Maragos P, Unser M (2015) Structure tensor total variation. SIAM J Imaging Sci 8(2):1090–1122

    Article  MathSciNet  Google Scholar 

  25. Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202

    Article  MathSciNet  Google Scholar 

  26. Zhou J, Civco DL, Silander JA (1998) A wavelet transform method to merge Landsat TM and SPOT panchromatic data. Int J Remote Sens 19(4):743–757

    Article  Google Scholar 

  27. Gillespie AR, Kahle AB, Walker RE (1986) Color enhancement of highly correlated images, decorrelation and HSI contrast stretches. Remote Sens Environ 20(3):209–235

    Article  Google Scholar 

  28. Haydn R, Dalke GW, Henkel J, Bare JE (1982) Application of the IHS color transform to the processing of multisensor data and image enhancement. In: Proceedings of the international symposium on remote sensing of environment

  29. Simões M, Bioucas-Dias J, Almeida LB, Chanussot J (2015) A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Trans Geosci Remote Sens 53(6):3373–3388

    Article  Google Scholar 

  30. Jagalingam P, Hegde AV (2015) A review of quality metrics for fused image. Aquat Procedia 4:133–142

    Article  Google Scholar 

  31. Zhang L, Zhang L, Mou X, Zhang D (2011) FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Process 20(8):2378–2386. https://doi.org/10.1109/TIP.2011.2109730

    Article  MathSciNet  MATH  Google Scholar 

  32. Wang Z, Bovik AC (2006) Modern Image Quality Assessment. Synthesis Lectures on Image, Video, and Multimedia Processing, vol 2, no 1, pp 1–156

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant nos. 61603249, 61673262, and the Shanghai Sailing Program (20YF1417300).

Funding

This work was jointly supported by the National Natural Science Foundation of China (Grant nos. 61603249, 61673262) and Shanghai Sailing Program (20YF1417300).

Author information

Authors and Affiliations

  1. School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai, 200240, China

    Yu Yuan, Han Pan & Zhong-liang Jing

  2. Shanghai Key Laboratory of Aerospace Intelligent Control Technology, Shanghai Aerospace Control Technology Institute, Shanghai, 201109, China

    Shu-qing Cao

Authors
  1. Yu Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Han Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shu-qing Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhong-liang Jing
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yu Yuan wrote the manuscript and performed the experiment and data analyses. Zhongliang Jing, Han Pan, and Shuqing Cao contributed to the conception of the study and review of the manuscript.

Corresponding author

Correspondence to Zhong-liang Jing.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical statement

This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yuan, Y., Pan, H., Cao, Sq. et al. Joint image pansharpening and registration via structure tensor total variation regularization. AS 5, 277–283 (2022). https://doi.org/10.1007/s42401-022-00138-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42401-022-00138-w

Keywords

Search

Navigation