Abstract
In recent years, the pansharpening strategies employing the Generalized Laplacian Pyramid (GLP) based on Gaussian filters that match the Modulation Transfer Function (MTF) of the source multispectral (MS) sensor have attracted attention in remote sensing community. The MTF-GLP-based pansharpening methods differ from each other in the way they obtain the injection coefficients, which are used to transfer the spatial details of the source panchromatic (PAN) image into the source MS image. Investigation of the pansharpening literature showed that the MTF-GLP-based pansharpening strategies generally estimate the injection coefficients using statistics-based deterministic approaches, which leads to a difficulty in identifying the non-linear relationship between the source MS and PAN data. Hence, this study proposes a metaheuristic optimization-based solution to this problem. The proposed method estimates the optimum injection coefficients through the Multi-Objective Symbiotic Organism Search (MOSOS) algorithm, which has been proven to efficiently find the optimum solutions in very complex search spaces. The success of the presented method was qualitatively and quantitatively tested on four test sites against several widely used pansharpening techniques. The experiments revealed that the presented approach did not only outperform some of the commonly used MTF-GLP-based methods, but also some of the other Multiresolution Analysis (MRA)-based, component substitution (CS)-based, deep learning (DL)-based, and variational optimization (VO)-based pansharpening methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aiazzi B, Alparone L, Baronti S, Garzelli A (2002) Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Trans Geosci Remote Sens 40(10):2300–2312. https://doi.org/10.1109/TGRS.2002.803623
Aiazzi B, Baronti S, Selva M (2007) Improving component substitution pansharpening through multivariate regression of MS + Pan data. IEEE Trans Geosci Remote Sens 45(10):3230–3239. https://doi.org/10.1109/TGRS.2007.901007
Alparone L, Baronti S, Garzell A, Nencini F (2004) A global quality measurement of pan-sharpened multispectral imagery. IEEE Geosci Remote Sens Lett 1(4):313–317. https://doi.org/10.1109/LGRS.2004.836784
Alparone L, Wald L, Chanussot J, Thomas C, Gamba P, Bruce LM (2007) Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest. IEEE Trans Geosci Remote Sens 45(10):3012–3021. https://doi.org/10.1109/TGRS.2007.904923
Alparone L, Aiazz B, Baronti S, Garzelli A, Nencini F, Selva M (2008) Multispectral and panchromatic data fusion assessment without reference. Photogramm Eng Remote Sens 74(2):193–200. https://doi.org/10.14358/PERS.74.2.193
Ballester C, Caselles V, Igual L, Verdera J, Rougé B (2006) A variational model for P+XS image fusion. Int J Comput vis 69(1):43–58. https://doi.org/10.1007/s11263-006-6852-x
Burt PJ, Adelson EH (1983) The Laplacian pyramid as a compact image code. IEEE Trans Commun COM-31 4:532–540
Censor Y (1977) Pareto optimality in multiobjective problems. Appl Math Optim 4(1):41–59. https://doi.org/10.1007/BF01442131
Chen CM, Hepner GF, Forster RR (2003) Fusion of hyperspectral and radar data using the IHS transformation to enhance urban surface features. ISPRS J Photogramm Remote Sens 58(1):19–30. https://doi.org/10.1016/S0924-2716(03)00014-5
Chen C, Li Y, Liu W, Huang J (2015) SIRF: Simultaneous satellite image registration and fusion in a unified framework. IEEE Trans Image Process 24(11):4213–4224. https://doi.org/10.1109/TIP.2015.2456415
Chen L, Zhang X, Ma H (2018) Sparse representation over shared coefficients in multispectral pansharpening. Tsinghua Sci Technol 23(3):315–322. https://doi.org/10.26599/TST.2018.9010088
Cheng MY, Prayogo D (2014) Symbiotic organisms search: a new metaheuristic optimization algorithm. Comput Struct 139:98–112. https://doi.org/10.1016/j.compstruc.2014.03.007
Deb K (2001) Multi-objective optimization using evolutionary algorithms. Wiley, Hoboken
Deb K, Pratap A, Agarwal S, Meyarivan TAMT (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6(2):182–197. https://doi.org/10.1109/4235.996017
Do MN, Vetterli M (2005) The contourlet transform: an efficient directional multiresolution image representation. IEEE Trans Image Process 14(12):2091–2106. https://doi.org/10.1109/TIP.2005.859376
Fei R, Zhang J, Liu J, Du F, Chang P, Hu J (2019) Convolutional sparse representation of injected details for pansharpening. IEEE Geosci Remote Sens Lett 16(10):1595–1599. https://doi.org/10.1109/LGRS.2019.2904526
Garzelli A, Nencini F (2006) PAN-sharpening of very high resolution multispectral images using genetic algorithms. Int J Remote Sens 27(15):3273–3292. https://doi.org/10.1080/01431160600554991
Garzelli A, Nencini F (2009) Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geosci Remote Sens Lett 6(4):662–665. https://doi.org/10.1109/LGRS.2009.2022650
Garzelli A, Nencini F, Capobianco L (2008) Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Trans Geosci Remote Sens 46(1):228–236. https://doi.org/10.1109/TGRS.2007.907604
González-Audícana M, Saleta JL, Catalán RG, García R (2004) Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition. IEEE Trans Geosci Remote Sens 42(6):1291–1299. https://doi.org/10.1109/TGRS.2004.825593
Khademi G, Ghassemian H (2017) A multi-objective component-substitution-based pansharpening. In: 2017 IEEE 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA), pp 248–252. https://doi.org/10.1109/PRIA.2017.7983056.
Kwarteng P, Chavez A (1989) Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogramm Eng Remote Sens 55(1):339–348
Laben C A, Brower BV (2000) Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. U.S. Patent No: 6,011,875
Lang W, Zhao Z, Fang S, Cao Y, Wang Y (2020) Sparse representation-based detail-injection method for pan-sharpening. J Appl Remote Sens 14(2):026523. https://doi.org/10.1117/1.JRS.14.026523
Ling Y, Ehlers M, Usery EL, Madden M (2007) FFT-enhanced IHS transform method for fusing high-resolution satellite images. ISPRS J Photogramm Remote Sens 61(6):381–392. https://doi.org/10.1016/j.isprsjprs.2006.11.002
Liu Q, Zhou H, Xu Q, Liu X, Wang Y (2020) PSGAN: A generative adversarial network for remote sensing image pan-sharpening. IEEE Trans Geosci Remote Sens 59(12):10227–10242. https://doi.org/10.1109/TGRS.2020.3042974
Mallat SG (1989) A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans Pattern Anal Mach Intell 11(7):674–693. https://doi.org/10.1109/34.192463
Masi G, Cozzolino D, Verdoliva L, Scarpa G (2016) Pansharpening by convolutional neural networks. Remote Sens 8(7):594. https://doi.org/10.3390/rs8070594
Mirjalili S, Dong JS (2020) Multi-objective optimization using artificial intelligence techniques. Springer, New York
Munechika CK, Warnick JS, Salvaggio C, Schott JR (1993) Resolution enhancement of multispectral image data to improve classification accuracy. Photogram Eng Remote Sens 59(1):67–72
Nason GP, Silverman BW (1995) The stationary wavelet transform and some statistical applications. Wavelets and statistics. Springer, New York, pp 281–299
Núñez J, Otazu X, Fors O, Prades A, Pala V, Arbiol R (1999) Multiresolution-based image fusion with additive wavelet decomposition. IEEE Trans Geosci Remote Sens 37(3):1204–1211. https://doi.org/10.1109/36.763274
Otazu X, González-Audícana M, Fors O, Núñez J (2005) Introduction of sensor spectral response into image fusion methods, Application to wavelet-based methods. IEEE Trans Geosci Remote Sens 43(10):2376–2385. https://doi.org/10.1109/TGRS.2005.856106
Ozcelik F, Alganci U, Sertel E, Unal G (2021) Rethinking CNN-based pansharpening: guided colorization of panchromatic images via GANS. IEEE Trans Geosci Remote Sens 59(4):3486–3501. https://doi.org/10.1109/TGRS.2020.3010441
Padwick C, Deskevich M, Pacifici F, Smallwood S (2010) WorldView-2 pan-sharpening. In: Proceedings of the ASPRS 2010 Annual Conference. San Diego, CA, USA. pp 1–14
Palsson F, Sveinsson JR, Ulfarsson MO (2014) A new pansharpening algorithm based on total variation. IEEE Geosci Remote Sens Lett 11(1):318–322. https://doi.org/10.1109/LGRS.2013.2257669
Palsson F, Sveinsson JR, Ulfarsson MO, Benediktsson JA (2015) Model-based fusion of multi-and hyperspectral images using PCA and wavelets. IEEE Trans Geosci Remote Sens 53(5):2652–2663. https://doi.org/10.1109/TGRS.2014.2363477
Palubinskas G (2014) Quality assessment of pan-sharpening methods. In: IEEE Geoscience and Remote Sensing Symposium, Quebec City, QC, Canada, pp 2526–2529. https://doi.org/10.1109/IGARSS.2014.6946987
Restaino R, Dalla Mura M, Vivone G, Chanussot J (2016) Context-adaptive pansharpening based on image segmentation. IEEE Trans Geosci Remote Sen 55(2):753–766. https://doi.org/10.1109/TGRS.2016.2614367
Saeedi J, Faez K (2011) A new pan-sharpening method using multiobjective particle swarm optimization and the shiftable contourlet transform. ISPRS J Photogramm Remote Sens 66(3):365–381. https://doi.org/10.1016/j.isprsjprs.2011.01.006
Scarpa G, Vitale S, Cozzolino D (2018) Target-adaptive CNN-based pansharpening. IEEE Trans Geosci Remote Sens 56(9):5443–5457. https://doi.org/10.1109/TGRS.2018.2817393
Serifoglu Yilmaz C, Yilmaz V, Gungor O, Shan J (2019) Metaheuristic pansharpening based on symbiotic organisms search optimization. ISPRS J Photogramm Remote Sens 158:167–187. https://doi.org/10.1016/j.isprsjprs.2019.10.014
Serifoglu Yilmaz C, Yilmaz V, Güngör O (2020) On the use of the SOS metaheuristic algorithm in hybrid image fusion methods to achieve optimum spectral fidelity. Int J Remote Sens 41(10):3993–4021. https://doi.org/10.1080/01431161.2019.1711244
Serifoglu Yilmaz C, Yilmaz V, Gungor O (2022) A theoretical and practical survey of image fusion methods for multispectral pansharpening. Inf Fusion 79:1–43. https://doi.org/10.1016/j.inffus.2021.10.001
Seshadri A (2006) A fast elitist multiobjective genetic algorithm: NSGA-II ToolBox, Technical report
Shahdoosti HR, Ghassemian H (2016) Combining the spectral PCA and spatial PCA fusion methods by an optimal filter. Inf Fusion 27:150–160. https://doi.org/10.1016/j.inffus.2015.06.006
Shensa MJ (1992) The discrete wavelet transform: wedding the a trous and Mallat algorithms. IEEE Trans Signal Process 40(10):2464–2482. https://doi.org/10.1109/78.157290
Starck JL, Fadili J, Murtagh F (2007) The undecimated wavelet decomposition and its reconstruction. IEEE Trans Image Process 16(2):297–309. https://doi.org/10.1109/TIP.2006.887733
Tian X, Chen Y, Yang C, Gao X, Ma J (2020) A variational pansharpening method based on gradient sparse representation. IEEE Signal Process Lett 27:1180–1184. https://doi.org/10.1109/LSP.2020.3007325
Tierney S, Gao J, Guo Y (2014) Affinity pansharpening and image fusion. In: IEEE 2014 International Conference on Digital Image Computing: Techniques and Applications (DICTA). pp 1–8. https://doi.org/10.1109/DICTA.2014.7008094
Tran DH, Cheng MY, Prayogo D (2016) A novel Multiple Objective Symbiotic Organisms Search (MOSOS) for time–cost–labor utilization tradeoff problem. Knowl Based Syst 94:132–145. https://doi.org/10.1016/j.knosys.2015.11.016
Tu TM, Su SC, Shyu HC, Huang PS (2001) A new look at IHS-like image fusion methods. Inf Fusion 2(3):177–186. https://doi.org/10.1016/S1566-2535(01)00036-7
Vicinanza MR, Restaino R, Vivone G, Dalla Mura M, Chanussot J (2015) A pansharpening method based on the sparse representation of injected details. IEEE Geosci Remote Sens Lett 12(1):180–184. https://doi.org/10.1109/LGRS.2014.2331291
Vivone G, Restaino R, Dalla Mura M, Licciardi G, Chanussot J (2014) Contrast and error-based fusion schemes for multispectral image pansharpening. IEEE Geosci Remote Sens Lett 11(5):930–934. https://doi.org/10.1109/LGRS.2013.2281996
Vivone G, Alparone L, Chanussot J, Dalla Mura M, Garzelli A, Licciardi GA, Restaino R, Wald L (2015) A critical comparison among pansharpening algorithms. IEEE Trans Geosci Remote Sens 53:2565–2586. https://doi.org/10.1109/TGRS.2014.2361734
Vivone G, Restaino R, Chanussot J (2018a) A regression-based high-pass modulation pansharpening approach. IEEE Trans Geosci Remote Sens 56(2):984–996. https://doi.org/10.1109/TGRS.2017.2757508
Vivone G, Restaino R, Chanussot J (2018b) Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Trans Image Process 27(7):3418–3431. https://doi.org/10.1109/TIP.2018.2819501
Vivone G, Marano S, Chanussot J (2020) Pansharpening: context-based generalized laplacian pyramids by robust regression. IEEE Trans Geosci Remote Sens 58(9):6152–6167. https://doi.org/10.1109/TGRS.2020.2974806
Vivone G, Dalla Mura M, Garzelli A, Restaino R, Scarpa G, Ulfarsson MO, Alparone L, Chanusso J (2021) A new benchmark based on recent advances in multispectral pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods. IEEE Geosci Remote Sens Mag 9(1):53–81. https://doi.org/10.1109/MGRS.2020.3019315
Wald L (2000) Quality of high resolution synthesised images: Is there a simple criterion? In: Third conference" Fusion of Earth data: merging point measurements, raster maps and remotely sensed images". SEE/URISCA, pp 99–105
Wald L, Ranchin T, Mangolini M (1997) Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images. Photogramm Eng Remote Sens 63(6):691–699
Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Proc Let 9(3):81–84. https://doi.org/10.1109/97.995823
Wang P, Sertel E (2021) Channel–spatial attention-based pan-sharpening of very high-resolution satellite images. Knowl Based Syst 229:107324. https://doi.org/10.1016/j.knosys.2021.107324
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861
Wang X, Bai S, Li Z, Song R, Tao J (2019) The PAN and MS image pansharpening algorithm based on adaptive neural network and sparse representation in the NSST domain. IEEE Access 7:52508–52521. https://doi.org/10.1109/ACCESS.2019.2910656
Wei Y, Yuan Q, Shen H, Zhang L (2017) Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geosci Remote Sens Lett 14(10):1795–1799. https://doi.org/10.1109/LGRS.2017.2736020
Xing Y, Wang M, Yang S, Jiao L (2018) Pan-sharpening via deep metric learning. ISPRS J Photogramm Remote Sens 145:165–183. https://doi.org/10.1016/j.isprsjprs.2018.01.016
Xu H, Ma J, Shao Z, Zhang H, Jiang J, Guo X (2021) SDPNet: a deep network for pan-sharpening with enhanced information representation. IEEE Trans Geosci Remote Sens 59(5):4120–4134. https://doi.org/10.1109/TGRS.2020.3022482
Yang XS (2011) Metaheuristic optimization. Scholarpedia 6:11472
Yilmaz V (2021) A Non-Dominated Sorting Genetic Algorithm-II-based approach to optimize the spectral and spatial quality of component substitution-based pansharpened images. Concurr Comput Pract Exp 33(5):e6030. https://doi.org/10.1002/cpe.6030
Yilmaz V, Serifoglu Yilmaz C, Güngör O, Shan J (2020) A genetic algorithm solution to the gram-schmidt image fusion. Int J Remote Sens 41(4):1458–1485. https://doi.org/10.1080/01431161.2019.1667553
Yilmaz V, Serifoglu Yilmaz C, Gungor O (2021) Genetic algorithm-based synthetic variable ratio image fusion. Geocarto Int 36(9):989–1006. https://doi.org/10.1080/10106049.2019.1629649
Yuan Q, Wei Y, Meng X, Shen H, Zhang L (2018) A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE J Sel Top Appl Earth Obs Remote Sens 11(3):978–989. https://doi.org/10.1109/JSTARS.2018.2794888
Zhang Y (1999) A new merging method and its spectral and spatial effects. Int J Remote Sens 20(10):2003–2014. https://doi.org/10.1080/014311699212317
Zhang H, Wang H, Tian X, Ma J (2023) P2Sharpen: A progressive pansharpening network with deep spectral transformation. Inf Fusion 91:103–122. https://doi.org/10.1016/j.inffus.2022.10.010
Zhou J, Civco DL, Silander JA (1998) A wavelet transform method to merge Landsat TM and SPOT panchromatic data. Int J Remote Sens 19(4):743–757. https://doi.org/10.1080/014311698215973
Acknowledgements
All pansharpening methods except the DRPNN were employed with the MATLAB scripts provided through https://openremotesensing.net/. The DRPNN method was applied using the MATLAB script provided by Wei et al. (2017) through https://github.com/Decri/DRPNN-Deep-Residual-Pan-sharpening-Neural-Network. We would like to thank Liu et al. (2020) for providing the source images of the site 2 through https://github.com/zhysora/PSGan-Family. We also thank the Department of Geomatics Engineering of Karadeniz Technical University for providing the imagery data for the other sites.
Funding
Not applicable.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Serifoglu Yilmaz, C., Gungor, O. A Metaheuristic Optimization-Based Solution to MTF-GLP-Based Pansharpening. PFG 91, 245–272 (2023). https://doi.org/10.1007/s41064-023-00248-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41064-023-00248-w