Abstract
The remote sensing applications demand the imaging products composed of rich spatial details and high spectral resolution, namely high resolution multispectral (HRMS) images which can not be generated by current generation satellite sensors due to technological and physical constraints. Pansharpening is an image fusion approach that combines the rich geometric details of the panchromatic image and the color information from the multispectral image into a single product. Inspite of the availability of numerous pansharpening techniques, there is a necessity to develop an effective mechanism for the balanced enhancement of spatial and spectral quality in the fused HRMS image. In this paper, a constrained optimization approach for multispectral and PAN image fusion (COAMP) is proposed. The total variation is an efficient operator to extract the geometric features of an image. The spatial differences are exported into the HRMS image with the aid of TV based gradient operator. The spectral information is transferred to the fused image using spectral vectors obtained by taking gradients in the third dimension along the bands of the multispectral image. The optimization problem is formulated and minimized using an efficient solver to get the HRMS image. The comprehensive experiments performed on three different datasets at both reduced scale and full scale resolutions. Additionally, quantitative assessment is also performed using six well-known metrics. The experiments approves the efficacy of the proposed COAMP model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F., & Selva, M. (2008). Multispectral and panchromatic data fusion assessment without reference. Photogrammetric Engineering & Remote Sensing, 74(2), 193–200. https://doi.org/10.14358/PERS.74.2.193
Ballester, C., Caselles, V., Igual, L., Verdera, J., & Rougé, B. (2006). A variational model for p+ xs image fusion. International Journal of Computer Vision, 69(1), 43–58. https://doi.org/10.1007/s11263-006-6852-x
Cheng, M., Wang, C., & Li, J. (2014). Sparse representation based pansharpening using trained dictionary. IEEE Geoscience and Remote Sensing Letters, 11(1), 293–297. https://doi.org/10.1109/LGRS.2013.2256875
Deng, W., & Yin, W. (2016). On the global and linear convergence of the generalized alternating direction method of multipliers. Journal of Scientific Computing, 66(3), 889–916. https://doi.org/10.1007/s10915-015-0048-x
Fei, R., Zhang, J., Liu, J., Fang, D., Chang, P., & Junying, H. (2019). Convolutional sparse representation of injected details for pansharpening. IEEE Geoscience and Remote Sensing Letters, 16(10), 1595–1599. https://doi.org/10.1109/LGRS.2019.2904526
Garzelli, A., Nencini, F., & Capobianco, L. (2007). Optimal mmse pan sharpening of very high resolution multispectral images. IEEE Transactions on Geoscience and Remote Sensing, 46(1), 228–236. https://doi.org/10.1109/TGRS.2007.907604
Gogineni, R., & Chaturvedi, A. (2019). A robust pansharpening algorithm based on convolutional sparse coding for spatial enhancement. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(10), 4024–4037. https://doi.org/10.1109/JSTARS.2019.2945815
Gogineni, R., & Sangani, D. J. (2022). A two-stage pan-sharpening algorithm based on sparse representation for spectral distortion reduction. International Journal of Image and Graphics, 22(01), 2250007. https://doi.org/10.1142/S0219467822500073
Gogineni, R., Chaturvedi, A., & Daya Sagar, B. S. (2021). A variational pan-sharpening algorithm to enhance the spectral and spatial details. International Journal of Image and Data Fusion, 12(3), 242–264. https://doi.org/10.1080/19479832.2020.1838629
Jon, K., Liu, J., Deng, L.-J., & Zhu, W. (2022). P3net: Pansharpening via pyramidal detail injection with deep physical constraints. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–18. https://doi.org/10.1109/TGRS.2022.3214209
Laben, CA., & Brower, BV. (2000). Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. eastman kodak company, rochester. NY US Patent, 6011875. https://patents.google.com/patent/US6011875A/en
Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., & Chanussot, J. (2022). Deep learning in multimodal remote sensing data fusion: A comprehensive review. International Journal of Applied Earth Observation and Geoinformation, 112, 102926. https://doi.org/10.1016/j.jag.2022.102926
Li, S., & Yang, B. (2010). A new pan-sharpening method using a compressed sensing technique. IEEE Transactions on Geoscience and Remote Sensing, 49(2), 738–746. https://doi.org/10.1109/TGRS.2010.2067219
Liu, P. (2019). A new total generalized variation induced spatial difference prior model for variational pansharpening. Remote Sensing Letters, 10(7), 659–668. https://doi.org/10.1080/2150704X.2019.1597299
Liu, Q., Meng, X., Shao, F., & Li, S. (2023). Supervised-unsupervised combined deep convolutional neural networks for high-fidelity pansharpening. Information Fusion, 89, 292–304. https://doi.org/10.1016/j.inffus.2022.08.018
Masi, G., Cozzolino, D., Verdoliva, L., & Scarpa, G. (2016). Pansharpening by convolutional neural networks. Remote Sensing, 8(7), 594. https://doi.org/10.3390/rs8070594
Otazu, X., González-Audícana, M., Fors, O., & Núñez, J. (2005). Introduction of sensor spectral response into image fusion methods. application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing, 43(10), 2376–2385. https://doi.org/10.1109/TGRS.2005.856106
Panchal, S., & Thakker, R. A. (2017). Improved image pansharpening technique using nonsubsampled contourlet transform with sparse representation. Journal of the Indian Society of Remote Sensing, 45(3), 385–394. https://doi.org/10.1007/s12524-016-0608-z
Rajput, U. K., Ghosh, S. K., & Kumar, A. (2014). Multisensor fusion of satellite images for urban information extraction using pseudo-wigner distribution. Journal of Applied Remote Sensing, 8(1), 083668–083668. https://doi.org/10.1117/1.JRS.8.083668
Sangani, D. J., Thakker, R. A., Panchal, S. D., & Gogineni, R. (2021). Pansharpening of satellite images with convolutional sparse coding and adaptive pcnn-based approach. Journal of the Indian Society of Remote Sensing, 49, 2989–3004. https://doi.org/10.1007/s12524-021-01440-4
Shi, W., ChaoBen, D., Gao, B. B., & Yan, J. N. (2021). Remote sensing image fusion using multi-scale convolutional neural network. Journal of the Indian Society of Remote Sensing, 49, 1677–1687. https://doi.org/10.1007/s12524-021-01353-2
Tambe, R. G., Talbar, S. N., & Chavan, S. S. (2021). Fusion of multispectral and panchromatic images by integrating standard pca with rotated wavelet transform. Journal of the Indian Society of Remote Sensing, 49(9), 2033–2055. https://doi.org/10.1007/s12524-021-01373-y
Tian, X., Chen, Y., Yang, C., & Ma, J. (2021). Variational pansharpening by exploiting cartoon-texture similarities. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–16. https://doi.org/10.1109/TGRS.2020.3048257
Te-Ming, T., Huang, P. S., Hung, C.-L., & Chang, C.-P. (2004). A fast intensity-hue-saturation fusion technique with spectral adjustment for ikonos imagery. IEEE Geoscience and Remote Sensing Letters, 1(4), 309–312. https://doi.org/10.1109/LGRS.2004.834804
Upla, Kishor P., Gajjar, Prakash P., & Joshi, Manjunath V. (2013). Pan-sharpening based on non-subsampled contourlet transform detail extraction. In 2013 fourth national conference on computer vision, pattern recognition, image processing and Graphics (NCVPRIPG), pages 1–4. IEEE. https://doi.org/10.1109/NCVPRIPG.2013.6776258
Vicinanza, M. R., Restaino, R., Vivone, G., Mura, M. D., & Chanussot, J. (2015). A pansharpening method based on the sparse representation of injected details. IEEE Geoscience and Remote Sensing Letters, 12(1), 180–184. https://doi.org/10.1109/LGRS.2014.2331291
Vivone, G., Alparone, L., Chanussot, J., Mura, M. D., Garzelli, A., Licciardi, G. A., Restaino, R., & Wald, L. (2015). A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, 53(5), 2565–2586. https://doi.org/10.1109/TGRS.2014.2361734
Wald, L., Ranchin, T., & Mangolini, M. (1997). Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63(6), 691–699. https://hal.science/hal-00365304
Xiao, J. L., Huang, T. Z., Deng, L. J., Wu, Z. C., Wu, X., & Vivone, G. (2023). Variational pansharpening based on coefficient estimation with nonlocal regression. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2023.3305296
Yilmaz, C. S., Yilmaz, V., & Gungor, O. (2022). A theoretical and practical survey of image fusion methods for multispectral pansharpening. Information Fusion, 79, 1–43. https://doi.org/10.1016/j.inffus.2021.10.001
Zhang, K., Zhang, F., Wan, W., Hui, Yu., Sun, J., Del Ser, J., & Elyan, E., Hussain, A., (2023). Panchromatic and multispectral image fusion for remote sensing and earth observation: Concepts, taxonomy, literature review, evaluation methodologies and challenges ahead. Information Fusion,93, 227–242 https://doi.org/10.1016/j.inffus.2022.12.026
Acknowledgements
I would like to express my gratitude to my research supervisor Dr. Christeena joseph, and also the coauthor Dr. Mandava Venka Subbarao for their valuable guidance and providing support in writing and developing the manuscript.
Funding
The authors declared that this work is not funded by any organization.
Author information
Authors and Affiliations
Contributions
All the three authors are equally contributed for the literature survey and problem formulation. The method was formulated by first author(Darisi Girish Kumar), the qualitative results was generated by second author (Christeena Josph) and the quantitative results are generated by third author (Mandava venkata subbarao). All the three authors reviewed and approved the final manuscript that was submitted.
Corresponding author
Ethics declarations
Conflict of interest
All the three authors are declared that they have no Conflict of interest to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Kumar, D.G., Joseph, C. & Subbarao, M.V. Constrained Optimization Guided Approach for Multispectral and Panchromatic Image Fusion. J Indian Soc Remote Sens (2024). https://doi.org/10.1007/s12524-024-01876-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s12524-024-01876-4