A spatiotemporal transferable image fusion technique for GeoEye-1 satellite imagery

This study proposed a novel technique to solve the problem of color distortion in the fusion of the GeoEye-1 satellite's panchromatic (PAN) and multispectral (MS) images. This technique suggested reducing the difference in radiometry between the PAN and MS images by using modification coefficients for the MS bands in the definition of the intensity (I) equation, which guarantees using only the overlapped wavelengths with the PAN band. These modification coefficients achieve spatiotemporal transferability for the proposed fusion technique. As the reflectance of vegetation is high in the NIR band and low in the RGB bands, this technique suggested using an additional coefficient for the NIR band in the definition of the I equation, which varies based on the ratio of the agricultural features within the image, to indicate the correct impact of vegetation. This vegetation coefficient provides stability for the proposed fusion technique across all land cover classes. This study used three datasets of GeoEye-1 satellite PAN and MS images in Tanta City, Egypt, with different land cover classes (agricultural, urban, and mixed areas), to evaluate the performance of this technique against five different standard image fusion techniques. In addition, it was validated using six additional datasets from different locations and acquired at different times to test its spatiotemporal transferability. The proposed fusion technique demonstrated spatiotemporal transferability as well as great efficiency in producing fused images of superior spatial and spectral quality for all types of land cover.


Introduction
Remote sensing aims to extract information about the Earth's features by interpreting the spectral data obtained from a distance [1]. Modern satellites provide images with various spatial, spectral, temporal, and radiometric resolutions. Merging the key characteristics of each image may generate a new image with more information than any of the input images [2]. There are two forms of satellite imagery: the panchromatic (PAN) image and the multispectral (MS) image. The key feature of the PAN image is the superior spatial resolution, which means that the small objects appear with high accuracy. The key feature of the MS image is the superior spectral resolution, which means that the objects appear in their correct colors [3].

B Mohamed Elshora
Mohammad.elShora@f-eng.tanta.edu.eg 1 Department of Public Works Engineering, Faculty of Engineering, Tanta University, Tanta 31511, Egypt Image fusion or pan-sharpening proposes enhancing the low spatial quality MS image by adding the details of the high spatial quality PAN image [4]. Recently, the fusion of PAN and MS images has become an important requirement for many remote sensing applications. The primary drawback of the existing fusion techniques is color distortion, which results from radiometric disparities between PAN and MS images. The proposed fusion technique aims to solve this problem by (1) modifying the MS bands based on their intersecting areas with the PAN band to make the PAN and MS bands have the same spectral wavelengths, and (2) using a variable coefficient for the NIR band to indicate the correct reflectance of the NIR in the agricultural areas.
In this study, the performance of the proposed fusion technique was assessed by comparison with five different fusion techniques on three different datasets of GeoEye-1 PAN and MS images acquired in Tanta City, in the north part of Egypt, and covering different types of land cover (urban, agricultural, and mixed areas). These fusion techniques are fast-intensity-hue-saturation (FIHS), principal component analysis (PCA), Gram-Schmidt fusion (GS), hyper-spherical color space (HCS), and Ehlers fusion. Statistical analysis and visual examination were used to evaluate the output fused images and their correlation with the original PAN and MS images. In addition to the first three datasets from Tanta, Egypt, the proposed fusion technique was validated using six additional datasets from different locations, acquired at different times, and covering different land cover classes (urban, agricultural, and mixed) to test its spatiotemporal transferability.

Related work
Several image fusion techniques have been developed to merge the spectral characteristics of the MS image with the spatial characteristics of the PAN image to produce fused images of high spectral and spatial quality; some of these techniques are explained below.

FIHS fusion technique
The traditional IHS fusion technique depends on the transformation from the RGB color space to the IHS color space, which can separate the spectral characteristics in the hue (H) and saturation (S) components, and the spatial characteristics in the intensity (I) component [5]. The PAN image replaces the I component [6], followed by applying the reverse transformation from IHS color space to RGB color space for PAN, H, and S to create the fused image.
The fast IHS fusion technique [7] is a fast method of the traditional IHS fusion technique [8] that can be implemented according to the following equation: where, R', G', and B' are the fused bands, δ is the spatial details: δ (Pan -I), v 1 and v 2 are spectral characteristics variables, which are calculated based on the hue and saturation components: v 1 S cos (H) and v 2 S sin (H). The FIHS fusion technique is based on transforming the RGB bands into IHS components, then the I component is subtracted from the histogram-matched PAN image to get the spatial details, which will be added to the original MS bands by simple addition to generate the fused image.
Due to the differences in radiance between the PAN and I image, the fused image may contain color distortion. It was suggested to add the NIR band into the I equation to minimize the radiance difference between PAN, and I images. The formula for deriving the I component is:

PCA fusion technique
This technique utilizes the principal component transformation to transfer the correlated MS bands into uncorrelated principal components (PCs) [9]. The uncorrelated PCs have variance information from the original MS bands [10]. This technique supposes that the first PC1 component, which has the highest variance, contains the overall luminance and is close to the PAN image [11]. Hence, PC1 is replaced by the histogram-matched PAN image, and the inverse transformation from PCA to RGB is implemented for the histogram-matched PAN band and the rest of the PCs to produce the fused image.

GS fusion technique
This technique primarily uses the Gram-Schmidt transformation to reduce the correlation between the MS bands [9]. This technique creates a low spatial resolution PAN image by taking the average of the MS bands. A Gram-Schmidt transformation is performed for the low-resolution PAN image as the first band and the MS bands to produce the GSs components. Then, GS1 is replaced by the histogram-matched PAN image, and the reverse GS transformation is performed for the histogram-matched PAN band and the rest of the GSs to produce the fused image [12].

HCS fusion technique
This technique depends on the transformation between the RGB and hyper-spherical color spaces [13], and its steps are: (1) calculating an intensity component (I) from the MS bands; (2) performing a hyper-spherical color space transformation; (3) matching the PAN image to the I component; (4) using the matched PAN to calculate an adjusted intensity (I adj ); (5) replacing the I component by the I adj ; and (6) performing the reverse transformation from the HCS color space back to the RGB color space to produce the fused image.

Ehlers fusion technique
This technique is based on a combination of color and Fourier transforms [14]. Firstly, an IHS transformation is performed to separate the spectral characteristics of the Hue (H) and Saturation (S) components and the spatial characteristics of the Intensity (I) component. Secondly, by using a fast Fourier transform, the PAN and I images are converted into the frequency domain where the spatial information can be easily enhanced or suppressed. A low-pass filter is used for the I component, whereas a high-pass filter is used for the PAN image. Thirdly, by using the reverse fast Fourier transform, the filtered PAN and I images are converted to the spatial domain and combined to create the fused intensity. Fourthly, the original I component is replaced by the histogram-matched fused intensity and the reverse IHS transformation is performed for the histogram-matched fused intensity and the H and S components to produce the fused image.

Study site and data sets
The entire PAN and MS images, shown in Fig. 1

The proposed fusion technique
The proposed fusion technique, shown in Fig. 5, depends on the accurate calculation of the intensity (I) component from the MS bands. Then, the I component is subtracted from the histogram-matched PAN image to get the spatial details, which will be injected into the original MS bands to generate the fused image. Therefore, the accuracy of calculating the I component controls the quality of the output fused images. The radiometric differences between the PAN and MS images are the main reason for the color distortion in the fused images. The GeoEye-1 spectral response curve, shown in Fig. 6, demonstrates that the MS bands and the PAN band have different ranges of wavelengths. While the blue, green, and red bands have acceptable overlap with the PAN band, the NIR band has poor overlap. Such spectral dissimilarity of the PAN band with the MS bands, and consequently with the I component, will lead to color distortion. Accordingly, the equal representation of the MS bands in the formulation of the I equation, as in Eq. (2), was a mistake. The proposed fusion technique suggested modifying the MS bands before their participation in the I equation to avoid the parts of the MS bands outside the PAN band range, which will reduce the radiance differences between the PAN band and the I image. This technique proposed using the modification coefficient (α i ) for each MS band according to its overlapping area with the PAN band to use only the shared wavelengths between the PAN and MS images in the calculation of the I component. Therefore, the I component of the proposed fusion technique is an average of four modified MS bands, as shown in Eq. (3). The intersection areas of each MS band with the PAN band were divided by the entire area of that MS band to calculate the modification coefficients (α i ). After calculating the full areas of the MS bands and their overlapping areas with the PAN band from the GeoEye-1 spectral response curve, the modification coefficients were as follows: α red 0.9885, α green 0.9470, α blue 0.8480, and α NIR 0.1733.
where α i is the modification coefficient of the MS band i,Ā i is the intersection area of the MS band i with the PAN band, and A i is the full area of the MS band i. Then the I component will be as follows: Because the reflectance of vegetation is high in the NIR band and low in the RGB bands, it was a mistake for the modification coefficient of the NIR band to be constant across the different classes of land cover. It should be increased by increasing the agricultural areas within the image. Therefore, this technique suggested including an extra coefficient (β) for the NIR band in the formulation of the I equation, as shown in Eq. (5), to add the correct impact of vegetation. This coefficient is dependent on the percentage of agricultural areas within the image and is varied for all types of land cover.
where, β is the vegetation coefficient, which varies based on the percentage of the agricultural features within the image.
To determine the appropriate β coefficient for the agricultural dataset, it was fused by the proposed technique under different values of β coefficient. It was found that increasing the β coefficient significantly enhances the spectral and spatial quality, as shown in Fig. 7. After testing numerous agricultural datasets acquired at different times and places, the β coefficient of 7 significantly improved the spectral and spatial resolution of the fused images. For mixed datasets, it was found that increasing the β coefficient enhances the spectral quality for agricultural features while reducing it for urban features. So, the proportion of the agricultural features within the image controls the spectral quality improvement. A lot of mixed datasets with different percentages of agricultural features were used to determine the appropriate β coefficient, which was found as shown in Table 1.
After calculating the β coefficient for the agricultural datasets and the mixed datasets with different ratios of agricultural features, the I equation for each land cover type will be as follows: (1) For agricultural areas: 2) For mixed areas: • The percentage of agricultural areas is (> 80%): • The percentage of agricultural areas is (50-80%): • The percentage of agricultural areas is (20-50%): • The percentage of agricultural areas is (< 20%): The intersection areas of the GeoEye-1 spectral response curve Fig. 7 The impact of β coefficient on the spectral and spatial quality of the agricultural dataset (3) For urban areas:

Experiments and results
After preprocessing steps, the MS image was geometrically aligned with the PAN image, and its pixel size was transformed to equal that of the PAN image. Three different datasets including different classes of land cover (agricultural, urban, and mixed areas) were chosen from the PAN image and its corresponding registered MS image. Then, the fusion methods were utilized to fuse the three datasets. Figures 8, 9, and 10 show the original and fused images of all datasets.
The following statistical parameters were used to evaluate the spectral characteristics of the output fused images: • The correlation coefficient (CC) The CC uses the following equation to compute the correlation between each fused band and the associated MS band: where X i the digital number of pixel i in the MS band, X m the mean of the digital numbers of the MS band, Y i the digital number of pixel i in the fused band, Y m the mean of the digital numbers of the fused band, and n the number of the pixels.
The high values of the CC indicate that the fused image has great spectral quality. The sum of the CC values of all bands is divided by the bands' number to calculate the average CC between the output fused image and the original MS image.
• Relative dimensionless global error in synthesis (ERGAS) The standard deviation of the difference image (SDD) between each fused band and its corresponding MS band is calculated according to the following equation: The absolute value of the mean difference (bias) between each fused band and its corresponding MS band is calculated according to the following equation: The root mean square error (RMSE) between each fused band and its subsequent MS band is calculated according to the following equation: where X i the digital number of pixel i in the MS band, X m the mean of the digital numbers of the MS band, Y i the digital number of pixel i in the fused band, Y m the mean of the digital numbers of the fused band, n the number of the pixels, (h/l) the ratio between the pixel size of the PAN image and that of the MS image, and N the fused bands' number.
ERGAS measures the error in the fused bands because it is based on the RMSE of each band. Therefore, the low values of the ERGAS indicate that the fused image has great spectral quality. Tables 2, 3, and 4 show the values of the CCs and ERGAS for all datasets. To evaluate the spatial characteristics of the output fused images, the PAN and fused images were filtered by a highpass Laplacian filter. Then, the CC values between the filtered PAN band and the filtered fused bands were determined according to Eq. (10). The sum of the HPF CC values of all bands is divided by the bands' number to calculate the average of the HPF CC between the output fused image and the original PAN image. The high values of the HPF CC indicate that the fused image has great spatial quality. Tables 2,  3, and 4 show the values of the HPF CCs for all datasets.

Analysis of results
In terms of spectral quality, the results in Tables 2, 3, and 4 show that: The proposed fusion technique, followed by the Ehlers method, offered the greatest spectral quality for the three data sets. Furthermore, the spectral quality of the Ehlers method is close to that of the proposed fusion technique for data sets 2 and 3 (agricultural and mixed areas). However, for data set 1 (urban area), the spectral quality of the proposed fusion technique is very high and far from that of the Ehlers method.
The reasons for the high spectral characteristics of the proposed fusion technique are: (1) using the modification coefficients for the MS bands in the formulation of the I image makes PAN and I very close spectrally to each other and reduces the gray level differences between them; (2) adding the variable coefficient for the NIR band in the formulation of the I image provides the correct reflectance of the vegetation in the NIR band.
The reason for the high spectral quality of the Ehlers fusion technique is the use of the fast Fourier transform that transforms the PAN and I images into the frequency domain where the spatial information can be easily enhanced or suppressed. Using the high-pass and low-pass filters for the PAN image and I component, consequently, prevents inserting new graylevel values into the MS image during the process of injecting the spatial details.
The other fusion methods are ranked from best to worst: PCA, GS, FIHS, HCS for data set 1 (urban area), HCS, GS, FIHS, PCA for data set 2 (agricultural area), and FIHS, GS, PCA, HCS for data set 3 (mixed area). It should be noted that the type of the scene and how the scene's various features are distributed affect how well a given fusion technique performs.
The FIHS fusion method produces low spectral characteristics because all bands are equally represented in the definition of the I equation, while the spectral association between the PAN image and each of the MS bands is different. According to the GeoEye-1 spectral response curve, while the red band has perfect overlap with the PAN band and the green and blue bands have acceptable overlap, the NIR band has poor overlap. The spectral differences between PAN and I images are the main source of color distortion.
In terms of spatial quality, the results in Tables 2, 3, and 4 show that: All fusion techniques provide spatial improvements in the fused images with different levels of sharpness. The edges of fields in the agricultural areas and the sharper edges and small objects in the urban areas are visible in the output fused images. For all data sets, the GS and PCA fusion techniques provided the greatest spatial quality. Moreover, the PCA fusion method provided a spatial quality that is slightly better than the GS fusion method for data set 1 (urban area). While the GS fusion method provided a spatial quality that is slightly better than the PCA fusion method for data sets 2 and 3 (agricultural and mixed areas).
The proposed fusion technique followed the PCA and GS fusion techniques and produced significant and stable results for all land cover types. The spatial quality obtained due to the Ehlers fusion technique is also accepted. For all data sets, the worst spatial characteristics were shown in the fused images of the HCS fusion method.
In terms of visual inspection, Figs. 8, 9, and 10 show that: Parallel to the statistical assessment, a visual inspection reveals that the proposed fusion technique produced undistorted fused images for the three data sets. The Ehlers fusion technique provided acceptable spectral quality with low color distortion. On the other hand, the FIHS, PCA, and GS fusion methods provided some color distortion in the green areas, and the HCS fusion method provided some color distortion in the dark areas.
The PCA and GS fusion techniques produced high spatial quality in the three data sets, followed by the proposed fusion technique. The HCS fused images are the smoothest among the resultant fused images for the three data sets.
Overall evaluation The analysis of the results in Tables 2,  3    spectral quality for the urban and mixed areas, while its performance was acceptable for the agricultural areas. On the other hand, it provided high spatial quality for all datasets. The PCA and GS fusion techniques produced the highest spatial quality for all land cover classes. This was at the expense of spectral quality, as they produced poor spectral quality for urban and mixed datasets and acceptable quality for the agricultural dataset. While the spectral and spatial performance of the HCS fusion technique was acceptable for the agricultural areas, it was very poor for the urban and mixed areas. The Ehlers fusion technique provided significant spatial quality for all land cover types. Its spectral performance was significant in agricultural and mixed datasets but poor in urban datasets. As the proposed fusion technique uses modification coefficients for the MS bands in the formulation of the I component to use only the overlapped wavelengths with the PAN band, it reduces color distortion and accomplishes spatiotemporal transferability. Additionally, this technique uses an additional coefficient for the NIR band based on the ratio of the agricultural features within the image to indicate the correct impact of vegetation, which provides performance stability across the different land cover classes. Therefore, the proposed fusion technique could overcome the limitations of the existing fusion techniques and produce significant spectral and spatial quality for all land cover types.

Validation of the proposed fusion technique
When a novel method is proposed, it should be thoroughly assessed from the perspective of spatiotemporal transferability to guarantee its effectiveness across various locations and times. In addition to the first three datasets from Tanta [15]. There were two PAN and MS image pairs of the GeoEye-1 satellite in this dataset. Therefore, they were used in the validation process under the names of dataset 4 and 5, which cover urban areas in Trenton, United States of America, and London, United Kingdom, respectively. Furthermore, other four datasets, under the names of dataset 6 through 9, covering agricultural and mixed areas in different places and acquired at different times, were also used in the validation process; their information is shown in Table 5.
The proposed fusion technique was validated against the original FIHS fusion technique for the six datasets to make sure it maintained its superiority. The original and fused images of the six datasets are shown in Figs. 11, 12, 13, 14, 15, and 16. The spectral quality of the fused images was assessed by the correlation coefficients between the fused bands and the original MS bands. Also, the spatial quality of the fused images was assessed by the correlation coefficients between the filtered fused bands and the filtered PAN band. For the six datasets, the spectral and spatial quality of the fused images is shown in Tables 6, 7, 8, 9, 10, and 11.
-The results in Tables 6, 7, 8, 9, 10, and 11 show that the proposed fusion technique had the highest spectral quality for all datasets. It provided a spectral quality that was roughly (1.5-5.5%) better than the FIHS fusion technique. This is because (1) using only the parts of the MS bands inside the area of the PAN band in the definition of the I equation reduces the difference in radiometry between the PAN band and the MS bands and decreases the resultant color distortion, and (2) using the additional coefficient β for the NIR band in the definition of the I equation, which is variable for all types of land cover based on the percentage of the agricultural areas within the image, adds the correct effect of vegetation and produces significant and stable performance for all types of land cover. Both fusion techniques produced close spatial quality for the urban datasets, but for the agricultural and mixed datasets, the proposed fusion technique outperformed the original FIHS fusion technique by about (1 to 5%). The visual inspection of the Figs. 11,12,13,14,15, and 16 demonstrated that the proposed fusion technique significantly enhanced the spectral quality, reduced color distortion, and produced sharp fused images for all land cover types.

Conclusion
This study demonstrated that the proposed fusion technique can significantly solve the problem of color distortion and produce fused images of superior spectral and spatial resolution for all land cover classes because of: (1) using modification coefficients for the MS bands, in the formulation of the I equation, which are calculated based on their overlapping areas with the PAN band; and (2) adding a variable coefficient for the NIR band, which is dependent on the proportion of the agricultural regions within the image. In addition to reducing the radiometric differences between the PAN image and the I component, the modification coefficients provide spatiotemporal transferability for the proposed fusion technique. While the vegetation coefficient indicates the correct impact of vegetation in the NIR band, it also provides stability for the proposed fusion technique across all land cover classes. In future work, this technique will be developed to be used for satellites with different spectral response curves.
Funding The authors have not disclosed any funding.

Conflict of interest The authors declare no competing interests.
Ethical approval Not applicable.

Consent to publish Not applicable.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/.