High-resolution images based on directional fusion of gradient

This paper proposes a novel method for image magnification by exploiting the property that the intensity of an image varies along the direction of the gradient very quickly. It aims to maintain sharp edges and clear details. The proposed method first calculates the gradient of the low-resolution image by fitting a surface with quadratic polynomial precision. Then, bicubic interpolation is used to obtain initial gradients of the high-resolution (HR) image. The initial gradients are readjusted to find the constrained gradients of the HR image, according to spatial correlations between gradients within a local window. To generate an HR image with high precision, a linear surface weighted by the projection length in the gradient direction is constructed. Each pixel in the HR image is determined by the linear surface. Experimental results demonstrate that our method visually improves the quality of the magnified image. It particularly avoids making jagged edges and bluring during magnification.


Introduction
The aim of image magnification is to estimate the unknown pixel values of a high-resolution (HR) version of an image from groups of pixels in a corresponding low-resolution (LR) image [1]. As a basic operation in image processing, image magnification has great significance for applications in many fields, such as computer vision, computer animation, and medical imaging [2]. With the rapid development of visualization and virtual reality, image magnification has been widely applied to diverse applications, such as high-definition television, digital media technology, and image processing software. However, image magnification methods face great challenges because of the increased demand for robust technology and application challenges. In recent years, although many researchers have proposed a variety of methods for image magnification, there is not yet a unified method suitable for all image types. Considering the characteristics of different types of images, it is still hard to achieve low computational time while maintaining edges and detailed texture during the process of magnification. Based on the analysis above, this paper focuses on generating an HR image maintaining the edge sharpness and structural details of a single LR image by means of the directional fusion of image gradients.

Traditional methods
Traditional methods, including nearest neighbor, bilinear [3], bicubic [4,5], and Lanczos resampling [6], are widely applied in a variety of commercial software and business applications for image processing. The main advantages of such conventional methods are that they are easy to understand, simple to implement, and fast to calculate. However, there are limitations for these methods. Using a unified mathematical model causes loss of high frequency information at edges. Thus, conventional methods are likely to introduce jagged edges and blur details at significant transitions in an image, such as edges and texture details.

Advanced methods
Studies have shown that human eyes are more sensitive to the edges of an image that transmit most of the information of the image, so images with good quality edges can help to clearly describe boundaries and the outlines of objects. Edges that contain important information are of great significance in image magnification. Various edge-directed methods have been proposed in recent years, most of which take advantage of edge information to overcome the shortcomings of conventional methods, e.g., Refs. [7][8][9][10][11][12][13].
The edge-guided interpolation method put forward by Li and Orchard [10] is based on image covariance, and exploits local covariance coefficients estimated from the pixel values of the LR image to calculate the covariance coefficients of the HR image, utilizing the geometric duality between LR and HR images. These covariance coefficients are used to perform interpolation. Zhang and Wu [12] present a nonlinear interpolation method, based on inserting a missing pixel in two mutually orthogonal directions, and use a minimum mean square error estimation technique to fuse them for realizing interpolation.
Zhang et al. [8] propose a method based on a combination of quadratic polynomials to construct a reverse fitting surface for a given image in which the edges of the image act as a constraint, which ensures the fitted surface has a better approximation accuracy. Fan et al.
[14] present a robust and efficient high-resolution detail-preserving algorithm based on a least-squares formulation. A gradientguided image interpolation method is presented in Ref. [9], assuming that the variation in pixel values is constant along the edge. The method can be implemented simply and has good edge retention, but it leads to a wide edge transition zone because of the diffusion of the HR image gradients, and so it is not suitable for magnification of images with complicated textures and detail.
Traditional methods often introduce artifacts such as jagged edges and blurred details during magnification. Often, edge-based methods tend to generate artifacts in small scale edge structures and complicated texture details. Learning-based techniques are complex and time-consuming, with the outcome influenced by the training data. Because of these issues, this paper proposes a novel method to produce an HR image based on the directional fusion of gradients.

Related work
In this study, we use a degradation model that assumes the LR image can be directly down-sampled from the HR image, rather than by using Gaussion smoothing. Since the proposed method is partly based on CSF [8] and GGI [9], this section will briefly introduce both methods.

Quadratic surface fitting constrained (CSF) by edges
In CSF, image data is supposed to be sampled from an original scene that can be approximated by piecewise polynomials [8]. The fitted surface is constructed by a reversal process of image sampling using the edge information as constraints. That makes the surface a good approximation to the original scene, with quadratic polynomial precision. Assuming that P i,j is an image of size N × N generally sampled from the original scene F (x, y) on a unit square, so where w(x, y) is a weight function set to be 1.
(2) where a 1 , a 2 , a 3 , a 4 , a 5 , and a 6 are to be determined. Determination of the unknown coefficients is performed by a least-squares method constrained by edge information [8]. Since a good quality surface can help to produce high precision interpolation, we will later make use of the constructed surface to interpolate gradients.

Gradient-guided interpolation (GGI)
In order to eliminate jagged edges, a gradientguided interpolation method is proposed in Ref. [9], based on the idea that the variation in pixel values is constant along the edge direction. GGI uses a Sobel kernel to calculate gradients of the LR image, and adopts bicubic interpolation to determine the gradients of the HR image, then uses gradient diffusion. Finally, the unknown HR pixels P i,j to be interpolated are divided into three categories with different LR pixels P x,y in the neighborhood N ij .
P i,j is estimated by summing the neighborhood pixels N ij weighted by w xy , where a shorter distance carries greater weight. Let d xy denote the distance between P x,y and P i,j projected along the gradient direction of P i,j . Then where a = 0.2 controls decrease of the exponential, and S is defined as Although the method of Ref. [9] provides good quality interpolation at edges by significantly decreasing jagged edges, it can cause loss of detail in non-edge regions in some cases. In particular, it is unsuitable for image areas containing complex details and abundant texture.

High-resolution image based on directional fusion of gradient
In this section, a new magnification method is put forward based on fusion of gradient direction, which exploits the property that the pixel values change very quickly in the gradient direction. From the analysis above, maintaining is sharpness of edges and the clarity of detailed textures becomes the key mission in image magnification, since most information in the image is transmitted by edges and detail textures. Our method first finds approximate gradients of the LR image, then calculates those of the HR image. We estimate the gray values of the unknown pixels in the HR image, using a linear approximation of the neighboring pixels. For simplicity of discussion, we mainly focus on enlargement by a factor of 2, to produce an HR image of size 2m × 2n from an LR image of size m×n. The general information flows in our proposed method are shown in Fig. 2.

Calculating the gradients of the HR image
In order to compute the LR gradients with high accuracy, our method adopts Eq. (2) to compute the LR gradient for each P i,j . The gradient vector of the LR image is defined as − → g = (g x , g y ), where g x and g y are defined as Thus, for each P i,j we can get the LR gradients as g x = a 4 , g y = a 5 . The LR gradients are used to calculate HR gradients, denoted by − → IG = (G X , G Y ), by bicubically interpolating the LR gradients.

Diffusing the gradients of the HR image
The GGI method [9] utilizes the gradient information in order to maintain the sharpness of edges. However, the spatial distribution of gradients is not considered effectively during diffusion: the norm of the gradient takes a local maximum in the gradient direction [21]. It may cause the gradient direction to change in an inapprorpiate way in detail-rich portions by directly replacing the gradient at a central pixel by the mean of some region, which may Therefore, we take account of the spatial correlation between the gradient directions to improve the diffusion of gradients − → IG. Diffusion deals with gradient values in the vertical G X and horizontal G Y directions. A local window of size 5 × 5, with P i,j as the central pixel, see Fig. 3, is used to adjust the gradient direction. Our method adjusts the gradient vector of the center pixel using the average value of gradients whose direction falls within a certain rage relative to that of the central pixel.
By considering the spatial correlations between gradient directions, our method can approximate HR gradients that not only maintain the sharpness of edges, but also better retain the structure of textures and details. Let k denote the number of pixels satisfying the condition β xy < α, and α = 45 • .
After conducting the diffusion of − → IG = (G X , G Y ), we obtain the adjusted HR gradients − − → CG = (G X , G Y ), which are used to calculate the gray values of HR pixels.

Estimation of HR image
In this section, we give the strategy for calculating the unknown pixels of the HR image. In Section 2.2 we noted that the GGI method [9] yields a precise constant. In comparison with GGI, our method Diffusion of gradient. The blue dots Px,y stand for neighboring HR pixels of Pi,j , and the blue arrow represents the gradient direction at Px,y, while the red arrow indicates the gradient direction at Pi,j . βxy is the angle between the gradient directions Pi,j and Px,y. The dashed area defines the range of angles for which the gradient direction of Px,y is positively correlated with that of Pi,j .
provides higher precision of polynomial interpolation by constructing a linear surface to approximate the intensity of the HR image. It performs well in maintaining the details of the image. Depending on the known pixels in the neighborhood window with the unknown pixel as the center (see Fig. 4(b)), the unknown pixels of the HR image may be divided into three categories: (1) Black I(2n − 1, 2m − 1) H ; (2) Blue I(2n, 2m) H ; (3) Pink I(2n−1, 2m) H and I(2n, 2m−1) H , where n = 1, · · · , N, m = 1, · · · , M . Therefore, the estimation of the unknown pixels in the HR image is achieved in three steps.
Step 1: In this step, we assign the values of LR pixels to the corresponding HR pixels. For an LR image I L of size n × m enlarged to give an HR image of size 2n × 2m, we have I(2n − 1, 2m − 1) H = I(n, m) L , where n = 1, · · · , N and m = 1, · · · , M . I(n, m) L and I(2n−1, 2m−1) H are the solid black dots shown in Fig. 4(a) and Fig. 4(b), respectively.
Step 2: In this step, we use four neighboring black pixels to calculate the central pixels P i,j (the blue dots in Fig. 5(a)) satisfying P i,j ∈ I(2n, 2m) H . In order to precisely obtain P i,j , we construct a linear surface to approximate the image data via directional fusion of gradients. Within the neighborhood window N ij centered on P i,j , our method constructs a linear surface f H i,j using a linear polynomial as follows:  where a, b, and c are unknown coefficients to be found.
We determine the unknown coefficients (i.e., a, b, c) in Eq. (8) by a least-squares method, weighted by the gradients and the values of pixels in the neighborhood window.
Finally, the pixels located on the image boundary are calculated by averaging the existing neighboring pixels, instead of by constructing a surface.

Results and discussion
In order to verify the effectiveness of the proposed method, we have carried out many experiments with different kinds of images, including natural images, medical images, and synthetic images. The results of our experiments demonstrate that the proposed method can obtain better quality image magnification, especially at edges and in detailrich areas.
To demonstrate the advantages of our proposed method, we compare magnification results with several methods, including bicubic interpolation (Bicubic) [4], cubic surface fitting with edges as constraints (CSF ) [8], the new edgedirected interpolation method (N EDI) [10], and gradient-guided interpolation (GGI) [9]. We now analyze the experimental results in detail.
In the experiment, we carried out tests with different types of images by magnifying LR images of size 256 × 256 to get HR images of size 512 × 512. Figures 7 and 8 show the magnified images with labeling of local windows containing edges and details extracted from the HR image. Comparing the corresponding regions of the boat image in Fig. 7, we can see that our method is more capable of dealing with edge portions of an image, while other methods introduce jagged edges or blurring artifacts near edges. It is also clear from Fig. 8 that Bicubic [4] and CSF [8] methods tend to introduce bluring artifacts: see the moustache of the baboon. NEDI [10] produces zigzags that are particularly  evident, while GGI [9] causes loss of detail in the area of the moustache. Our method leads to better visual quality than other methods.
We also conducted experiments with MRI images of a brain which were segmented into four classes by the MICO (multiplicative intrinsic component optimization) segmentation algorithm [22]. Although the results of MICO algorithm provide high accuracy segmentation, there are still rough edges due to limitations of the segmentation method. Figures 9(a)-9(f) show Bicubic, CSF, NEDI, GGI, and our results from top to bottom. The results of magnification shown in Fig. 9 illustrate that our method can deal well with a segmented image with severe zigzags, effectively retaining sharp edges while avoiding jagged artifacts during magnification.
For synthetic images, Fig. 10, Fig. 11, and Fig. 12 show the map of gray values at edge portions after applying several methods mentioned above. It is clear that our method is able to maintain the sharpest edges with less blur: other methods produce fuzzy data around the edges which results in blurring artifacts.
In order to evaluate the quality of the magnification results, we use the three objective methods based on comparisons with explicit numerical criteria [23] , including peak signal to noise ratio (PSNR), structural similarity (SSIM), and percentage edge error (PEE). PSNR measures the disparity between the magnified image and the ground truth image, and is defined as where the mean square error (MSE) between two images is SSIM measures the similarity of the structural information between the magnified image and the ground truth image [24]. It is related to quality perceived by the human visual system (HVS), and is given by where µ S and µ I denote the mean value of the ground truth image and the magnified image respectively, σ S and σ I represent variances of the corresponding images, and σ SI denotes the covariance of the two images.
For the images shown in Fig. 13, values of PSNR and SSIM are listed in Table 1 and Table 2, respectively. It is clear that our proposed method performs well in most cases, giving the highest values for PSNR and SSIM.
In addition, the percentage edge error (PEE) [25] was also used to measure perceptual errors. PEE is very suitable for measuring dissatisfaction of image magnification, where the major artifact is blurring. PEE measures the closeness of details in the interpolated image to the ground truth image. Generally in image interpolation, a positive value of PEE means that the magnified image is over smoothed, with likely loss of details. Thus, a method with smaller PEE performs better at avoiding blurring artifacts. PEE is defined by where ES S denotes the edge strength of the ground truth image and ES I is that of the magnified image. ES is defined as where EI (i,j) denotes the edge intensity value of the image.
The PEE values for each interpolation method are shown in Table 3. It is clear that the PEE value for the proposed method is very low compared with the values for other techniques, so structural edges are better preserved and less blurring is produced in our method.
The analysis of the experimental results above shows that the proposed method achieves a good balance between edge-preservation and blurring, performing especially well on synthetic images and segmented medical images. The major drawback of this method lies in the limitation of using the gradients only in horizontal and vertical directions, making it hard to get accurate gradient values for images with very low contrast. Our future work will consider how to calculate gradients in more directions, and use a surface of high accuracy to approximate the image data. We hope to develop a method for magnification that can maintain edges and detailed texture perfectly with low computational time.

Conclusions
This paper presents a novel method of producing an  to constrain the gradients of the HR image. Finally it estimates the missing pixels using a linear surface weighted by neighboring LR pixels. Experimental results demonstrate that our proposed method can achieve good quality image enlargement, avoiding jagged artifacts that arise by direct interpolation; it preserves sharp edges by gradient fusion.      Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http:// creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.