Multi-focus Image Fusion Based on Random Walk

An effective multi-focus image fusion algorithm based on random walk is proposed in this paper. Random walk and guided filter have attracted extensive attention in image fusion. Random walk is usually used to solve probability problems and it has a good smoothing effect, and guided filter can preserve the gradient information of the image well. The combination of two algorithms can better retain the edge information of the input image. Six sets of source images and five existing methods are used in the experiment and the experimental results show that the proposed algorithm outperforms the existing methods in both subjective and objective evaluation.


Introduction
Multi-focus images are a set of images obtained by the imaging equipment with different focal distances of the same scene. Because the depth of field of image capture device is limited, it is difficult to directly obtain an image with all the objects focused. However, an image with all the objects focused is necessary for image process, such as image segmentation and recognition. Therefore, multi-focus image fusion algorithm, a method used to integrate multiple images captured with different focal distances into a single image, is presented to solve the above problems.
In recent years, multi-focus image fusion technology has advanced rapidly. Many new fusion algorithms [1][2][3][4][5][6] are introduced, which make the fusion results better and the fusion accuracy higher. Generally, the fusion algorithms are able to be divided into two classes, spatial domain algorithms [7][8][9][10][11] and transform domain algorithms [2,[12][13][14][15][16]. To obtain the fused image, the spatial domain algorithms directly deal with the intensity values of pixels. The transform domain algorithms usually transform the source images into the frequency domain, and use the frequency coefficients to obtain the fused images. However, both spatial domain algorithms and transform domain algorithms have advantages and disadvantages.
For spatial domain algorithms, the average algorithm is simple but reduces the sharpness and contrast. The fusion methods through selecting the sharper pixel can solve the above problem, but it requires a series of evaluation metrics to calculate the focus degree of the fused image. All the above methods are affected by noise. The algorithms based on region [17,18] are proposed to overcome the noise. The region-based methods could be affected by the artifacts at the boundary of region, which will reduce the quality of the results. Transform domain algorithms mainly fuses multiresolution images. Although this method reduce the boundary artifacts, it is sensitive and shift-variant. In order to solve the above problem, a multi-focus image fusion algorithm based on random walk and guided filter in spatial domain is proposed.
In 1905, the random walk was proposed in the study of [19]. And authors in [20] used in computer vision in 1979. After that, it becomes a hot topic in computer vision and it is widely used in the field of image processing, such as image enhancement [21], image smoothing [22], image fusion [23] and so on. Based on random walk, the fuse weighting factor of the input image is calculated by random walk [24]. This method is superior to many new techniques in both subjective and objective image quality measurement. A robust multi-focus image fusion method based on lazy random walks is proposed, which can obtain the fused images with few artifacts [25]. Generally, the method of random walk is described by a connecting graph and the vertices stand for the image pixels. In the proposed method, the random walk is regarded as a smoothing filter, because it can preserve important features and edges as much as possible and remove tiny noise.
Guided filter, a novel edge-preserving filter, was proposed in 2010 [26], and it was used in image fusion in 2013 [27]. In recent years, the guided filter is widely utilized in image processing. The region-based image fusion method has many advantages, so a multi-focus image fusion method based on guided filtering and region analysis is proposed [28]. A multi-focus image fusion algorithm based on guided filter and improved PCNN is proposed [29]. The guided filter cannot retain all the important features of the source image in the process of image fusion. To solve this problem, a method combining static and dynamic filters [30] is proposed. Guided filter is good at preserving important edge information, which can avoid ringing artifacts.
Based on the above advantages, we combine random walk and guided filter to propose a multi-focus image fusion algorithm. The novelties of the algorithm mainly conclude the following three points: (1) In order to construct an effective weight map, preprocessing method based on guided filter is adopted and random walk is used to optimize the weight map. (2) Random walk is introduced in multi-focus image decomposition to extract the focused region as much as possible. (3) The difference images are used to compare the fusion results, which makes the advantages and disadvantages of the experimental results more obvious. This improved combination fusion rule improves the quality of the fused image and makes the fused image more suitable for human visual system. The rest of our paper is organized as follows. The related work is described in Sect. 2 and the proposed algorithm is illustrated in Sect. 3. In Sect. 4, the simulation experiment and result are presented and discussed. Section 6 is the conclusion of this algorithm.

Guided Filter
The filtering output Z is a linear transform of the guidance image I by the guided filter. The filter uses a local square window mn of size (2r + 1) × (2r + 1) . The center pixel of mn is (m, n) . And the relationship is described by Eq. (1).
where I st is the pixel value at (s, t) of I. Z st presents the output of I st and (s, t) ∈ mn denotes that the location (s, t) belongs to mn . X mn and Y mn are constant values in mn and they can be computed through minimizing E X mn , Y mn in Eq. (2).
where is a regularization parameter and P stands for the input image. The linear regression [31] can help to compute X mn and Y mn .
where | | is the total number of pixels in mn , mn is the mean value of mn and mn is the variance of mn . P st and P mn stand for the pixel value at (s, t) of P and the mean value of P in mn . It is necessary to calculate the average values of all the windows. Then the output Z st is obtained by Eq. (7).

Random Walk for Image Smoothing
The random walk [32] is able to be expressed by a connected graph Gm(V, Q, W) , including vertices V v 1 , v 2 , … , v and edges Q q 1 , q 2 , … , q l . e ij is defined an edge linking two neighboring vertices v i with v j and w ij is the weight of e ij .
where I i and I j are pixel intensities at the point i and j.
A symmetric positive semi-definite sparse matrix is defined as L, which is expressed by Eq. (9). d i is the weights of the edges connecting v i with its neighboring vertexes. (1) The combinatorial formulation of the Dirichlet integral, denoted as E, is the weights on the boundary of the segmentation.
To receive the minimum of E, supposing the data points contain pre-labeled points N of K classes, unlabeled points M remain to be labeled and E has another expression, which is defined by Eq. (11).
where L U , O T and x L are matrices and the size of them are M × M , M × N and N × K , respectively. Accordingly, L U is considered as closeness between the unlabeled points, O T is regarded as closeness between the labeled and the unlabeled points, and x L is considered as relationship between the labeled and the labels, which represents the pre-labeled information.
Minimizing above integral with respect to f U , and we will get it from Eq. (12): where f U stands for the closeness between the unlabeled points and the labels. So each of the unlabeled has a K-tuple of closeness number indicating their closeness with K labels.
The solution of random walker probabilities and the combinatorial Dirichlet problem is same in fact. Wang et al. proposed a random walk based smoothing filter [22]. And it is described as follows.
In order to adapt random walk algorithm to smooth input images, Eq. (13) is minimized.
where a constant number const is independent of f. The smoothed pixel intensity f is the only critical point to minimize E RW , and it can be obtained by differentiating E RW to f.
is the trade-off value.
The Eq. (14) is capable of solving by inverse matrix.
Consequently, an edge-preserving filteris Rr is obtained.

The Proposed Fusion Method
This section introduces the proposed algorithm in detail and the main process is shown in Fig. 1. The fusion algorithm consists of four steps. Firstly, the source image is decomposed into the base layer and the detail layer by random walk. Secondly, the final weight maps of the detail layer and the base layer are gained by optimizing the initial weight maps, which are obtained by guided filter and random walk. Next, the fused detail layer and base layer are obtained by weighted average rule. Finally, the fused image is acquired by summing up the fused detail layer and base layer.

Decomposing Source Images
The source images A and B are decomposed into base layers Bi c and detail layers Di c by random walking. Assuming that The base layer is denoted by Eq. (17). Bi c is the base layer of the source image.
As for the detail layer, it can be gained by Eq. (18). Di c is the detail layer of the source image.

Obtaining the Weights
The process of obtaining the weight map is as follows. Firstly, the high-pass image Hm is gained by Laplacian filter to the source image, and it is defined by Eq. (19). Lf is a Laplacian filtering of size 3 × 3. Fig. 1 The multi-focus image fusion process After that, the saliency map Sm c is constructed by the local average of the absolute value of H c , and it is denoted by Eq. (20). G is a Gaussian low-pass filter with the size of 2r g + 1 × 2r g + 1 , and both r g and g are set to 5.
Next, the measured saliency map is defined by Eq. (21), which is compared to determine the weight map. And it has the advantage of providing a good characterization of the saliency level of detailed information.
where C is the number of source images and Sm k c stands for the saliency value of the pixel k in the c th image.
However, the weight maps obtained by the above operation are not satisfactory, and they are always noisy and not aligned with object boundaries. Therefore, in order to avoid introducing artifacts, it is necessary to optimize the fused image.
where r 1 , 1 , r 2 and 2 are the parameters of the guided filter. W Bi 1c and W Di 1c are the weight maps of the based layer and the detail layer of the c th image after the first optimization. At last, normalizing the values of the C weight maps to make the sum of the weight of each pixel k to be 1.
The second optimization of weight map is obtained by random walk filter, and it is defined by Eqs. (24)- (25).
where 1 , 1 , 2 and 2 are the parameters of the random walk filter. W Bi 2c and W Di 2c are the weight maps of the base layer and the detail layer of the c th image after the second optimization.

Weight Averaging
The fused detail layer D F and base layer B F are obtained by weighted average of detail layer and base layer, which are denoted by Eqs. (26)- (27).
where C is the number of the source images.

Fused Image
Finally, the final fused image F is obtained as follows.

Simulation Experiment and Result
There are six sets of source images in the experiment. The source images are shown in Fig. 2. Five algorithms are compared with the proposed method in this paper. At frirst the parameters are given, because they will influence the fusion results. The difference method is used to decompose the images as shown in Figs

Parameter Setting
The parameters of the proposed algorithm are set as follows.
In the experiment, the proposed algorithm is compared with the following five algorithms and the parameters for the comparison methods are given as follows. The parameters of wang's method and GFF are set as provided in paper [3] and [27], respectively. For random walk with FSWM (RW-FSWM), the parameters are set as provided in paper [24]:

Subjective Analysis
This section shows the fusion results of multi-focus images, and makes a series of observations and analysis of these images. In Figs. 3, 4, 5, 6, 7 and 8, the middle difference images of each algorithm are the differences of the fused images and the corresponding right focused images, and the left difference images are the differences of the fused images and the corresponding left focused images. As for the difference images, if the corresponding fused image is better, the focused area is smoother and the unfocused area is clearer.  As shown in Fig. 3, the right half of the difference image in the middle of the proposed method is smoother than other results, indicating that the fusion image obtained by the proposed method can completely extract the information of the focusing region. It can be found that the middle difference images in (b), (c), (d), (e) and (f) is relatively rough, which fully shows that the proposed method has a good effect. As it can be seen from Fig. 4, the edge and details of the difference image on the right obtained by our method are clearer, while some details are lost in the results of other algorithms.
In Fig. 5, the middle difference image obtained by the proposed method and RW-FEWM is smoother than other images. In (c), (e) and (f), the lower part of the difference image is rough. Compared with other algorithms, the nonfocusing region of the difference image obtained by the proposed method contains more information and the edge is clearer. In other words, the fusion effect of our algorithm is better than other algorithms. It can be seen from Fig. 6 the difference between the middle image of (a) and (d) is smoother in the right part, while (e) and (f) are not only rough but also contain a lot of invalid information.   Fig. 7, the right part of the middle image of (a), (b), (c) and (d) is smooth, while the left part of (a) is the one with the richest information and the clearest edge details. The result of the proposed method has the best performance. In Fig. 8, the left part of the difference image on the right side of (a), (b), (d) and (e) is smooth, indicating that the four images extract more information in the focus area. In (c), a small part of the region fails to extract effective information. And the method of (f) not only fails to extract enough effective information, but also has serious blurring phenomenon.

Objective Metrics
In this subsection, six objective metrics are described. Normalized mutual information ( Q MI ) [34,35] is used to indicate the association between source images and the corresponding fused image. Correlation coefficient  The schematic diagram of Clock (Fig. 3 (CORR) [35] reflects the degree of linear correlation between source images and fused image. Nonlinear correlation information entropy ( Q NCIE ) [36] measures the degree of association between source images and the corresponding fused image. And gradient-based fusion performance ( Q G ) [36] is applied to measure the edge information of the fused image. Q AB∕F [37] uses local information to estimate the degree of significant information in the fusion image. Total fusion artifacts for the fusion process ( N AB∕F ) [37] is measured by the weighted integration of the noise estimates over the entire fused image. A fused image with good quality has a higher Q MI . And Q MI is denoted as follows.
where A and B are the two source images, and F stands for the fused image. H(F) , H(A) and H(B) are the information entropy of F, A and B. And the mutual information between the fused image and the source images is expressed by MI(A, F) and MI(B, F).
where p(u, v) is the joint probability density function of U and V. p(u) and p(v) stand for the probability density functions of U and V, respectively. As defined in Eqs. (32)- (36), CORR reflects the degree of linear correlation between A, B and F. The value of CORR is close to 1, which indicates that the fused image is similar to the source images. Fig. 10 The schematic diagram of Disk (Fig. 4 where Q AF (i, j) and Q BF (i, j) are gained by Eq. (46). And A (i, j) and B (i, j) are weights for Q AF (i, j) and Q BF (i, j) , which can be obtained by Eqs. (32,45). Besides, g A (i, j) and g B (i, j) can be computed by Eq. (46).
where Q AF g (i, j) and Q BF (i, j) stand for the corresponding Sobel edge strength and orientation preservation value, respectively.
where s x A (i, j) and s y A (i, j) refer to the horizontal Sobel operator and the vertical Sobel operator at (i, j) of image A. g A (i, j) and g F (i, j) can be computed by Eq. (49). The parameter settings are shown below: g = 1 , K g = −10 , g = 0.5 , = 1 , K = −20 and = 0.75.
Total fusion performance Q AB∕F evaluates the performance of the fused image by calculating the weight of local quality index Q A ( ) and Q B ( ) in Eq. (54). By the sliding window with size of W, the local quality indax Q A ( ) and Q B ( ) are obtained, where the s A ( ) and s B ( ) is the local saliences of the input images A and B.
N AB∕F represents the noise generated during the fusion process as follow. For noiseless fused images, N AB∕F should be close to 0.

Objective Analysis
The objective analysis is shown in this subsection. The fusion results with some algorithms are different but similar in visual effect, so it is very important to apply objective Fig. 11 The schematic diagram of Tools (Fig. 5 From Figs. 9, 10, 11, 12, 13 and 14, it is clear that our method always has the best performance in Q MI , Q NCIE , Q G and N AB∕F meaning that our method has a good ability to retain the detailed information and the artifact noise of our's is the least. From Figs. 9, 10, 11 and 14, it can be seen that Fig. 13 The schematic diagram of Magazine (Fig. 7 10,11,12 show that the images obtained by the RW-FSWM, GFF and m-PCNN are higher than that of our's, indicating that the proposed method does not perform well in the total fusion performance and does not fully fuse the effective information of the source image. Therefore it indicates that the proposed method needs further improvement. It is obvious in Fig. 11, the Q G , Q MI and Q NCIE of GFF are higher than that of m-PCNN.
The experimental results show that the proposed method performs best in four of six objective indicators, which indicates that the proposed method has certain validity and practicability, and is superior to the other five algorithms.

Discussion
In Fig. 3, it is obvious that the proposed algorithm is smoother than other algorithms and the head of the person in (b), (c), (d), especially (e) and (f) are rough. In Fig. 4, both edge and numbers of the clock of the right difference image are clearer. It can be seen that the fused image with the proposed algorithm is the best one. From Fig. 5, it is clear that the proposed algorithm and wang's method are smoother than others, especially (c), (e), (f) are rougher. Keeping under observation, it can be found that the fused image of the proposed algorithm is better than wang's method. In Fig. 6, the right parts of (a) and (d) are smoother and they are a little rough in (b) and (c). It is easy to find that the qualities of the fused image in (e) and (f) are poor. From Fig. 7, the right parts of the middle images in (a), (b), (c) and (d) are smoother. In Fig. 8, the left parts of the right images in (a), (b), (d) and (e) are smoother. In summary, it can be clearly seen that the algorithm proposed in this paper is more consistent with the human visual system and more suitable for observing.
Experimental results show that only the proposed method has no distortion, with details and structure perfectly preserved, and the proposed method can obtain the fused image with the highest quality. Based on the above analysis, we can draw a conclusion that the proposed algorithm is superior to other methods in both visual observation and objective indicators, and has practical value and effectiveness.

Conclusion
In this paper, the optimized ideas of guided filter and random walk are adopted to preserve the edge information and image structure of the fused image. Firstly, the proposed method decomposes the source image by guided filtering and random walk to obtain the detail layer and the base layer. Then the weight map is obtained according to the significance, and the weight graph is optimized by the random walk model. Finally, the fusion result is obtained according to the final weight map.
In terms of subjective evaluation, this method can effectively extract the focused region and it is suitable for human visual system. Six evaluation indexes were used to evaluate the quality of fused images. The performance of this method is better than other methods through comparative experiments. Experimental results show that this method can effectively retain the edge informatio, display the details of the source image more clearly, and ensure that no additional artifacts and information are added in the fusion process. This method not only preserves the detailed information of the focused region, but also ignores the interference information of the fuzzy region. In addition, the proposed method still has some defects, for example, the results of this method are not optimal in CORR and Q AB∕F indicators. During visual observation, the fusion image of this algorithm is not very clear. Therefore, in the future, the algorithm needs to be improved in the direction of effectively decomposing images and constructing optimal weight maps.

Conflict of interest
The authors declare that they have no conflict of interest.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.