Non-uniform Noise Image Denoising Based on Non-local Means

Abstract

Restricted by camera hardware, digital images captured by digital cameras are noisy, and the noise content of each color channel of the digital image is not balanced. However, most of the existing denoising algorithms assume that the entire image noise is constant, causing errors in the denoising of color images (non-uniform noise images), affecting image noise removal and texture detail protection. To solve this problem, we propose an evaluation operator that can describe the noise content and texture content in the local area of the image. According to the description value, the image pixels are classified, and heuristic denoising parameters are selected for each class to achieve a balance between noise removal effect and texture retention effect. Experimental results of multiple denoising methods show that the proposed algorithm has better denoising effect on color images.

1 Introduction

Digital images help us learn. It can transmit and store information. Digital image is often polluted during the process of acquisition and transmission that the image quality is reduced. Therefore, image noise removal is an important research direction of image processing that has been extensively studied in the past several decades [1,2,3,4,5,6,7,8]. Most existing denoising methods are concentrated in additive white Gaussian noise (AWGN) [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16], in which the observed noisy image is modeled as a composition of clean image and AWGN noise: $z(i) = x(i) + n(i)$. It is important to note that most of these methods assume that the noise variance of the entire image is fixed so that will inevitably bias the denoising result in the subsequent experiments, which will also have a certain impact on the subsequent application.

As a matter of fact, the noise level in the real noisy image is often non-uniform. In other words, the noise variance is not fixed and is randomly distributed over the entire image. Nam et al. [17] pointed out that the real color image can be modeled as mixed Gaussian noise among different color channels, and a Bayesian non-local mean denoising algorithm is employed in their paper to denoise images with non-uniform noise. Other related denoising methods are also proposed recently [18,19,20,21,22]. For example, Xu et al. [18] proposed a combined method which leverages a guided external prior and internal prior learning for non-uniform noise image denoising. A multi-channel (MC) denoising model is proposed in Xu et al. [19] to use the redundancy between color channels to distinguish different noise statistics among color channels for real color image denoising. Tian et al. [20] proposed a new Direction-of-arrival (DOA) estimation algorithm, which is suitable for dealing with unknown non-uniform noise. In Chen et al. [21], an adaptive BM3D filter was proposed to deal with non-uniform noise images, and in Plötz and Roth [22], a noise reduction algorithm for real photos was proposed. In this paper, a new image denoising method within the framework of non-local means (NLM) regarding non-uniform noise is proposed. Compare with the above algorithm, the proposed algorithm does not need to learn from the image and is computational efficient. More specifically, an evaluation operator is leveraged to measure local patch’s noise level and texture strength. After that, image pixel classification is carried out according to the evaluation value. And parameters of non-local means are heuristically selected accordingly. The main contributions of the proposed method are:

• (1) The proposed algorithm is devised for non-uniform noise images.

• (2) An evaluation operator is used to roughly obtain pixel noise level and texture strength, and then a voting strategy is used to distinguish smooth and texture image areas for more accurate denoising.

• (3) For regions containing different image texture degree, the inner parameters of NLM are adaptively selected according to patch property, leading to better denoising results.

The rest of the paper is organized as follows: In the next section, the proposed algorithm is described in detail. Experimental results are provided in Sect. 3 which compares the proposed algorithm with other state-of-the-art image denoising algorithms. Section 4 concludes this paper.

2 The Proposed Method

2.1 Non-uniform Noise Model

Non-uniform noise images can be can be expressed as in Nam et al. [17]:

$$ z(i) = x(i) + n(i)n(i) \sim N(0,\delta (i)) $$

(1)

Where $x(i)$ refers to the intensity of a noise-free image at pixel $i$, $n(i)$ is the non-uniform white noise, and $\delta (i)$ is the noise standard deviation, $z(i)$ is the non-uniform noisy image. As for color images, non-uniform noise is added respectively to the R, G, B color channels.

2.2 Framework of NLM

In 2005, Buades et al. [5] proposed a non-local means (NLM) denoising algorithm. Its basic idea is that the estimated value of the current pixel value is calculated by weighted average of the pixels in the image that have a similar neighbourhood structure, and the weight function is determined according to the similarity between pixels. NLM is defined as follows:

$$ NL[x](i) = \sum\limits_{j \in I} {\omega (i,j)z(j)} ,\quad \omega (i,j) = \frac{1}{C(i)}exp\left( { - \frac{{\left\| {z(N_{i} ) - z(N_{j} )} \right\|_{2,a}^{2} }}{{h^{2} }}} \right) $$

(2)

Where $x(i)$ is the denoised pixel value, $\omega (i,j)$ is the weight value between pixels, $z(j)$ is the pixel value of the noise image, $C(i)$ is the normalization parameter, and $h$ is the filter parameter. The algorithm makes full use of the self-similarity of the image and the redundancy of the structure information and achieves a good denoising effect.

2.3 Evaluation Operator for Noise Level and Texture Strength

From the traditional NLM, we can draw a conclusion that a large size of image patch works well on a smooth area, while a small patch size is suitable for a texture area. Furthermore, if bandwidth parameter is large, it is not suitable for retaining details, but a small bandwidth leads to a poor denoising effect. Therefore, the image can be divided into the texture and the flat area, so that we can set parameters suitable for the area for different regions, in order to obtain better denoising effects and preserve the details of the image. First, we use an evaluation operator to roughly obtain the noise distribution of the image and the structure of the image, and then a voting strategy is embedded to accurately distinguish different areas of the image.

Our evaluation operator follows the way illustrated in our previous work Hu and Luo [10] to perform rough image pixel classification. $R$ is a combination of noise level indicator $H$ and texture descriptor $F$. And the mechanism for $H$ is that noisy image’s eigenvalues $\tilde{S}_{i,1}^{2} ,\tilde{S}_{i,2}^{2}$ (in descending order) are increasing with the local noise variance $\delta_{l}^{2}$ when the original clean image patch is corrupted by noise.

$$ H(i) = \frac{{\tilde{S}_{i,1}^{2} - \tilde{S}_{i,2}^{2} }}{{\tilde{S}_{i,1}^{2} + \tilde{S}_{i,2}^{2} }} = \frac{{\mathop s\nolimits_{i,1}^{2} - \mathop s\nolimits_{i,2}^{2} }}{{\mathop s\nolimits_{i,1}^{2} + \mathop s\nolimits_{i,2}^{2} + 2\xi N^{2} \delta_{l}^{2} }},F(i) = 1 - \frac{1}{{1 + {{\varsigma_{i} } \mathord{\left/ {\vphantom {{\varsigma_{i} } {255^{2} }}} \right. \kern-0pt} {255^{2} }}}} $$

(3)

$$ R(i) = (1 - \frac{1}{{1 + {{\varsigma_{i} } \mathord{\left/ {\vphantom {{\varsigma_{i} } {255^{2} }}} \right. \kern-0pt} {255^{2} }}}})\frac{{\tilde{S}_{i,1}^{2} - \tilde{S}_{i,2}^{2} }}{{\tilde{S}_{i,1}^{2} + \tilde{S}_{i,2}^{2} }} $$

(4)

The $\varsigma_{\text{i}}$ in formula (3) refers to the second order moment of the grayscale cumulative histogram with pixel $i$ as the neighbourhood ($9 \times 9$ in the experiment), and $\tilde{S}_{i,1}^{2} ,\tilde{S}_{i,2}^{2}$ is the structural tensor of the neighbourhood. It can be seen from Eq. (3) that the value of $H(i)$ is determined by the local nature of the image (flatness and texture) and the local noise criteria. Because neither $\delta$ nor the eigenvalues are fixed, the value of $H(i)$ cannot effectively discriminate the comparative strength between noise and texture, which would leads to classification error in the texture area. Therefore, $R(i)$ can only obtain a rough pixel classification result.

2.4 Vote Strategy for Image Pixel Classification

Based on the cumulative histogram of $R$, it was determined that classification thresholds $T_{1} ,T_{2} ,T_{3}$ are determined by the 30%, 70%, and 90% of the $R$ cumulative histogram. Hence, the whole image is divided into 4 parts by way of voting. The texture area with a small noise variance ($c_{1}$), the medium texture area ($c_{2}$), the texture area with a large noise variance ($c_{3}$) and the flat area ($c_{4}$). We take a patch centered on pixel $i$ in $R$ (the patch size in the experiment is $5\times 5$). The $r(j)$ is the corresponding value in the patch. $count(r)$ represents the number of $r(j)$ that satisfied the condition. $f_{1} ,f_{2,} f_{3} ,f_{4}$ correspond to the count value of area $c_{1}$, $c_{2}$, $c_{3}$, $c_{4}$ respectively.

$$ \begin{array}{*{20}c} {if} & {r(j) \ge T_{ 3} } & {then} & {\sum\limits_{{}} {count(r) = f_{1} } } \\ {if} & {T_{2} \le r(j) \le T_{ 3} } & {then} & {\sum\limits_{{}} {count(r) = f_{2} } } \\ {if} & {T_{ 1} \le r(j) \le T_{2} } & {then} & {\sum\limits_{{}} {count(r) = f_{3} } } \\ {if} & {r(j) \le T_{ 1} } & {then} & {\sum\limits_{{}} {count(r) = f_{4} } } \\ \end{array} ,i \in \left\{ {\begin{array}{*{20}l} {c_{1} } \hfill & {if} \hfill & {\hbox{max} (f_{1} ,f_{2} ,f_{3} ,f_{4} ) = f_{1} } \hfill \\ {c_{2} } \hfill & {if} \hfill & {\hbox{max} (f_{1} ,f_{2} ,f_{3} ,f_{4} ) = f_{2} } \hfill \\ {c_{3} } \hfill & {if} \hfill & {\hbox{max} (f_{1} ,f_{2} ,f_{3} ,f_{4} ) = f_{3} } \hfill \\ {c_{4} } \hfill & {if} \hfill & {\hbox{max} (f_{1} ,f_{2} ,f_{3} ,f_{4} ) = f_{4} } \hfill \\ \end{array} } \right. $$

(5)

This is where the last area of pixel $i$ belongs. We take a patch centered on pixel $i$ in $R$, and compare the values in the patch one by one with $T_{1} ,T_{2} ,T_{3}$, and count the value that satisfies the condition. Which finally the count value of the area is the biggest, which area the pixel belongs to. Figure 1 demonstrates the classification results on noisy image and clean image respectively. The noise level is between 1 and 30. Figure 1.(c) has slight distortions in some places. These areas are smooth areas (dark blue) in Fig. 1(b) but are divided into sub-textured areas (light blue) in Fig. 1.(c). This is due to the fact that a simple texture descriptor $F$ is very sensitive to noise leading to an incorrect classification. In addition, we also presented the classification results based on the classification method in Hu and Luo [10]. Compared with (b), we can see that there is the same problem as in (c), but its distortion is more serious.

Fig. 1.

The color image with a noise variance of 1–40 is based on the block diagram of R. $c_{1}$ is orange, $c_{2}$ is green, $c_{3}$ is light blue, $c_{4}$ is dark blue. (a) original image; (b) original image classification; (c) noise image classification; (d) classification of literature [10]. (Color figure online)

2.5 Adaptive Setting of Denoising Parameters

Patch size and bandwidth are two important parameters in the NLM denoising algorithms. Taking into the consideration that a large patch size is more suitable for smooth regions and a small patch size is more suitable for texture regions, we choose the patch size for the area types $c_{1} ,c_{2}$ and $c_{4}$ to be $7 \times 7,9 \times 9$ and $13 \times 13$ respectively. Moreover, since $c_{3}$ is a texture area with a large noise variance, which implies the noise variance exceeds the ones in texture area $c_{1}$ and $c_{2}$. In order to make a better denoising effect and a better texture preservation for this area, the neighbourhood block of this area should be small enough. In our experiments, it is chosen to be a size of $5 \times 5$. As for the bandwidth parameter, it is devised as in Zeng et al. [13]:

$$ h(i) = \left\{ {\begin{array}{*{20}l} {a_{1} \times \delta \times exp( - (D_{1} /\beta_{1} )^{2} )} \hfill & {} \hfill & {if\,i \in c_{1} } \hfill \\ {a_{2} \times \delta \times exp( - (D_{2} /\beta_{2} )^{2} )} \hfill & {} \hfill & {if\,i \in c_{2} } \hfill \\ {a_{3} \times \delta \times exp( - (D_{3} /\beta_{3} )^{2} )} \hfill & {} \hfill & {if\,i \in c_{3} } \hfill \\ {a_{4} \times \delta \times exp( - (D_{4} /\beta_{4} )^{2} )} \hfill & {} \hfill & {if\,i \in c_{4} } \hfill \\ \end{array} } \right. $$

(6)

Where $a_{1} ,a_{2} ,a_{3} ,a_{4}$ are constant values 2.4, 2.5, 2.6, and 2 respectively, $\delta$ is the estimated noise variance by Chen et al. [23]. Note that Chen’s method assumes that the noise level of the entire image is fixed. Therefore, this method cannot accurately estimate the noise level in the non-uniform noisy image, $D_{i}$ is the average $R$ value for the region, $\beta$ is computed according to the MAD estimator,

$$ \beta_{j} = b*C*median\left[ {\left| {R_{j} - median\left( {R_{j} } \right)} \right|} \right],R_{j} = \left\{ {\begin{array}{*{20}c} {(R > T_{ 3} ).*R} & {} & {if\,j = 1} \\ {(R > T_{ 2} ).*R} & {} & {if\,j = 2} \\ {(R > T_{ 1} ).*R} & {} & {if\,j = 3} \\ R & {} & {if\,j = 4} \\ \end{array} } \right. $$

(7)

Where $\left| \cdot \right|$ and $median( \cdot )$ denote norm and median operator, respectively. $C = 1.4826 \times \nu$, where $\nu$ is the variance of $R_{j}$.

3 Experimental Results and Analysis

In the following experiments, RGB image is transformed into the YCbCr space first. The proposed method is only performed on the Y channel for computational efficiency, and the other two channels are denoised using a Gaussian filter. Both the synthetic non-uniform noise images and real noise color images are used in our experiments. As shown Fig. 2, six images including bikes, man-fishing, coin-fountain, ocean, building2 and woman are used. Three different non-uniform noise are added to the image, within the range of [1, 20], [1, 30] and [1, 40], respectively.

Fig. 2.

Natural images for simulation experiments from the Kodak PhotoCD Dataset. (a) bikes, (b) man-fishing, (c) coin-fountain, (d) ocean, (e) building2, (f) woman

The traditional NLM algorithm, the algorithm of Hu and Luo [10] are used for comparison. The median of the noise variance is used as the guidance for the bandwidth parameters for the compared algorithms. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used for quantitative comparison.

Table 1 shows the PSNR and SSIM results for different denoising methods on Fig. 2(a) and (b). It can be seen from Table 1 that when the image noise is small, our method has a better effect on noise removal. When the noise is large, our method is better on the texture protection of the image. Figure 3 shows the denoised results on noisy image that is contaimed by noise range [1, 40]. It can be clearly seen that our algorithm is better for noise removal and texture protection. Next, our method is evaluated on real images from the dataset provided by Xu et al. [18], where images are captured either indoor or outdoor lighting conditions with different types of camera and camera settings. Each noise image in this dataset has an average image, which can be regarded as a “ground truth”. Figure 4 shows the selected real noise images in our experiment. Table 2 shows the PSNR and SSIM results for different denoising methods on this dataset. We can see that our method is more capable of real image denoising. When our method is applied to a real natural image, the protection of the texture of the image and the effect of denoising are superior to other methods. Figure 5 shows the denoised images of a scene captured by Canon 5D Mark III at ISO = 3200. We can see that our method is better at texture preservation and noise removal.

Table 1. PSNR/SSIM results for different denoising methods on Fig. 2 (a) and (b).

Fig. 3.

Denoised images on image bike containmed by Gaussian noise within the noise standard deviation [1, 40]. Zoom for better comparison

Fig. 4.

Seven cropped noiseless images used in the experiment.

Table 2. PSNR/SSIM results for different denoising methods on natural images.

Fig. 5.

Denoised images of a region cropped from the real noisy image “Canon 5D Mark 3 ISO 3200 1” [18] by different methods. The images are better to be zoomed in on screen.

4 Conclusions

A non-uniform denoising method is proposed in this paper under the framework of non-local means. More specifically, a pixel-wise evaluation operator is devised to describe local patch’s noise level and texture strength. After that, image pixel classification is carried out according to the evaluation value. And parameters of non-local means are heuristically selected in order to achieve a balance between noise removal effect and texture retention effect. The algorithm proposed in this paper has a good effect on denoising and detail preservation of natural images. In the future work, we will investigate the parameter tuning in NLM as well as the speed up, and the generalization to medical image denoising.