Visibility enhancement using an image filtering approach

Abstract

The misty, foggy, or hazy weather conditions lead to image color distortion and reduce the resolution and the contrast of the observed object in outdoor scene acquisition. In order to detect and remove haze, this article proposes a novel effective algorithm for visibility enhancement from a single gray or color image. Since it can be considered that the haze mainly concentrates in one component of the multilayer image, the haze-free image is reconstructed through haze layer estimation based on the image filtering approach using both low-rank technique and the overlap averaging scheme. By using parallel analysis with Monte Carlo simulation from the coarse atmospheric veil by the median filter, the refined smooth haze layer is acquired with both less texture and retaining depth changes. With the dark channel prior, the normalized transmission coefficient is calculated to restore fogless image. Experimental results show that the proposed algorithm is a simpler and efficient method for clarity improvement and contrast enhancement from a single foggy image. Moreover, it can be comparable with the state-of-the-art methods, and even has better results than them.

Introduction

Visibility is the ability to see through air, irrelevant to the sunlight or the moonlight. Clear clean air has a better visibility than air polluted with dust particles or water droplets. There are a number of factors affecting visibility, including precipitation, fog, mist, haze, smoke, and in coastal areas sea spray, and they are generally composed principally of water droplets or the particles whose size cannot be ignored for the wavelength. The difference between fog, mist, and haze can be quantified as the visibility distance. Visibility degradation is caused by the absorption and scattering of light by particles and gases in the atmosphere. Scattering by particulate, impairs visibility more severe than absorption. Visibility is mainly reduced by scattering from particles between an observer and a distant object. Particles scatter light from the sun and the rest of the sky through the line of sight of the observer, thereby decreasing the contrast between the object and the background sky.

Images or videos acquired are often affected by visibility in surveillance, traffic, and remote sensing systems, due to light scattering and absorbtion by the atmospheric particles and water droplets. For the rest of the paper, the the atmospheric particles and water droplets from the mist, haze, fog, smog, and cloud are not distinguished for convenience. Visibility enhancement methods of degraded outdoor images fall into two main categories. The first category is non-model-based methods, such as histogram equalization[1, 2], Retinex theory[3], and Wavelet transform[4]. However, the shortcomings of these methods are that they have less effectiveness on maintaining the color fidelity, and also seriously affect the clear region. The second category is model-based methods, who can achieve better results by modeling from the scattering and absorbtion, but usually need additional assumptions of imaging environment or imaging system, such as scene depth[5–7] or multiple images[8–11]. Nonetheless, when their assumptions are not accurate, the effectiveness is greatly compromised. Consequently, visibility enhancement using a single image for haze removal has become a key focus of ongoing studies in image restoration in recent years. Very recently, Sun et al.[12] provided a de-fogging method based on Poisson matting which uses the boosting brush to tune the transmission coefficient in the selected region to generate the fogless image by solving Poisson equations, which combines the local operations including channel selection and local filtering. In this respect, the least square filters[13, 14] and the patch-based method[15] attract great attention in visibility enhancement. But this de-fogging method needs the user to manually adjust the local scattering coefficient and scene depth. During the subsequent years,the four approaches in[16–21] were proposed for single image dehazing without the need of any extra information. The algorithm in[16] is based on color information under the assumption that the surface shading and transmission functions are locally uncorrelated. This method can only be applied to color images. Those algorithms in[17, 18, 20] can be applied to both gray and color images, but they are computationally intensive. The results of the algorithm in[17] depend on the scene saturation, and both of the algorithms in[18, 20] introduce dark channel prior to remove haze, but sometimes have very serious color distortion and poor results. Then, Tarel et al.[19, 21] proposed the visibility restoration method with low complexity for gray and color images, which adopted white balance, gamma correction, and tone mapping to maintain color fidelity. But, the results of some degraded images processed by the algorithm in[19, 21] still show high level residual haze.

Inspired by a multilayer image formed from several types of measurement on the same detection area covered by a single pixel[11, 22], the hazy image can be seen as a combined one in the presence of a multilayer structure consisting of both the clean attenuated component image and the haze layer, which mainly result from the medium absorbtion and atmospheric scattering, respectively. Since the atmospheric veil almost has no specific edges or textures, we may adopt image filtering approach to estimate the haze layer from the hazy image. In this article, different from the existing visibility restoration methods in the previous studies, with the help of dimension reduction technique, the proposed method utilizes the image filtering approach consisting of the median filter and the truncated singular value decomposition to estimate atmospheric veil with dark channel prior to restore the haze-free image. And a comparison of the proposed approach with the state-of-the-arts is also presented.

The rest of this article is organized as follows. In “Image degradation model” section, we briefly review the outdoor degraded bilayer image model. We adopt this model for visibility enhancement from a single hazy image in “The image filtering approach” section. The proposed scheme of the image filtering approach using low-rank technique that produces the haze layer approximation is described here. The experimental results and analysis of the comparison between developed approach and the state-of-the-arts are shown in “Results and analysis” section, and the conclusion and future work are given at the last section.

Image degradation model

Assume that the atmosphere is homogenous in space, the image degraded model caused by hazy weather conditions is often described as[16, 19, 20]:

$$\mathbf{I}\left(x\right)=\mathbf{J}\left(x\right)t\left(x\right)+\mathbf{A}(1-t(x\left)\right)$$

(1)

where x denotes pixel position, I∈ C^M×Nis an observed image, J is scene radiance, A is global atmosphere light, and t is medium transmission. J(x)t(x) is direct attenuation term, representing the scene radiation decay effect in the medium; A(1−t(x)) is airlight term, describing the light scattering from atmosphere particles inducing color distortion.

In fact, since the atmospheric layer depends on the object depths, visibility restoration is related with the estimation of the true colors of the objects, the haze properties, and the scene depth map. Due to lack of scene structure or scene depth for a single gray or color image, it is impossible to accurately distinguish transmission coefficient t and global atmosphere light A. Therefore, the degraded image model (1) cannot directly be used to reach contrast enhancement. So, we take the airlight term on the right-hand side of Equation (1) as the haze layer:

$$\mathbf{B}=\mathbf{A}(1-t(x\left)\right)$$

(2)

Then Equation (1) can be written as in another form

$$I\left(x\right)=J\left(x\right)\left(1-B\left(x\right)/A\right)+B\left(x\right)$$

(3)

Through inverse operation for Equation (3), so the final image after visibility enhancement is acquired as

$$J\left(x\right)=\left(I\left(x\right)-B\left(x\right)\right)/\left(1-B\left(x\right)/A\right)$$

(4)

As a consequence, instead of estimating the medium transmission coefficient t, the haze removal algorithm can be decomposed into several steps: inference of B(x) from I(x), estimation of A, and derivation of J(x) after inverting Equation (3).

The image filtering approach

From the multilayer image model (1), each pixel of the observed image I is composed of two components: the scene radiance and the airlight, respectively. Assume that the global atmosphere light A is isotropic. Therefore, the estimation of the transmission coefficient t can be replaced by the airlight term A(1−t(x)) also called the haze layer B(x) under the supposed condition of the constant atmosphere light A. According to the mathematical formula derivation in “Image degradation model” section, for the modified image degraded model in Equation (4), there are two steps for visibility enhancement. The first step of image restoration is to infer the haze layer B(x) with the pixel position x. Through the observation of the mist, haze, or fog, haze density is proportional to the scene depth. Attributing to its physical properties, the haze layer has two constraints: 0 ≤ B(x) and not more than the minimal one of the components of I(x). And the whiten image as the channel minimal value at each pixel of gray or color image is derived as[19]:

$$\mathbf{g}\left(x\right)=min\left(\mathbf{I}\left(x\right)\right)$$

(5)

Tarel and Hautiere[19] adopted a fast visibility restoration algorithm by using the median filter to compute atmosphere veil from the whiten image. However, this algorithm brings about the relatively severe atmosphere veil discontinuities. To tackle this problem, we proposed the dimension reduction technique to correct the preliminary haze layer estimation. First, the median filter is used to calculate the coarse haze layer prediction F(x) generated from the whiten image g(x) as follows:

$$\mathbf{F}\left(x\right)={\text{median}}_{\Omega }\left(\mathbf{g}\right(x\left)\right)$$

(6)

where Ω denotes the local neighborhood at each pixel.

It is observed that the haze layer is smooth, while retaining the depth changes. To remove the redundant image texture affecting the haze layer, we first explain how the low-rank technique can be applied to the coarse haze layer F(x) to extract the corrected haze layer. Let F _i be the i th sample in F and F be a column stacked representation of F(x), i.e., F is a matrix of size MN × L, whose each row contains the$\sqrt{L}\times \sqrt{L}$ patch around the location of F _i in the image F.

By removing the mean value from each row, the difference matrix$\bar{\mathbf{f}}$ is derived as

$$\bar{\mathbf{f}}(i,p)=\mathbf{f}(i,p)-\frac{1}{\mathit{\text{MN}}}\sum _{i=1}^{\mathit{\text{MN}}}\mathbf{f}(i,p)$$

(7)

To reduce calculation time, the matrix${\bar{\mathbf{f}}}^T\bar{\mathbf{f}}$ can be decomposed by this form:

$${\bar{\mathbf{f}}}^T\bar{\mathbf{f}}=\mathbf{U}{\mathrm{\Sigma }}^2{\mathbf{U}}^{\text{T}}$$

(8)

where U is the matrix of eigenvectors derived from${\bar{\mathbf{f}}}^T\bar{\mathbf{f}}$,$\mathrm{\Sigma }=\text{diag}\left\{{\sigma }_1,\begin{array}{l}{\sigma }_2,\dots \end{array},{\sigma }_r\right\},$$r=\text{rank}\left({\bar{\mathbf{f}}}^T\bar{\mathbf{f}}\right)$, and its diagonal singular values σ₁ ≥ σ₂ ≥ ⋯ ≥ σ _r > 0.

After the projection of$\bar{\mathbf{f}}$ onto the new basis U, the reformed matrix$\widehat{\mathbf{f}}$ is

$$\widehat{\mathbf{f}}=\bar{\mathbf{f}}\times \mathbf{U}$$

(9)

Therefore, the new axes are the eigenvectors of the correlation matrix of the original variables, which captures the similarities of the original variables based on how data samples are projected onto them. As we know, if the eigenvalues are very small and the size of image patch from a single hazy image is large enough, the less significant components can be ignored without loss much information. Only the first K eigenvectors are chosen based on their eigenvalues. Since the parameter K should be both large enough to allow fitting the characteristics of the data and small enough to filter out the non-relevant noise and redundancy; therefore, the K largest values are selected by parallel analysis (PA) with Monte Carlo simulation. Many literatures[23, 24] prove that PA is one of the most successful methods for determining the number of true principal components. In our algorithm, without the assumption of a given random distribution, we generate the artificial data by randomly permuting each element across each patch in the image F. And the improved PA with Monte Carlo simulation estimates data dimensionality as

$$K=max\left(\left\{1\le p\le r|{\sigma }_p\ge {\alpha }_p\right\}\right)$$

(10)

where σ _p and α _p are the singular values of the raw image$\bar{\mathbf{f}}$ and the simulated data, respectively. The intuition is that α _p is a threshold for σ _p below which the p th component is judged to have occurred due to chance. Currently, it is recommended to use the singular values that corresponds to a given percentile, such as the 95th of the distribution of singular values derived from the random data. From the diagonal singular values, K(K ≪ r) principal components is truncated as σ _i (i = 1,2,…,K).

The eigenvectors of the matrix$\bar{\mathbf{f}}$ can be used for multivariate analysis of the coarse layer F. The image is decomposed into a sum of components from the primary to the secondary. To remove residual image textures, we develop one novel filtering scheme based on projection onto the signal subspace spanned by the first K eigenvectors with noise removal and texture reduction. The straightforward way to restore a corrected haze layer is to directly project the MN by L matrix$\bar{\mathbf{f}}$ onto the subspace spanned by the top K eigenvectors. The projected weight matrix on the signal subspace basis is

$$W\left(l,p\right)=\widehat{\mathbf{f}}\left(·,l\right)\setminus \bar{\mathbf{f}}\left(·,p\right)$$

(11)

where l,p = 1,2,…,L, the matrix left division operator ∖ returns a basic solution with K non-zero components where K is less than the rank of the eigenvector U. And the projected matrix is reconstructed based on weighted subspace basis:

$$R\left(i,p\right)=\widehat{\mathbf{f}}\left(i,l\right)\times \mathbf{W}\left(l,p\right)+\frac{1}{\mathit{\text{MN}}}\sum _{i=1}^{\mathit{\text{MN}}}\mathbf{f}\left(i,p\right)$$

(12)

where i = 1,2,…,MN; p = 1,2,…,L; and l = 1,2,…,K. Then, through the overlap averaging scheme, the reorganized reconstructed refined haze layer approximation V _c is reshaped as

$${\mathbf{V}}_c\left(x\right)=\frac{1}{N_x}\sum _{p=1}^{L}R\left(i,p\right)$$

(13)

where N _x is the number of a pixel used in patch stacks for the whole image; and the variables p,L are defined as above.

Since the strength of the image degradation caused by haze is less than the minimum intensity of image pixels, so the final revised atmospheric veil can be obtained as

$$B\left(x\right)=q·max\left\{V_c\left(x\right),0\right\}$$

(14)

where the parameter q ∈ (0,1).

In single-image dehazing applications, the global atmosphere light A is usually estimated from pixels with most dense haze. However, the brightest pixels may be the white objects. In our approach, the dark channel prior in[18, 20] is also used to improve the approximation of the atmosphere light. For simplicity, Let B _a = B(x)/A. Considering the results of the division operation B(x)/Amay be more than one, the normalization is necessary. Therefore, the medium transmission coefficient is calculated as

$$t\left(x\right)=1-q·B_a\left(x\right)/\underset{x\in \Re }{max}\left(B_a\left(x\right)\right)$$

(15)

Through substituting Equation (14) and (15) in Equation (4), the final haze-free image J(x) after contrast enhancement is acquired from a single hazy image.

Results and analysis

The degraded gray and color images caused by haze were used for both the subjective evaluation and the objective evaluation of the dehazing performance between the proposed algorithm based on image filtering approach using low-rank technique and the state-of-the-arts[20, 21]. For different types of test images, a lot of experiments for image restoration have been done and the results were compared to validate the proposed method. In our experimental setting, the parameter q = 0. 90, the sliding window size of the median filter is 15×15 pixels, and the patch size L = 25. Table1 demonstrates that the proposed algorithm and the methods in the literatures[20, 21] were compared in computation time using Matlab version 7.8 on the platform of Pentium(R) Dual-Core CPU E5800 @ 3.20 GHz 2-GB cache for 300 × 200 and 250 × 400 test images, respectively. Compared with the state-of-the-art methods, the proposed algorithm based on image filtering approach has a little longer computation time than that of[21], and much shorter calculation time than that of[20]. Figure1 shows mist removal results of current excellent methods[20, 21] and the proposed algorithm for visibility enhancement from a single degraded gray image, respectively. And the resolution improvement and contrast enhancement from three color misty images with different haze density containing different scenes, such as, road, tree, car, and house, for the proposed method is demonstrated in Figures2 and3 where the restored image by the proposed algorithm is compared with those by the authors of[20, 21]. And Figure4 demonstrates that a comparison of visibility enhancement results from a color image with high-density haze between the proposed algorithm and the two popular methods[20, 21]. Except for the computation speed and the visual effect, several objective evaluation criteria are also used to analyze the experimental results. The evaluative criteria in[19] including three indicators$e,\bar{r},$ and H, which denote the newly visible edges after restoration, the average visibility enhancement, and the image information entropy, are used here to compare two gray level or color images: the input image and the restored image. The quantitative evaluation and comparative study of He et al.[20], Tarel et al.[21], and our algorithm have also been implemented on three test images in this experiment. Table2 gives the similar or better quality results of the proposed method compared with the other two popular algorithms. For these assessment indicators, the higher value means the greater visibility of the restored image and the better dehazing effect. However, according to the visual results shown in from Figures1,2,3 and4, we can find that these objective metrics, i.e.,$e,\bar{r}$ and H introduced in the literature[19] are not exactly consistent with the human subjective reception. The proposed method works well for a wide variety of outdoor foggy images and can remove more haze and restore clearer images with more details. As seen from the experimental results, the proposed algorithm with less computation time can be comparable with the state-of-the-art methods, and even reach better effectiveness than them on haze removal.

Figure 1

Visual comparison of the dehazing results of these methods for a misty gray image. These methods are[20, 21] and the proposed algorithm using image filtering approach. (a) Original image, (b)[20], (c)[21], (d) the proposed algorithm.

Figure 2

Visual comparison of the dehazing results of these methods for a foggy color image. These methods are[20, 21] and the proposed algorithm using image filtering approach. (a) Original image, (b)[20], (c)[21], (d) the proposed algorithm.

Figure 3

Visual comparison of the dehazing results of these methods for a heavy foggy color image. These methods are[20, 21] and the proposed algorithm using image filtering approach. (a) Original image, (b)[20], (c)[21], (d) the proposed algorithm.

Figure 4

Visual comparison of the dehazing results of these methods for a dense foggy color image. These methods are[20, 21] and the proposed algorithm using image filtering approach. (a) Original image, (b)[20], (c)[21], (d) the proposed algorithm.

Table 1 Comparison of computation time between [[20, 21]] and the proposed algorithm (unit: seconds)

Table 2 Rate e of new visible edges, mean ratio $\bar{r}$ of the gradients at visible edges and the image information entropy H for these methods on three test images

Conclusion and future work

We analyze and compare the experimental results in visual effects, speed, and objective evaluation criteria. Though comparing the results, we demonstrate the advantage and disadvantage of these methods. We have proposed simple but powerful algorithm based on median filtering using low-rank technique for visibility enhancement from a single hazy image. Since the computational complexity of the low-rank technique is low, it is shown that the proposed approach for haze removal is fast, and can even achieve better results than the state-of-the-art methods in a single image dehazing.

However, the proposed approach maybe not work well for the far scenes with heavy fog and great depth jump. The restored image has the halos or residual haze at depth discontinuities that can be observed in these experimental results. And another shortcoming is unable to obtain the actual value of global atmosphere light A. To overcome these constraints of our current method, we intend to incorporate better edge-preserving image filtering method with low complexity and other techniques. This is our future research.