Research on Image Quality Assessment in Foggy Conditions

Abstract

Recently, no-reference image quality assessment has been followed with interest by researchers, but no-reference quality assessment of foggy images is rarely reported. This paper proposes a no-reference quality assessment of foggy images based on codebooks. Proposed method aims to be consistent with human subjective perception. The Technical roadmap of the method is from feature extraction to quality metric model. The features need to reflect characteristics of foggy images exactly. Then codebook is built by the features and used to acquire feature vectors of training images by encoding. At last, regression is introduced to quality model construction by feature vectors and subjective ratings. The method is tested in simulation library of foggy images. Results show that Pearson Linear Correlation Coefficient (PLCC) and Spearman rank Order Correlation Coefficient (SROCC) are both above 0.99. And compared with state-of-art algorithms, our method perceives higher performance, and it can be a good predictor of subjective perception of foggy images.

1 Introduction

In National Standard of China GB/T 27964-2011 [1], fog is classified to five grades. Mist is a weather phenomenon of visibility above 1 km. Visibility of fog is from 500 m to 1 km. Visibility of heavy fog is from 200 m to 500 m. Visibility of strong fog is from 50 m to 200 m. Visibility of ultra strong fog is below 50 m.

Currently researches on foggy images processing have focused on image defog and enhancement, quality assessment of image in foggy conditions has focused on image defog [2, 3], the result of assessment is used to determine defog effect. Research on quality assessment of foggy image has rarely been reported. How to evaluate quality grades of foggy image has important significance to real-time forecasting of fog, defog algorithms and foggy image enhancement.

It is difficult to obtain a reference image, only no-reference method can be used to evaluate quality of foggy image. No-reference IQA can be divided into two categories, distortion specific method and non distortion specific method.

Distortion specific method refers distortion to evaluate is known. For example, Zhou Wang [4] proposed algorithm to evaluate JPEG distortion. Sheikh [5] proposed an algorithm based on NSS to evaluate JP2 K distortion. But in most cases, type of distortion is unknown, and it is difficult to determine which no-reference algorithm to use. This limits the application of these algorithms.

Non-distortion specific method refers to unknown distortion, which is a general no-reference IQA algorithm. We can also divide it into two categories, OA-DU (Opinion Aware- Distortion Unknown) and OU-DU (Opinion Unaware- Distortion Unknown). OA-DU is the algorithms of requiring subjective score in training process, such as BIQI [6], BRISQUE [7], CBIQ [8], CORNIA [9] and so on. OU-DU is the kind of algorithm which does not require subjective score. For example, in QAC [10] algorithm, subjective score does not be used directly. But in training process full-reference IQA algorithm was used to acquire ‘subjective rating’. And the benefit of the algorithm does not require classical image databases to train, so that the scope of the training images was larger. NIQE [11] algorithm aims to require the reference by training, which is equivalent to construct a ‘reference image’. It may not select training images from classical databases. The benefit is not dependent on the database and has adaptability.

Quality assessment of foggy image is also a kind of distortion specific method. However, fog is not only unlike any known distortion, but also is not included in classical image databases.

Figure 1 shows quality assessment results of foggy image by new presented no reference algorithms of NIQE and CORNIA. X-axis is 20 fog densities which gradually rise, y-axis is quality score. In figure, we can see that the two no-reference algorithms cannot evaluate quality of foggy image in accordance with subjective perception.

Fig. 1.

Quality assessment curves of foggy image by different algorithms (Results of the two methods are normalized)

This paper is to propose a no-reference IQA method of foggy specific, the results of its evaluation is consistent with human subjective perception.

The remainder of this paper is organized as follows. Section 2 analyzes features of foggy image. Section 3 presents structure of algorithm design. Section 4 reports the experimental results and Sect. 5 concludes the paper.

2 Feature Ananlysis of Foggy Images

Influence on image quality in foggy weather contains three aspects. Firstly, fog breaks structure information of image. Secondly, existence of fog adds several channel information as a mask, and it enhances overall luminance. Lastly, new multiplicative information is derived from fog, it makes image blur and degrades contrast.

Figure 2 shows that contrast and gradient of the image apparently decrease with the increase of fog density.

Fig. 2.

Contrast, gradient map (1st row is original image, 2nd row is contrast map, and 3rd row is gradient map. Every row is from mist to ultra strong fog)

Although contrast and gradient can well reflect the characteristics of foggy image, in order to get a better quality mapping, natural scene statistics (NSS) are used to learn image features in foggy conditions.

In BRISQUE proposed by Anish Mittal [7], MSCN coefficients are first used in the algorithm to represent the natural scene statistics of images.

$$ \mathop I\limits^{ \wedge } (i,j) = \frac{I(i,j) - \mu (i,j)}{\sigma (i,j) + c} $$

(1)

In formula (1), $I(i,j)$ represents the image, and $\mathop I\limits^{ \wedge } (i,j)$ are MSCN coefficients. $c = 1$ is a stable parameter.

$$ \mu (i,j) = \sum\limits_{k = - K}^{K} {\sum\limits_{l = - L}^{L} {W_{k,l} I_{k,l} (i,j)} } $$

(2)

$$ \sigma (i,j) = \sqrt {\sum\limits_{k = - K}^{K} {\sum\limits_{l = - L}^{L} {W_{k,l} (I_{k,l} (i,j) - \mu (i,j)} } )^{2} } $$

(3)

Here $\left. {W_{k,l} } \right|k = - K \ldots K,l = - L \ldots L$ is two-dimensional Gaussian weighting function of $K \times L$ block size, here we take $K = L = 3$.

Previous researches on the natural scene statistics show that generalized Gaussian distribution (GGD) can effectively obtain coefficients changes in natural images and their distorted images.

$$ f(x;\alpha ,\beta ) = \frac{\alpha }{2\beta \varGamma (1/\alpha )}\exp ( - (\frac{\left| x \right|}{\beta })^{\alpha } ) $$

(4)

$$ \varGamma (a) = \int_{0}^{\infty } {t^{a - 1} e^{ - t} dt} \quad a > 0 $$

(5)

($\alpha ,\beta$) of generalized Gaussian distribution can be effectively estimated [12]. In formula (1), there is a relatively regular distribution, but the distortion changed this distribution, and this change can be calculated by the product of four directions of adjacent coefficients, which can be fitted by adjacent asymmetric generalized Gaussian distribution (AGGD).

$$ f(x;\gamma ,\beta_{l} ,\beta_{r} ) = \left\{ {_{{\frac{\gamma }{{(\beta_{l} + \beta_{r} )\varGamma (\frac{1}{\gamma })}}\exp ( - (\frac{x}{{\beta_{r} }})^{\gamma } )\forall x \ge 0}}^{{\frac{\gamma }{{(\beta_{l} + \beta_{r} )\varGamma (\frac{1}{\gamma })}}\exp ( - (\frac{ - x}{{\beta_{l} }})^{\gamma } )\forall x \le 0}} } \right. $$

(6)

AGGD parameters ($\gamma ,\beta_{l} ,\beta_{r}$) can also be effectively estimated [13].

Mean of the distribution is one of the parameters.

$$ \eta = (\beta_{r} - \beta_{l} )\frac{{\varGamma (\frac{2}{\gamma })}}{{\varGamma (\frac{1}{\gamma })}} $$

(7)

18 parameters are extracted in one scale, but extracted information will not produce significant changes. So two scales are selected, one is scale of the original image, and another scale is after the next sample image. On each scale a 18-dimensional feature extraction, feature vector to form a total of 36 dimensions. Add up to luminance, contrast, and gradient magnitude, dimension of features is 39.

3 Structure of Proposed Algorithm

Figure 3 shows structure of the algorithm design, which contains feature extraction, codebook construction and encoding, feature training of support vector regression. Codebook construction follows the CORNIA Algorithm [9].

3.1 Local Feature Extraction

In this paper, an image is divided into B $\times$ B size of blocks. In each block, NSS are extracted to forms local features vector $x_{i}$.

Fig. 3.

Flow chart of proposed algorithm

3.2 Codebook Construction

When features extraction of all the training images complete, extracted features are clustered by K-mean. Then codebook is built. Codebook is represented by $D_{{ [ {\text{d}} \times {\text{K]}}}} = [D_{1} ,D_{2} \ldots D_{K} ]$. Code word $D_{i(i = 1,2 \ldots K)}$ is cluster centers. Here none of the training images have labels. And a general codebook not sparse is used in this paper because that encoding is more important than codebook construction [14].

3.3 Coding Local Features

A soft-assignment encoding is used to calculate distance between code words and the local features. Denotes $s(i,j) = x_{i} \cdot D_{j}$ as a partial similarity between local features vector $x_{i}$ and code word $D_{j}$. Local features $x_{i}$ are represented by

$${\begin{aligned} x_{i} = [\hbox{max} (s_{i1} ,0), \ldots ,\hbox{max} (s_{ik} ,0), \hfill \\ \hbox{max} ( - s_{i1} ,0)], \ldots ,\hbox{max} ( - s_{ik} ,0) \hfill \\ \end{aligned}}$$

(8)

Similarity has positive and negative components, it will increase the degree of recognition.

3.4 Features Pooling

A coefficient matrix $C_{2K \times N} = [c_{1} ,c_{2} \ldots c_{n} ]$ is used in encoding, and $c_{i}= [c_{i,1} , c_{i,2} \ldots c_{i,2k} ]^{T}$. In order to support vector regression, a fixed-length feature vectors are needed. A lot of image quality assessment algorithms [15, 16], the percentage of pooling have been adopted. The worst image region of image affects perception greatest. In image classification, the maximum pooling proved to have classification results. This paper uses the maximum pooling, which can be considered as a special percentage pooling. Max pooling is

$${\mathop \beta \limits^{ \wedge } = \psi_{ \hbox{max} } (C)}$$

(9)

Here we define $\psi_{ \hbox{max} }$ as the maximum value of each row.

$${\mathop {\beta_{i} }\limits^{ \wedge } = \hbox{max} \left\{ {c_{1i} ,c_{2i} \ldots c_{Ni} } \right\}}$$

(10)

$\mathop {\beta_{i} }\limits^{ \wedge }$ will be used to regression training.

4 Experiments Results and Analysis

4.1 Database of Foggy Images

There is no established databases of foggy image, in order verify the proposed we have set up foggy image database. 29 reference images are selected in LIVE database. For each image, we plus fog from 0 % to 95 % density by Photoshop software in accordance with interval of 5 % density. Each group of images contains a reference image and 19 foggy images. The entire image database contains 29 images and 551 foggy images. Fog density of each image is used as DMOS. In this paper, 80 % images of database are used for training, and 20 % for test.

4.2 Experimental Results

In experiment, block size is 96 $\times$ 96, and the size of codebook is 500. In order to evaluate performance of algorithm, two correlation coefficients of SROCC and PLCC are introduced. SROCC is monotonic measurement, and PLCC is consistency measurement. Table 1 shows the evaluation results.

Table 1. Comparison to four no-reference IQA algorithms

4.3 Experimental Analysis

From Table 1, it is clear that our algorithm is superior to CORNIA and NIQE. It mainly because that the two classical algorithms are both based on existing image quality assessment database, but fog distortion is not involved, so evaluation results are not good. The proposed algorithm aims to appropriate features, and it is based on foggy image database, so the evaluation results are better.

4.3.1 Selection of Codebook Size

We change codebook size of 50, 100, 200, 300, 400, 500, 1000, the performance is shown in Fig. 4. When the size of the codebook reaches 500, the performance is no more increase. Therefore, we determine codebook size as 500.

Fig. 4.

Effect of codebook size

4.3.2 Effect Encoding

Codebook encoding impacts larger on the final quality score. Here we select three encoding pattern to comprise, which are hard-assignment encoding, LLC-assignment encoding [17], and soft-assignment encoding. The indexes of comparison are SROCC, PLCC, and algorithm time. The comparison is selected codebook size of 500 as in Table 2.

Table 2. Comparison to different encoding

In Table 2, Soft-assignment may be the best encoding pattern.

5 Conclusions

This paper presents an efficient codebook-based no-reference quality assessment algorithm of foggy image, which performance is higher than some classical algorithms. The algorithm selects the best codebook size and the optimal codebook encoding. This algorithm obtains a better linearity and consistency with foggy image database. The proposed method gives a good solution to the problem of image quality assessment in conditions and provides a reference to solve foggy weather forecasting. The next step will be further looking for features to reflect foggy image quality well, and more accurate image quality assessment of foggy image.