Abstract
This paper presents a low-light image restoration method based on the variational Retinex model using the bright channel prior (BCP) and total-variation minimization. The proposed method first estimates the bright channel to control the amount of brightness enhancement. Next, the variational Retinex-based energy function is iteratively minimized to estimate the improved illumination and reflectance using the BCP. Contrast of the estimated illumination is enhanced using the gamma correction and histogram equalization to reduce a color distortion and noise amplification. Experimental results show that the proposed method can provide the better restored result than the existing methods without unnatural artifacts such as noise amplification and halo effects near edges.
Similar content being viewed by others
1 Introduction
Various imaging systems that consist of an optical system and imaging sensor have been widely used in various industrial and consumer application fields such as advanced driver assistance systems (ADAS), surveillance systems, robot vision, and medical imaging [1]. To acquire the high-quality images, sophisticated but compact imaging systems are particularly useful by reducing the size of a sensor and increasing the pixel density. However, in low-light condition, it is hard to obtain the high-quality input image since the sensor cannot sufficiently react to the very small amount of incoming light. In addition, the interference of the light between the reduced pixels leads to the chromatic noise. As a result, the low-light artifacts make the post-processing step difficult such as object recognition, detection, and tracking. To solve this problem, various image brightness enhancement methods have been proposed in the literature.
Histogram-based methods enhance the contrast of an input image by redistributing the intensity bins or modifying the cumulative distribution function (CDF) at the low-computational cost [2–7]. However, since the low-light image provides the narrow histogram distribution, the CDF has many abrupt increases, which results in the brightness saturation and color distortion.
Recently, the haze removal method is adapted to the low-light image enhancement because the inversion of the low-light input image shows the visually similar property to the hazy image [8]. This method is based on the statistical prior knowledge of the haze-free natural image called dark channel prior (DCP) [9]. The DCP is estimated by finding the minimum value among the R, G, and B pixels, and it is used to estimate the transmission map of the non-haze region. Although the haze removal-based method can better preserve the bright region than the histogram-based methods without brightness saturation, it cannot avoid the noise amplification and color distortion in the resulting image.
On the other hand, Retinex-based methods are based on the human color perception system [10, 11]. Provenzi et al. mathematically analyzed the Retinex algorithm and demonstrated the performance according to various parameters such as threshold and the number of path of light paths to a pixel [11]. The Retinex methods enhance the input image by eliminating the illumination component using the low-pass filtering and logarithmic transformation [12–14]. However, the resulting image shows the halo effect near the edges.
To solve this problem, the state-of-the-art Retinex methods incorporate with the variational optimization method using l 1- and l 2-norm minimization [15, 16]. Li et al proposed a variational Retinex method using the constraint term that minimizes the combined reflectance component and the image gradient to reduce the halo effect [17]. Ma et al. performed the variational optimization by minimizing the constraint term on the reflectance component using l 1-norm [18]. Fu et al. proposed the bright channel prior (BCP) to reduce the halo effect and color distortion using l 2-norm minimization on the illumination and reflectance components [19]. However, since the BCP is estimated in the image patch, it results in the blocking and halo artifacts in the resulting image.
In this paper, we present the low-light image restoration method using the variational optimization-based Retinex model and BCP. In order to reduce the blocking and halo artifacts, the proposed method estimates the pixel-wise bright channel. In addition, the l 1-norm minimization of the reflectance component provides the edge-preserving noise reduction without the halo effect near the edges. In order to reduce color distortion and brightness over-enhancement, the proposed method performs the local histogram equalization on the illumination component. Finally, the restored result is generated by combining the optimized reflectance and enhanced illumination components. Experimental results show that the proposed method can provide significantly enhanced result without the halo effect, noise amplification, and color distortion.
This paper is organized as follows. Section 2 briefly describes variational Retinex model and total variation as theoretical background. Section 3 presents the proposed variational Retinex model. Experimental results are shown in Section 4, and Section 5 concludes the paper.
2 Theoretical background
2.1 Variational Retinex model using bright channel prior
Retinex-based image enhancement methods assume that an image can be formulated as a multiplication of the illumination and reflectance components as
where g represents the observed image, f L is the illumination component, which can be regarded as the light directly entered into an imaging sensor, and f R is the ratio of the reflected light by the object.
The conventional Retinex-based enhancement method is defined as
where ∗ represents the convolution operator, G is the Gaussian low-pass filter, and logf R is the reflectance image [12]. This method assumes that the slowly changing illumination component can be estimated by the Gaussian low-pass filtered version of the input image. Next, the reflectance component is computed by subtracting the estimated illumination component from the input image. However, the halo effect around the edges is generated according to the size of the Gaussian low-pass filter.
To solve this problem, the multi-scale Retinex (MSR) algorithm estimates several illumination components using multiple, different Gaussian low-pass filters [13]. The resulting enhanced image is reconstructed using the weighted sum of multiple reflectance components. However, the MSR method cannot completely remove the halo effect near the edges.
To solve the halo effect problem, a variational Retinex model using the l 1- and l 2-norm minimization on the illumination and reflectance components was recently proposed [15]. Specifically, Fu et al. restored the low-light image using the bright channel prior on the reflectance component in the variational Retinex method without logarithmic transform and Gaussian low-pass filtering [19]. The energy function is defined as
where g and g b represent the input low-light image and its bright channel, respectively. \(\left \| f_{R}f_{L}-g \right \|^{2}_{2}\) represents the data fidelity term, and \(\left \| \nabla f_{L} \right \|^{2}_{2}\) and \(\left \| \nabla f_{R} \right \|^{2}_{2}\) are the smoothness constraint term on the illumination and reflectance components. \(\left \| f_{L}-g_{b} \right \|^{2}_{2}\) penalizes the brightness of illumination component. α, β, and γ represent positive regularization parameters.
The bright channel g b is defined as
where ω(x) represents the image patch whose center is located at x and c∈{R,G,B} [19].
In the Retinex theory, since the illumination component is regarded as the low-frequency component, it can be replaced by a locally constant value in the specified region [19]. In addition, the reflectance component represents the ratio of the reflected light from an object in the range of 0≤f R ≤1. Therefore, the constraint term \(\left \| f_{L}-g_{b} \right \|^{2}_{2}\) in (3) can be derived by taking the maximum operation on both sides of (1) as
Since the maximum value of the reflectance component is 1, it can be expressed as
where \(\tilde {f_{L}}\) represents the illumination component of the image patch. The constraint term \(\left \| f_{L}-g_{b} \right \|^{2}_{2}\) implies that the illumination component of an image is close to the bright channel to prevent the over-enhanced result.
2.2 Edge-preserving noise reduction using total variation
The degradation model of noisy image is generally defined as
where g and f respectively represent the noisy and noise-free images and η is the additive white Gaussian noise.
In order to perform edge-preserving denoising, Rudin et al. minimized the magnitude of the gradient of an image using l 1-norm minimization [20]. The total variation-based noise reduction is defined as
where \(\left \| f-g \right \|^{2}_{2}\) represents the data-fidelity term, ∥∇f∥1 the total variation constraint term on the smoothness of the resulting image, and λ represents the regularization parameter. The energy function in (8) can be solved using the Euler-Lagrange equation as
The edges are preserved according to the magnitude of gradient of f. In the edge region, since the magnitude becomes lager than the flat region, the strong edges in the input noisy image are preserved in the resulting image.
3 Low-light image restoration method using the bright channel prior
In this paper, we present a variational retinex model using l 1- and l 2-norm minimization to enhance a low-light image. The reflectance component consists of rapidly changing high-frequency components such as edge and noise. For this reason, the proposed method estimates the illumination and reflectance components using l 2- and l 1-norm minimization, respectively, to suppress noise amplification while preserving the edge. Next, in order to prevent over-enhancement of the reflectance component, brightness of the estimated illumination component is corrected using histogram equalization and the sigmoid function.
The proposed image enhancement method estimates the contrast enhanced image by minimizing the regularized retinex model as
where λ 1, λ 2, and λ 3 represent the regularization parameters, \(\left \| f_{R}f_{L}-g \right \|^{2}_{2}\) the data-fidelity term, \(\left \| \nabla f_{L} \right \|^{2}_{2}\) and ∥∇f R ∥1, respectively, the smoothness constraints on the illumination and reflectance, and \(\left \| f_{L}-\hat {g}_{b} \right \|^{2}_{2}\) the data fidelity term between the illumination component and the bright channel.
3.1 Pixel-wise bright channel estimation
Existing Retinex-based methods estimate the illumination component using a Gaussian low-pass filter to extract the spatially smoothness component [15]. However, the estimated illumination component does not match the human visual system (HVS) near an edge since the estimated illumination component is continuous near the edge. The incorrectly estimated illumination component results in the halo effect.
In order to apply the discontinuity to the illumination component, Fu et al. proposed the bright channel which can suppress the halo effect [19]. However, this method cannot generate the optimal illumination component because of the blocking artifact caused by the patch-wise bright channel.
To solve this problem, the proposed enhancement algorithm estimates the bright channel at each pixel as
The proposed method suppresses noise by estimating the optimal bright channel using the bilateral filter [21] as
where
Ω represents the image patch whose center is located at x, x i is the ith pixel in the patch, G r and G s respectively are the range and spatial filters, and \(\hat {g}_{b}\) is the bright channel. The bilateral filtering process can effectively reduce the noise while preserving the edge. The enhanced bright channel enables to estimate the optimal illumination component as a constraint of the regularized minimization in (10).
Figure 1 shows the comparison of the bright channel used in Fu’s method [19] and the proposed method. As shown in Fig. 1 b, the patch-wise bright channel shows the halo artifact near the edge region whereas the proposed method can reduce halo effect using the pixel-wise bright channel, as shown in Fig. 1 e.
Comparative results using different bright channels: a input image, b patch-wise bright channel, c proposed bright channel, d resulting image using b and Fu’s method [19], and e resulting image using c and the proposed method
3.2 Optimal reflectance and illumination components estimation
To obtain the enhanced image, the proposed method first initializes the illumination component using a Gaussian low-pass filter. Next, the illumination and reflectance components are separated using variational minimization [19]. Specifically, the energy function related to f R is defined as
Existing Retinex-based variational optimization methods using l 2-norm minimization can reduce the noise with a large regularization parameter at the cost of blurred edge [19]. In order to perform edge-preserving denoising, the proposed method estimates the reflectance component, which contains high-frequency components, using l 1-norm minimization. The energy function in (15) can be solved using Euler-Lagrange equation as
The solution for f R in (15) can be solved using the gradient descent method as
where τ represents the parameter that controls the converge speed.
At each iteration, \(f_{R}^{k}\) is forced to be in the range [0,1].
Given the estimated f R , the illumination component is estimated by minimizing the energy function related to f L defined as
Since the energy function in (18) is quadratic and convex, its optimality condition is obtained by solving linear equation as
An efficient method to solve (19) is to use the fast Fourier transform (FFT) as
where F and F −1 respectively represent the forward and backward FFT operators. Since \(f_{R}^{k}\) is forced to be in the range [0,1], \(f_{L}^{k}\) is forced to be larger value than g at each iteration [19].
The estimated reflectance component cannot avoid the brightness over-enhancement. To solve this problem, the sigmoid function and locally adaptive histogram equalization is used to enhance the contrast of the estimated illumination component [16 , 22]. Finally, the resulting image is reconstructed by multiplying the enhanced illumination and estimated reflectance as
where f R represents the estimated reflectance component by (15) and \(f_{L}^{\text {corr}}\) the enhanced illumination component by (21). Figure 2 shows a step-by-step result of the proposed enhancement method.
4 Experimental results and discussion
In this section, to evaluate the performance of the proposed low-light enhancement method, the resulting image is compared with those of histogram-based [2,3], transmission map-based [8], variational optimization-based [23], and Retinex-based methods [14,16]. The regularization parameters λ 1, λ 2, and λ 3 are determined to have the visually best enhancement result. The objective comparison of image enhancement performance is evaluated using the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) [24]. The simulated low-light image were generated by decreasing 70% of the brightness and adding Gaussian noise of various standard deviations, such as σ=5,10,15, and 20.
4.1 Analysis on the regularization effect
In this subsection, the effect of regularization constraints is analyzed using a simulated low-light image with σ=5. Figure 3 shows the results of proposed method using various different regularization parameters to analyze the effect of each regularization constraint. Figure 3 a, b shows the input and simulated low-light images, respectively. Figure 3 c shows the experimentally best enhancement result.
Experimental results using various sets of regularization parameters: a input image; b simulated low-light image; c resulting image with λ 1=300, λ 2=0.1, λ 3=0.9, and ω=10; d resulting image set with different λ 1; e resulting image set with different λ 2; and f resulting image set with different λ 3
Figure 3 d shows a set of results with λ 2=0.1 and λ 3=0.9 and the three different values of λ 1, 1, 300, and 2000 from top to bottom. Since λ 1 is a parameter related to the smoothness of the illumination component, the low- and high-frequency components are not sufficiently separated with a small λ 1. Therefore, both the illumination component and the resulting image contain noise. Figure 3 e shows a set of results with λ 1=300, λ 3=0.9, and three different values of λ 2, 0.01, 0.1, and 0.2 from top to bottom. Since λ 2 is a parameter related to the smoothness of reflectance component, noise amplification cannot be suppressed with a small λ 2. However, when this parameter is set to too high, blurring artifacts occur. Figure 3 f shows a set of results with λ 1=300, λ 2=0.1, and three different values of λ 3, 0.05, 0.9, and 10 from top to bottom. Since λ 3 is the parameter related to the data fidelity term which controls the brightness of the illumination component, noise amplification and color distortion are unavoidable with a small λ 3. Based on the observation, the optimal parameters are experimentally determined to produce satisfactory result.
4.2 Objective performance evaluation using simulated low-light images
As shown in Fig. 4, six test images were used to compare the enhancement performance of the proposed method with conventional methods.
Figure 5 shows the enhanced results of simulated low-light images using the proposed and conventional methods. The objective comparison of image enhancement performance is performed after modifying the average intensity value of each resulting image to the average value of Fig. 5 a. PSNR and SSIM values are summarized in Table 1.
Figure 5 a, b shows the ideal image and simulated low-light image with σ=5, respectively. Figure 5 c, d shows the results of histogram-based methods with saturation in the bright region. Figure 5 e shows the result of transmission map-based method that produces less saturation than the histogram-based methods. However, this method cannot avoid color distortion. Ravi et al. proposed variational optimization-based method using l 1-norm minimization that can suppress noise while preserving the sharp edge [23]. However, as shown in Fig. 5 f, it loses the dynamic range because of the brightness constraint term using gamma correction. Figure 5 g, h shows the result of Retinex-based methods with undesired artifacts. Although the Retinex-based variational optimization method provides a better enhanced result, it cannot suppress noise amplification. On the other hand, the result of proposed method, as shown in Fig. 5 i, shows the significantly enhanced result with minimum color distortion, saturation, and noise amplification than conventional enhancement methods.
Figures 6, 7, and 8 show the enhanced results of simulated low-light images with different levels of noise using existing and the proposed methods. The PSNR and SSIM values are summarized in Tables 2, 3, and 4. As shown in Figs. 6, 7, and 8, the proposed method provided better results than other existing methods in the sense of bright enhancement without noise amplification. In addition, the proposed method provided improved objective quality assessments at higher standard deviation.
Comparative results using simulated low-light image with σ=10: a input image, b simulated low-light image, c Chen’s method [2], d Kim’s method [3], e Jiang’s method [8], f Ravi’s method [23], g Jobson’s method [14], h Fu’s method [16], and i the proposed method (λ 1=300, λ 2=0.15, λ 3=0.9, and ω=10)
4.3 Subjective evaluation using real low-light images
Figures 9 and 10 show the enhanced results of real low-light images using conventional and the proposed methods. The histogram-based method exhibits brightness saturation and noise amplification in the resulting image. The transmission map-based method produces the enhanced result with noise amplification and color distortion. Although Ravi’s method can successfully reduce the noise, it loses the dynamic range because of the brightness constraint term using gamma correction. Since Retinex-based methods estimate the reflectance component using the incorrectly estimated illumination component, they cannot avoid both halo effect and noise amplification. On the other hand, the proposed method can produce naturally enhanced result with a sufficiently wide dynamic range while reducing the halo effect and noise amplification. However, since the proposed method estimated the solution f R using the gradient descent method, the processing time is longer than existing methods.
5 Conclusions
In this paper, a bright channel prior (BCP)-based variational Retinex model is presented to enhance the low-light image restoration. The existing low-light image enhancement methods cannot avoid undesired artifacts such as noise amplification and halo effect. To solve this problem, the proposed method simultaneously estimates the optimal illumination and reflectance components by minimizing the Retinex-based regularized energy functional to suppress noise amplification during brightness enhancement process. The constraint term related to the smoothness of reflectance component suppresses the noise while preserving the edge using l 1-norm minimization. In addition, the data-fidelity term on the illumination component prevents the halo effect near the edge. Experimental results show that the proposed method can provide better enhanced result than conventional low-light enhancement methods in the sense of both better brightness enhancement and less undesired artifact.
References
S Jang, I Yoon, D Kim, J Paik, Image processing-based validation of unrecognizable numbers in severely distorted license plate images. IEIE Trans. Smart Process. Comput. 1(1), 17–26 (2012).
S-D Chen, AR Ramli, Contrast enhancement using recursive mean separate histogram equalization for scalable brightness preservation. IEEE Trans. Consum. Electron. 49(4), 1301–1309 (2003).
T Kim, J Paik, Adaptive contrast enhancement using gain-controllable clipped histogram equalization. IEEE Trans. Consum. Electron. 54(4), 1803–1810 (2008).
S-H Lee, D Zhang, S-J Ko, Image contrast enhancement based on a multi-cue histogram. IEIE Trans. Smart Process. Comput. 4(5), 349–353 (2015).
J Im, J Paik, Spatially adaptive histogram equalization for single image-based ghost-free high dynamic range imaging. J. Arts. Imaging Sci. 1(1), 55–59 (2014).
H Yang, J Park, Y Moon, Flickering effect reduction based on the modified transformation function for video contrast enhancement. IEIE Trans. Smart Process. Comput. 3(6), 358–365 (2014).
E Lee, W Kang, S Kim, Color enhancement of low exposure images using histogram specification and its application to color shift model-based refocusing. IEIE Trans. Smart Process. Comput. 1(1), 8–16 (2012).
X Jiang, H Yao, S Zhang, X Lu, W Zeng, in Proc. IEEE Int. Conf. on Image Processing. Night video enhancement using improved dark channel prior., (2013), pp. 553–557. http://ieeexplore.ieee.org/document/6738114/.
K He, J Sun, X Tang, Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011).
EH Land, JJ McCann, Lightness and retinex theory. J. Opt. Soc. Am. 61(1), 1–11 (1971).
E Provenzi, D Marini, LD Carli, A Rizzi, Mathematical definition and analysis of the retinex algorithm. J. Opt. Soc. Am. A. 22(12), 2613–2621 (2005).
DJ Jobson, Z Rahman, GA Woodell, Properties and performance of the center/surround retinex. IEEE Trans. Image Process. 6(3), 451–462 (1997).
Z Rahman, DJ Jobson, GA Woodell, Multi-scale retinex for color image enhancement. Proc. IEEE Int. Conf. Image Process. 3:, 1003–1006 (1996).
DJ Jobson, Z Rahman, A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Process. 6(7), 965–976 (1997).
R Kimmel, M Elad, D Shaked, R Keshet, I Sobel, A variational framework for retinex. Int. J. Comput. Vis. 52(1), 7–23 (2003).
X Fu, Y Sun, M LiWang, Y Huang, XP Zhang, X Ding, in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP). A novel retinex based approach for image enhancement with illumination adjustment, (2014), pp. 1190–1194. http://ieeexplore.ieee.org/document/6853785/?reload=true&arnumber=6853785.
C Li, S Gao, D Bi, A modified image enhancement algorithm based on color constancy. Chin. Opt. Lett. 7(9), 784–787 (2009).
W Ma, JM Morel, S Osher, A Chien, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition. An l1-based variational model for retinex theory and its application to medical images, (2011), pp. 153–160. http://ieeexplore.ieee.org/abstract/document/5995422/.
X Fu, D Zeng, Y Huang, X Ding, X-P Zhang, in Proc. IEEE Global Conf. on Signal and Information Processing. A variational framework for single low light image enhancement using bright channel prior, (2013), pp. 1085–1088. http://ieeexplore.ieee.org/document/6737082/.
L Rudin, S Osher, E Fatemi, Nonlinear total variation based noise removal algorithms. Phys. D. 60:, 259–268 (1992).
C Tomasi, R Manduchi, in Proc. IEEE Int. Conf. on Computer Vision. Bilateral filtering for gray and color images, (1998), pp. 836–846. http://ieeexplore.ieee.org/document/710815/.
SM Pizer, RE Johnston, JP Eriksen, BC Yankaskas, KE Muller, in Proc. the First Conference on Visualization in Biomedical Computing. Contrast-limited adaptive histogram equalization: speed and effectiveness, (1990), pp. 337–345. http://ieeexplore.ieee.org/document/109340/.
H Ravi, AV Subramanyam, S Emmanuel, ACE–an effective anti-forensic contrast enhancement technique. IEEE Trans. Signal Process. Lett. 23(2), 212–216 (2016).
Z Wang, A Bovik, H Sheikh, E Simoncelli, Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004).
Funding
This research was supported by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korean government (MSIP) (B0101-15-0525, Development of global multi-target tracking and event prediction techniques based on real-time large-scale video analysis) and Korea Aerospace Research Institute.
Authors’ contributions
SP initiated the research and designed the experiments. BM, SK, and SY performed the experiments and analyzed the data. JP wrote the paper. All authors read and approved the final manuscript.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Park, S., Moon, B., Ko, S. et al. Low-light image restoration using bright channel prior-based variational Retinex model. J Image Video Proc. 2017, 44 (2017). https://doi.org/10.1186/s13640-017-0192-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13640-017-0192-3