Keywords

1 Introduction

1.1 Motivation and Research Objectives

During the past decades, automatic face recognition has received widespread attention from research communities. However, very large intra-subject variations such as facial expressions, occlusion [9], aging, and outdoor illumination make the task of recognition more challenging, especially in real-world applications [1, 29]. Usually, face acquisition is realized by an outdoor camera, where the subjects may not be cooperative to match with a face database. Furthermore, images are so difficult and unclear since they are acquired under unpredictable varying lighting conditions. For this reasons, subject recognition involves a pre-processing stage which is commonly known as face image enhancement. This step aims to improve the face image quality in such way to bring out the occluded face characteristic. Particularly, finding a good trade-off between brightness and contrast is important for enhancing face images [19]. Despite the large variety of the existing face enhancement approaches in literature, yet, face images in conjunction with illumination variations still suffer from unclarity [12]. In order to compensate the illumination, Fuzzy [21], CLAHE [27] and multi-resolution pyramid [20] approaches have been harnessed.

1.2 Literature Review

During the last decade, the link between image prepocessing and pattern recognition area has been at the center of much attention from scholarly literature [15]. There are a large number of published methods (e.g., Iratni et al. [14]; Oulefki et al. [19, 21]) where they involved image analysis stage for the seek of face image enhancement. Moreover, enhancement strategies might include spatial [26, 30] and frequency domain [6, 13] to improve images. For example, Du et al. [6] presented discrete wavelets transform (DWT) in which they applied HE to the low frequency and accentuate the high frequency coefficients. Jobson et al. [16] improved the multiscale retinex (MSR) method which is the (HE) Histogram Equalization extension of the previous single-scale center/surround retinex. This last cancels lots of the low frequency information by subdividing the given image to a smoothed version of itself. Due to its fast implementation [36]. HE is one of the basic method for improving and adjusting the contrast of images. Despite its accuracy, it remains an uneven particularly when it processed under varying lighting conditions.

Pizer et al. [24] discussed and summarized Contrast-Limited Adaptive Histogram Equalization (CLAHE). The standard CLAHE computes distinct histograms blocks corresponding to a divided section of the given image then readjust the image’s brightness of each part. However, CLAHE has a tendency to over-amplify/enhance images which led to loss of information in some local region.

Generally, CLAHE is lacking the nice property of balancing overall patterns and image sharpness as stated in [32]. To overcome the limitations of CLAHE, many variants have been proposed in literature [4]. Toward the end of improving CLAHE performance for face enhancements under low light constraints, we propose in this paper a hybrid CLAHE-FUZZY enhancement pipeline. This method will produce optimal contrast without losing any local information of the face image which is most important for recognition and detection human faces. The proposed method consists of two stages of processing to increase the potentiality homogeneity regions of an image and to preserve the local details in the images. The details of the proposed method are presented in the next section.

1.3 Contributions

In this paper, an innovative spatial-frequency domain enhancement named Fuzzy Partial Enhancement Approach (FPEA) has been proposed seeking the correction of face images illumination in a balanced way. The FPEA compensation is based on the intensity of too dark, or too bright sides of face images. Firstly, the input face image is split into two areas of face images right and left sides. Then, the corresponding intensity of dark, normal, or bright pixels in the gray-scale channel of each part are calculated. Afterward, FPEA correction is applied to compensate the illumination of the affected part only (low or high intensity). Meanwhile, laplacian pyramid decomposition is used to reconstruct the facial image, in such way to smooth the line that separates the two face parts, thus blending the local area around the separation [20]. This latter approach provides the ability to deal with separate frequency while locally preserving the face information. The compensation in the FPEA method is adaptively applied based on the face affected part. Moreover, a region-based FPEA method that entails applying the FPEA algorithm in two sub-regions of the image reduces the computing time. In addition, the experimental results based on the four most widely used face databases proved the efficiency of the FPEA.

The remainder of this paper is organized as follows: Sect. 1 reviews the state-of-the-art face image enhancement. In Sect. 2, the proposed method is discussed in detail, followed by Sect. 3 which discusses in depth the quality assessment and database used in this paper, and Sect. 4, Sect. 5 presents the experimental results of quantitative and visual measurements respectively along with a discussion. Finally, Sect. 6 presents the conclusion by interpreting overall results.

2 The Proposed Enhancement Framework

Imperfections such as over/less enhancements are one of the drawbacks of using the conventional image enhancement methods. Moreover, artifacts effect could be generated also when applied to given images. Thus, the proposed approach aims to cut down the aforesaid issues is proposed. On one side, it compensates both brightness and the contrast of the input image, without ignoring hidden details that will appear after using our approach. Fuzzy-Reasoning Model (FRM) [21] along with Contrast Limited Adaptive Histogram Equalization (CLAHE) [24] to compromise in a more precise manner the problem of illumination variation on face images. Firstly, we apply Viola−Jones Algorithm for detecting faces. The cropped input face image (I) is then divided into two equal parts labeled (A) and (B). This two face parts denote the left and the right part, respectively (See Fig. 1). At that point, we calculate the intensity from each part including the input cropped face image (I).

Fig. 1.
figure 1

The pipeline of the proposed PFEA for face illumination enhancements

Full size image

After that, we perform enhancement first by using CLAHE. Following CLAHE stage, we carry out FRM operator as an exponentiation to produce a factor in such a way that adjust illumination at the part of face that has been affected by the lack of luminance.

According to FRM model, FRM generates an adaptive enhancement which corrects and improves non-uniform illumination and low contrasts of an image by comprising luminance component as given in Eq. 1:

$$\begin{aligned} FRM=I^{exp \left( \overline{\alpha }_2-\overline{\alpha }_2\right) } \end{aligned}$$
(1)

where I is the input image, and \(\overline{\alpha }_0\) represent the mean intensity of I. \(\overline{\alpha }_f\) is the resulting value from the Fuzzy system. The FRM rules used and its description be found in [21].

Prior to this, in order to detect face pixels that are affected by the lack of illumination. The intensity varies as a function of the relative direction to the illumination at the acquisition stage. The regions that could be enhanced depend on the poor intensity that are presented. First, we normalized histogram to calculate the changes of intensity of input image, left side and right as represented in Fig. 1 by the triplet [I, A, B].

After studious testing we have found that intensity belonging to face region with a poor illumination are located when A = [0.6, 1] and B = [0.0, 0.3] for the right side and vice versa for the left one.

Finally, in order to have smooth blending, multi-resolution pyramid approach [3] applied on the none affected area in a given band that should be recombined to obtain the final face image enhanced mosaic (see Fig. 2). In other words, for a greater realism of the enhanced image the multi-resolution pyramid approach is carried out only for the part that is not affected by lack of illumination for making a realistic face looking. This is proven by the results of visual assessments. The aim is to seamlessly stitch together both right and left images into a face image mosaic by smoothing the boundary in a scale-dependent way to avoid boundary artifacts.

Fig. 2.
figure 2

The spline applied to a sample face image from CMU-PIE database. When the left half of (c) is joined to the right half of face (d) without a spline, the boundary is clearly visible (e). However, by applying our approach. No boundary is visible when the multi-resolution spline is used (f)

Full size image

3 Quality Assessment and Databases Used

We have chosen four performance metrics for this paper. No-reference Image Quality Assessments (IQAs) tested in this paper are summarized in Table 1.

Table 1. Image Quality Assessments (IQAs) used in the present work.
Full size table

For the IQA without mathematical description (NIQE), the exact details of its implementation and more details can be found in the giving reference [18]. The data-sets used for this project consisted of CMU-PIE, E-Yale-B, CAS-PEAL and Mobio. These face databases were selected to represent a typical implementation of our approach along with specification of images included faces affected by illumination invariant.

4 Experiments

In this section, the performance of our technique (PFEA) compared with the state-of-the-art image improvement approaches such us HE [2, 25], BPDFHE [34], NMHE [26], LIME [5, 11], FRM [21], and AdaptGC [14] using CMU-PIE [31], mobio [17], CAS-PEAL [8] and E-Yale-B [10] databases. We use four IQAs to compare the performance of all these approaches including EME, SDME, CPP, and NIQE to indicate good performance in terms of correlation with human visual assessment [28, 37]. To display each quantitative metric in one single figure. We selected a bar plot to present the comparisons by calculating the means of EME, SDME, CPP and NIQE of the proposed and the stat-of-the-art of all face images from EYale-B to CMU-PIE, through CAS-PEAL and Mobio data set. While the color symbols blue, red, orange, and white, having been selected to represent E-Yale-B, CMU-PIE, Mobio and CAS-PEAL data-sets respectively. Moreover we include the information of Confident Interval (CI) (95%) [33] to refer to the level C of a confidence which provides the probability that the range CI obtained by the approaches employed includes the true mean value of the IQAs used over the entire database.

4.1 IQAs Results and Discussion

As can be seen in the Fig. 3, the means of EME of the improved face images using the proposed PFEA is greater than the five enhancement approaches picked for comparison, this means that the best enhancement effect is achieved by the developed approach. Meanwhile, NMHE provides the least indicator enhancement followed by PDFHE and HE, whereas AdaptGC maintaining competitive.

Fig. 3.
figure 3

Means of EME, and CI \((95\%)\) computed from PFEA, HE, BPDFHE, NMHE, LIME and AdaptGC by taking original images’ EME as reference of respectively EYale-B, CMU-PIE, mobio and CAS-PEAL face Data bases

Full size image

In order to assess the effects of the enhancement quality using SDME metric, the experimental results on Fig. 4 shows that the mean of HE, BPDFHE, NMHE are lower than those of the other methods. The provided results when performing PFEA methods along with the other enhancement methods assessed in this paper, proved that the PFEA produce a best result.

Fig. 4.
figure 4

Means of SDME, and CI \((95\%)\) computed from PFEA, HE, BPDFHE, NMHE, LIME and AdaptGC by taking original images’ SDME as reference of respectively EYale-B, CMU-PIE, mobio and CAS-PEAL face Data bases

Full size image

In terms of CPP metric where it is plotted in Fig. 5, we can say that across the different quality measurement the results are just about the same of all datasets, with a slight superiority of the PFEA.

Fig. 5.
figure 5

Means of CPP, and CI \((95\%)\) computed from PFEA, HE, BPDFHE, NMHE, LIME and AdaptGC by taking original images’ CPP as reference of respectively EYale-B, CMU-PIE, mobio and CAS-PEAL face Data bases

Full size image

The summary statistics of NIQE metrics which by definition a smaller score indicates better perceptual quality. In Fig. 6, it is obvious that across the different enhancements used the PFEA provide the best results as the mean NIQE are just about the same of all datasets.

Fig. 6.
figure 6

Means of NIQE, and CI \((95\%)\) computed from PFEA, HE, BPDFHE, NMHE, LIME and AdaptGC by taking original images’ NIQE as reference of respectively EYale-B, CMU-PIE, mobio and CAS-PEAL face Data bases

Full size image

5 Qualitative Results and Discussion

Samples of low-light face images before and after enhancement results are shown in Fig. 7. We picked up disparate low-light face images from already stated databases of E-Yale-B, CMU-PIE, mobio, and CAS-PEAL. On this spot, we compare the proposed algorithm PFEA with HE, BPDFHE, NMHE, LIME, and AdaptGC. For comparison approaches, we applied firstly Viola and Jones [35] to crop the face region. As we carried out both spacial and frequency tone to enhance both the dark and over light regions of face image, the visual comparisons demonstrate that the proposed PFEA can effectively face enhance images which differ from the darkest to the lightest ones even the lowest or highest-dark/light whatever the region affected. For example, the results of AdaptGC includes some saturated regions and the noises also come out. Whereas, face image improvement using HE, BPDFHE and NMHE does not add some sharpness and fix the poor blurriness. While the results of the proposed PFEA as shown in the third column from the left to right can enhance the images without such saturated region also removes some unnecessary disturbances by smoothing the face images. Another advantage of applying PFEA is the ability to extract significant facial features (eye contours, eyebrows, lip contours, and nose tip) also producing balanced resulting visually.

Fig. 7.
figure 7

Visual comparisons using PFEA, HE, BPDFHE, NMHE, LIME and AdaptGC test images: (from top to bottom) low-light face images from E-YaleB, CMU-PIE, CAS-PEAL, and mobio

Full size image

6 Conclusion

In this work, we proposed an efficient method for face enhancement. Both, the key to the low-light/hard-dark image enhancement and the affected part of the face map are estimated. The proposed PFEA uses fuzzy logic to handle the inexactness of face images in a better way compared to stat of the art techniques, resulting in higher quality performance. It can also improve the affected face image parts, regarding both the brightness and darkest sides. It also automatically takes into account the enhancement of the image contrast. The proposed algorithm was tested on different face images from EYale-B, CMU-PIE, mobio, and CAS-PEAL databases. The performance of proposed PFEA algorithm was evaluated and compared in terms of EME, SDME, CPP and NIQE metrics. Furthermore, PFEA was compared to state-of-the-art image enhancements methods using both qualitative and quantitative performance metrics. Experiments demonstrate that the proposed method significantly eliminate the washed-out appearance and adverse artifacts induced by several existing methods. This method is simple and suitable for consumer electronic products. We are investigating the proposed method for improving face recognition performances, where the preliminary results are promising.