Robust 2D Face Recognition Under Different Illuminations Using Binarized Partial Face Features: Towards Protecting ID Documents

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8915)

Abstract

Biometric recognition techniques have been widely employed in numerous applications, such as access control, identity check etc. In this paper we propose a method for protecting personal identity documents against forgery by using 2D face image on the ID document. The main components of this method includes detection of the face image from the document, extracting features from partial face images and converting the extracted features to binary feature vectors using Local Gradient Increasing Pattern (LGIP) approach. The binary feature vectors are concatenated to form a binary template which can be encoded and stored on the ID document in the form of a 2D bar code. This 2D bar code will be used to authenticate the ownership of the ID document. The face recognition method is evaluated using FRGC and FERET databases. The results show that this method can efficiently authenticate an ID document on the basis of the face image and the method can also be used to retrieve a subject from a database of images. The method is proved to be robust even if an ID document is scanned under different illuminations conditions. A report on document authentication where the face image is damaged is also presented.

Keywords

Biometrics Face recognition Document forgery Verification and identification 

1 Introduction

Biometric recognition techniques have been widely employed in numerous applications, such as access control, identity check etc. One of these applications is protecting the personal ID documents against forgery. In order to achieve a secure ID document, a Machine Readable Zone (MRZ) is commonly used, which basically protects against the manipulation of the alphanumeric data in an ID document. However, MRZ has not been used so far to protect the photo identity in an ID document. Another method is the use of electronic identity documents where the owner information is stored in an RFID chip. Although this method provides an enhanced security level, but it bears some disadvantages like unauthorized access on the wireless interface and higher costs. The limitations of previous methods motivate the use of biometric based techniques like face recognition to protect documents against falsification. However, with use of face recognition, challenges like sensor noise, illumination and quality of face image, size of the generated template needs to be properly addressed at ID document issuance and verification scenarios.

In this paper we propose a method for protecting personal identity documents against forgery by storing biometric face information in a 2D bar code on the document. This method shows robustness against illumination, white noise and generates a small binary feature vector that can be encoded in a 2D bar code. In addition to the biometric template, one may also store alphanumeric data in the 2D bar code, in order to protect the ID document against photo and alphanumeric data manipulation.

2 Proposed Approach

The proposed approach can be described in 5 steps: (i) face detection from ID document, (ii) partitioning of detected face image into vertical and horizontal sub-images (also referred as partial face images) and feature extraction from these partial face images, (iii) conversion of these extracted features to binary feature vectors using LGIP, (iv) concatenation of these binary feature vectors to a consolidated template and (v) storing this template as a 2D bar code on the document.

These steps are explained in the following sections:

2.1 Face Detection from Document

An image is detected from a document as shown in Fig. 1, where the region of interest is the face with forehead, hair and chin as shown in Fig. 1. The detection procedure comprises of the following four steps,
Fig. 1.

(a) Original image (b) Detected face in 56 × 46 pixels gray scale image

  1. 1.

    Conversion of the image to grayscale

     
  2. 2.

    Applying histogram equalization on the converted image

     
  3. 3.

    Detection of the face using the Cascade Classifier method [2]

     
  4. 4.

    Resizing of the image into 56 × 46 pixels (this step is performed in order to have equal size partial images in Sect. 2.2)

     

2.2 Partial Face Features Extraction

This step extracts the face features from different portions of the face using projected random matrices as described in [3]. Initially, the face \( X \epsilon {R^{q X p}} \) (where q is height and p is width in pixels of face shown in Fig. 1), is divided into different portions or sub-images in horizontal and vertical directions. The extraction of the features is carried on all horizontal and vertical sub-images. The feature extractions of each two dimensional sub-image matrix is done by using two projection matrices \( R_{1} \), \( R_{2} \), where \( R_{1} \) is used in the feature extraction of the vertical face sub-images and \( R_{2} \) is used in the feature extraction of horizontal face sub-images, respectively.

\( {\text{R}}_{1} \)\( \epsilon{\text{R}}^{{{\text{k}} \times {\text{p}}}} \) is a two dimensional matrix consisting of a sequence of the binary values “0” and “1”, where k is an arbitrary number representing the height of the projection matrices and p is width of the face image. Based on a previous work [3], the height of the projection matrices (k) in this work is set equal to 5. The number of ones in each row is arbitrary. We assumed a series of 15 ones for each row [3]. This matrix \( R_{1} \) is used to extract vertical features from the sub-image \( {\text{X}} \) with the same definition for \( {\text{R}}_{2} \)\( \epsilon {\text{R}}^{{{\text{q}} \times {\text{k}}}} \), which will be post-multiplied with the sub-image \( {\text{X}} \), to get horizontal features. Figure 2 illustrates the projection matrices \( R_{1} \) and \( R_{2} \). Binary projection matrices will induce the effect of the data compression as it accumulates some selective pixels values (pixels at those positions where projection matrix bit is set to 1) and not all pixels. The cumulated value of all pixels will be the partial face (or sub-image) feature matrix, from which a binary feature vector is generated using LGIP approach as defined in Sect. 2.3.
Fig. 2.

Projection matrices R1, R2 (R2 is transpose of R1) [3]

To extract the partial face features, the original image is divided into n vertical and m horizontal partial face images, as shown in Fig. 3. In this work, n and m is set equal to 2 and 8, respectively. To extract features from vertical and horizontal sub-images, projection matrix \( {\text{R}}_{1} \) is multiplied with vertical sub-images \( {\text{X}}_{\text{i}} \), \( {\text{i}} = 1, \ldots ,{\text{n}} \) and projection matrix R2 is multiplied with horizontal sub-images \( {\text{X}}_{\text{i}} \), \( {\text{i}} = 1, \ldots ,{\text{m}} \). The resulting vertical and horizontal sub-images products \( {\text{R}}_{1} . {\text{X}}_{\text{i}} \in R^{{{\text{k}} \times {\text{q}}/{\text{n}}}} \) and \( {\text{X}}_{\text{i}} .{\text{R}}_{2} \in R^{{{\text{p}}/{\text{m }} \times {\text{k}}}} \) are shown in Fig. 3.
Fig. 3.

Partial face features extraction [3], where k = 5, n = 2, m = 8

The proposed method can be more robust against identity forgery by using different projection matrix to generate the binary vectors for individual users or documents. In case of document-specific projection matrix, a new ID document can be issued by selecting another projection matrix. It is to be noted that, the projection matrices can be interpreted as a random seed that is used to encode features. The projection matrix itself does not reveal any sort of personal information.

2.3 Local Gradient Increasing Pattern

Local Gradient Increasing Pattern (LGIP) [4, 5] is a pixel-based binary face descriptor that shows robustness against variations in illuminations and white noise. Therefore, it is used to generate the binary vectors for the vertical and horizontal sub-image feature matrices resulted from the partial face division, shown in Fig. 3. LGIP employs Sobel masks [4, 5] \( S_{0} , \ldots ,S_{7} \) to calculate gradient responses in eight different orientations at each pixel. The gradient value resulted from each mask is converted into a single bit (1 or 0), depending on the sign of the gradient value. As a result, an 8-bit code descriptor of each pixel in the partial face image is obtained.

The Sobel gradient masks used in this work are shown in Fig. 4. These eight masks are applied on each pixel. If the pixel response to the mask is positive, a resultant bit is set to be 1, otherwise 0. As a result, an 8 bit value is generated for each pixel, where a single bit is the corresponding result of one mask. Since, the feature matrix is compressed (i.e., 56 × 23 pixels are reduced to 5 × 23 pixels in vertical sub-image and 7 × 46 pixels are reduced to 7 × 5 pixels in horizontal sub-image), it is necessary to take all the pixels in the binary vector generation. Therefore, a padding of 2 pixels is done at the boundaries of each generated feature matrix, so that 5 × 5 Sobel masks can also be applied on the boundaries of the vertical and horizontal sub-image feature matrix.
Fig. 4.

Sobel gradient masks [4, 5]

2.4 Template Generation and Comparison

Each face image is divided into 2 vertical sub-images and 8 horizontal sub-images, the projection matrices R1, R2 are multiplied at fixed positions to cover all image pixels, where R1 ϵ R5 × 56 and R2 ϵ R46 × 5. The number of ones in each row for R1 and each column for R2 is set to 15. For an image, the size of each vertical features and each horizontal feature template is 5 × 23 pixels and 7 × 5 pixels, respectively. The binarization of these templates using LGIP as defined in Sect. 2.3, results in a binary feature vector of a length equal to 5 × 23 (pixels) × 8 (bits/pixel) = 920 bits and 7 × 5 (pixels) × 8 (bits/pixel) = 280 bits, respectively. The final binary feature vector of the image is generated by concatenating all vertical and horizontal binary feature vectors. This results in a total template size of 920 × 2 + 280 × 8 = 4080 bits per image. Figure 5 illustrates the overall workflow of the proposed scheme.
Fig. 5.

Illustration of the overall scheme

The comparison of the extracted binary feature templates from two images can be performed using hamming distance, in two ways. First is to calculate the hamming distance considering the complete binary vector, and this is a good option in case of uniform brightness or white noise. The second choice is to calculate the hamming distance by taking the binary vectors of individual partial face features and then fuse these hamming distances to get a final distance. In this case, the hamming distance generated from the comparison of different parts or partial features are weighted. The weights can be assigned to different hamming distance values depending on the confidence of the distance value. For instance, higher weights should be assigned to the distance values resulting from the comparison of partial face images with higher quality, higher entropy or higher discriminability and vice versa. The different weighted hamming distances can be combined to get a final hamming distance value.

In both options, the value of the hamming distance would depict the measure of closeness of two images whose feature templates are compared. The decision, if both compared templates are generated from the same face or not, can be done by comparing the hamming distance with a threshold. In this work, the comparison is done using the first option (i.e., without weighted distance approach).

2.5 2D Bar Code Generation

The generated binary vector consists of a series of ones and zeros, and by converting the binary vector to hexadecimal characters we reduce the vector size to 25 % of the original vector. For encoding the hexadecimal data, a QR code is used. The capacity of a QR code is up to 7000 digits or 4000 characters, and is highly robust. Zxing [6], a multi-format 1D/2D bar code image processing library, is used for generating the QR code and the Quricol library [7], is used for decoding the generated QR code. An illustration of the bar code generation application is shown in Fig. 6.
Fig. 6.

QR code generation from binary template

It is possible that the 2D bar code also contains document owner personal information (like alphanumeric data [1], handwritten signature features [8, 9]) and other document information (like issuing and expiry dates) to detect a forgery in the ID document. Including personal data in the 2D bar code generation would be particularly useful to detect fake ID in case an attacker tries to replace both the face image and the bar code stored on the ID document simultaneously; thus additional information can be exploited to perform authentication of the ID document. However, it is not recommended to include the projection matrix information in the 2D bar code, as this may lead to function creep or attack on personal information.

3 Experimental Studies

In order to check the feasibility of given approach for ID document and its ownership authentication, three experimental studies were conducted. The first experimental study focuses on examining the effect of brightness during identity (ownership) verification of the subject on the ID document. The study has been carried out by comparing face image present on the ID document under different brightness with the compact 2D bar code feature representation also present on the ID document. The second experimental study focuses on the comparison of the binary feature vector extracted from 2D bar code stored on the ID document against the face image detected from the ID document or an image stored in a database. The result of the comparison is a hamming distance that can be compared with a threshold to make a decision, if the two images compared are matched or not matched. The third experimental study focuses on the identification of a person from a partially damaged face image. The original rank of the person is calculated given different kinds of damaged images. The test data set details and results of each of the experimental studies are explained as follows.

3.1 Robustness to Brightness

This experimental study was conducted on a data set of randomly selected 987 frontal posed images from 987 subjects (i.e., one image per subject). Out of these 987 subjects, 312 subjects were taken from the FRGC database [10] and the remaining 675 subjects were taken from the FERET database [11]. In each frontal image, the face region is detected and converted to a grayscale image of 46 × 56 pixels. Then from each frontal grayscale image, eight other images of different levels of uniform brightness were generated by adding a scalar value of ± 80, ± 60, ± 40 and ± 20 to the 2D grayscale frontal image, respectively. An example of a frontal image (labelled as 0 image) and the generated eight images is shown in Fig. 7. Partial face feature extraction was performed for all of these 987*9 images and the features were binarized using LGIP approach, as defined in Sects. 2.2 and 2.3, respectively.
Fig. 7.

Example of a frontal image (center image: labelled as 0) and the generated 8 images with different brightness levels (scalar value displayed as label under each individual image)

In order to verify the robustness of features against the brightness, hamming distances were generated by comparing each subject (original image marked as intensity zero in Fig. 7) with its other 8 different brightness images (examples shown in Fig. 7) and with all other subjects (imposters) original image and their variants. Figure 8 shows the maximum and minimum distance values obtained for a genuine and imposter comparison, respectively. Analyzing this plot of scores obtained from genuine subject comparisons (intra-class comparisons) and imposter subject comparisons (inter-class comparisons), it can be estimated that the ID authentication system will classify very well between genuine and fake IDs with a threshold for hamming distance equal to around 400 bits. A very low decision threshold (approx. 400 bits) as compared to full feature length (4080 bits) shows effectiveness of the proposed approach.
Fig. 8.

Max. genuine and min. imposter score distributions

3.2 Identification and Verification

The aim of this experimental study is to check the quality of the discriminative feature represented by the binary feature vector, for the purpose of identification and verification of a person. The use case for identification can be a, (i) lost and found scenario, where an owner of a lost ID document needs to be identified, or (ii) duplicate enrolment check scenario, where a person asking for a reissue of another ID document can be detected by comparing his/her face image with face images of other persons already enrolled in the database. The experimental study was conducted on a dataset of 1203 images taken randomly from selected 295 subjects of the FRGC databases. Multiple images were taken from different subjects. The images taken were mainly frontal images with their intrinsic different illuminations and different expressions. Unlike Sect. 3.1; an explicit set of 8 different brightness were not generated in this case. Partial face feature extraction was performed for all of these 1203 images and the features were binarized using LGIP approach, as defined in Sects. 2.2 and 2.3, respectively.

For each image in the dataset we compare its binary feature vector with binary feature vectors extracted from other images in the database. The comparison scores were then ranked on the basis of least hamming distance, as lower distance would indicate high measure of closeness of both the images. The identification process carried was in closed-set identification scenario. i.e., the probe subject always had mate templates in the enrolment database.

To visualize the identification (1:N comparisons) and verification (1:1 comparisons) performances, the CMC (cumulative match curve) and the ROC (receiver operating characteristic) is plotted in Figs. 9 and 10, respectively.
Fig. 9.

CMC plot for selective FRGC data set

Fig. 10.

ROC plot for selective FRGC data set

The CMC plot shows that the proposed method gives 80.3 % rank-1 performance for the given dataset. The ROC plot shows a true match rate of 58 % at false acceptance rate of 10 %. Figure 11 shows an illustration of identification application, where given an input test image, the top 8 closest images (based on hamming distance) were successfully retrieved.
Fig. 11.

Illustration of top ranked images (red highlighted: Input image)

3.3 Comparison with Damaged Images

In this experimental study we show that the proposed features can be used to identify a person from a damaged image in an ID document. To conduct this study, a data set consisting of 449 images from 166 subjects of FERET database was taken. We manually damaged 6 face images by scratching and fading some part of the face in the image. These damaged images are shown in Fig. 12. For testing of identification, the binary templates of 6 damaged images were generated and the rank value of corresponding mate subject was calculated using each damaged image.
Fig. 12.

Illustration of damaged images in an ID document

Table 1 shows the rank value of correct mate subject found when the damaged face image was given for identification of the person in the given data set. The results indicated in Table 1, shows the ability to identify person with damaged document, as the algorithm was able to detect correct image within first 3 ranks. The influence of damaged images (a, b, c and f) on the identification is low because of the local nature of LGIP binarization, as it would limit the adverse impact only to the features on the line that is separating the lighter and the darker parts of the image. In damaged image (e) where part of the face is scratched, this missing portion will have almost the same impact on the hamming distance against all images in the database thus the ranking did not appear to be affected.
Table 1.

Values of rank correctly detected under different damaged images

Damaged image

a

b

c

d

e

f

Rank of correct image

1

3

2

Face not detected

2

1

4 Discussion

It is important to note that this evaluation process should be made as close as possible to a real scenario, where the binary vectors of scanned face image from a document (using a scanner) and its bar code representation needs to be compared. It is evident that in our evaluation process, the used FRGC and FERET databases are not representative for scanned images. But few experiments were also performed using scanned images and the performance was found to be unaffected provided that both binary vectors for the face image and the bar code must be generated from images of same nature. i.e., both the vectors to be compared are generated either from digital images (i.e., image taken directly from the database) or are generated from scanned images. (i.e., image taken from database first printed on a paper using a printer and then scanned to generate the templates). The extraction of horizontal and vertical partial face image provide better performance than using global face images, because partial images helps in extracting local face features which are less sensitive to imaging conditions than those using global appearance based features [3]. Applying LGIP approach after feature extraction binarizes the features and makes the features more robust under different illuminations and noise. Partitioning of face image helps in generating a binary vector much smaller than directly apply LGIP on the complete image at once.

5 Conclusion and Future Work

In this paper, we have presented a 2D face recognition scheme based on extracting features from partial horizontal and vertical face images and feature level fusion (concatenation) of binary form of these features. These features are extracted using projection matrices and the extracted features are binarized using LGIP approach. The experimental results indicate 80.3 % rank-1 identification rate on a data set of FRGC database. The experimental results also showed that partial features are robust under different brightness conditions and are also suitable for recognition of faces even in case of damaged images. Future work will focus on encoding and compression of the generated binary template, so that it is as small as possible and can be easily printed using a much smaller 2D bar code. This will improve the practical usability of the face recognition approach for application of protecting ID documents. In addition, our future plan is to improve recognition performance in uncontrolled scenario (different illuminations with face expressions) by assigning optimum weights to different comparison scores extracted from different partial face binary features.

Notes

Acknowledgement

The work leading to these results has received funding from the European Community’s Framework Programme (FP7/2007-2013) under grant agreement no. 284862

References

  1. 1.
    Zhou, X., Schmucker, M.: Securing of Personal Identity Documents against Forgery (Sichern von personen-identitätsdokumenten gegen Falschung). Patent WO 2009/074342 A1, June 2009Google Scholar
  2. 2.
  3. 3.
    Oh, B.-S., Toh, K.-A., Choi, K., Teoh, A.B.J., Kim, J.: Extraction and fusion of partial face features for cancelable identity verification. Pattern Recogn. 45, 3288–3303 (2012)CrossRefGoogle Scholar
  4. 4.
    Lubing, Z., Han, W.: Local gradient increasing pattern for facial expression recognition. In: 19th IEEE International Conference on Image Processing (ICIP) (2012)Google Scholar
  5. 5.
    Zhou, L.B., Wang, H.: Local gradient increasing pattern (LGIP) for facial representation and gender recognition. In: Campilho, A., Kamel, M. (eds.) ICIAR 2012, Part II. LNCS, vol. 7325, pp. 46–53. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  6. 6.
    Odeh, S., Khalil, M.: Off-line signature verification and recognition: neural network approach. In: International Symposium on Innovations in Intelligent Systems and Applications (INISTA) (2011)Google Scholar
  7. 7.
    Karouni, A., Daya, B., Bahlak, S.: Offline signature recognition using neural networks approach. In: World Conference on Information Technology WCIT 2010, Procedia Computer Science (2011)Google Scholar
  8. 8.
    ZXing (zebra crossing), bar code image processing library. https://github.com/zxing/zxing. Accessed 4 June 2014
  9. 9.
    Quricol 2.0 QR Code Generator. http://delphi32.blogspot.de/2013/09/quricol-20-qr-code-generator.html. Accessed 4 June 2014
  10. 10.
    Phillips, P.J., Flynn, P.J., Scruggs, T., Bowyer, K.W., Worek, W.: Preliminary face recognition grand challenge results. In: 7th International Conference on Automatic Face and Gesture Recognition (2006)Google Scholar
  11. 11.
    Phillips, P.J., Moon, H., Rauss, P.J., Rizvi, S.: The FERET evaluation methodology for face recognition algorithms. IEEE Trans. Pattern 22, 1090–1104 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Competence Center Identification and BiometricsFraunhofer Institute for Computer Graphics Research IGDDarmstadtGermany

Personalised recommendations