Evaluation of Face Datasets as Tools for Assessing the Performance of Face Recognition Methods

  • Lior ShamirEmail author
Short Paper


Face datasets are considered a primary tool for evaluating the efficacy of face recognition methods. Here we show that in many of the commonly used face datasets, face images can be recognized accurately at a rate significantly higher than random even when no face, hair or clothes features appear in the image. The experiments were done by cutting a small background area from each face image, so that each face dataset provided a new image dataset which included only seemingly blank images. Then, an image classification method was used in order to check the classification accuracy. Experimental results show that the classification accuracy ranged between 13.5% (color FERET) to 99% (YaleB). These results indicate that the performance of face recognition methods measured using face image datasets may be biased. Compilable source code used for this experiment is freely available for download via the Internet.


Face recognition Biometrics FERET 


  1. Bishop, C. M. (2006). Pattern recognition and machine learning. New York: Springer. zbMATHGoogle Scholar
  2. Chen, L. F., Liao, H. Y., Lin, J. C., & Han, C. C. (2001). Why recognition in a statistics-based face recognition system should be based on the pure face portion: a probabilistic decision-based proof. Pattern Recognition, 34, 1393–1403. zbMATHCrossRefGoogle Scholar
  3. Fei-Fei, L., Fergus, R., & Perona, P. (2006). One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 594–611. CrossRefGoogle Scholar
  4. Gabor, D. (1946). Theory of communication. Journal of IEEE, 93, 429–457. Google Scholar
  5. Georghiades, A. S., Belhumeur, P. N., & Kriegman, D. J. (2001). From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis Machine Intelligence, 23, 643–660. CrossRefGoogle Scholar
  6. Goldberg, I. G., Allan, C., Burel, J. M., Creager, D., Falconi, A., Hochheiser, H., Johnston, J., Mellen, J., Sorger, P. K., & Swedlow, J. R. (2005). The Open Microscopy Environment (OME) Data Model and XML file: open tools for informatics and quantitative analysis in biological imaging. Genome Biology, 6, R47. CrossRefGoogle Scholar
  7. Gradshtein, I., & Ryzhik, I. (1994). Table of integrals, series and products (5th edn., p. 1054). New York: Academic Press. Google Scholar
  8. Gray, S. B. (1978). Local properties of binary images in two dimensions. IEEE Transactions on Computers, 20, 551–561. CrossRefGoogle Scholar
  9. Gregorescu, C., Petkov, N., & Kruizinga, P. (2002). Comparison of texture features based on Gabor filters. IEEE Transactions on Image Processing, 11, 1160–1167. CrossRefMathSciNetGoogle Scholar
  10. Gross, R., Baker, S., Matthews, I., & Kanade, T. (2004). Face recognition across pose and illumination. In S. Z. Lin & A. K. Jain (Eds.), Handbook of face recognition. Berlin: Springer. Google Scholar
  11. Gurevich, I. B., & Koryabkina, I. V. (2006). Comparative analysis and classification of features for image models. Pattern Recognition and Image Analysis, 16, 265–297. CrossRefGoogle Scholar
  12. Hadjidementriou, E., Grossberg, M., & Nayar, S. (2001). Spatial information in multiresolution histograms. In IEEE conf. on computer vision and pattern recognition (Vol. 1, p. 702). Google Scholar
  13. Haralick, R. M., Shanmugam, K., & Dimstein, I. (1973). Textural features for image classification. IEEE Transaction on System, Man and Cybernetics, 6, 269–285. Google Scholar
  14. Jain, V., & Mukherjee, A. (2002).
  15. Kong, S. G., Heo, J., Abidi, B. R., Paik, J., & Abidi, M. A. (2005). Recent advances in visual and infrared face recognition: a review. Computer Vision and Image Understanding, 97, 103–135. CrossRefGoogle Scholar
  16. Lim, J. S. (1990). Two-dimensional signal and image processing. In Signals, systems, and the Fourier transform (pp. 42–45). Englewood Cliffs: Prentice Hall. Google Scholar
  17. Lynos, M., Akamatsu, S., Kamachi, M., & Gyboa, J. (1998). Coding facial expressions with Gabor wavelets. In Proceedings of the third IEEE international conference on automatic face and facial recognition (pp. 200–205). Google Scholar
  18. Murphy, R. F., Velliste, M., Yao, J., & Porreca, G. (2001). Searching online journals for fluorescence microscopy images depicting protein subcellular location patterns. In Proceedings of the second IEEE international symposium on bioinformatics and biomedical engeering (pp. 119–128). Google Scholar
  19. Orlov, N., Johnston, J., Macura, T., Shamir, L., & Goldberg, I. G. (2007). Computer vision for microscopy applications. In G. Obinata & A. Dutta (Eds.), Vision systems—segmentation and pattern recognition (pp. 221–242). Vienna: ARS. Google Scholar
  20. Orlov, N., Shamir, L., Johnston, J., Macura, T., Eckley, D. M., & Goldberg, I. G. (2008, in press). WND-CHARM: Multi-purpose image classification using compound image transforms. Pattern Recognition Letters. Google Scholar
  21. Otsu, N. (1979). A threshold selection method from gray level histograms. IEEE Transactions on System, Man and Cybernetics, 9, 62–66. Google Scholar
  22. Phillips, P. J., Wechsler, H., Huang, J., & Rauss, P. (1998). The FERET database and evaluation procedure for face recognition algorithms. Journal of Image and Vision Computing, 16, 295–306. CrossRefGoogle Scholar
  23. Phillips, P. J., Moon, H., Rizvi, S. A., & Rauss, P. J. (2000). The FERET evaluation methodology for face recognition algorithms. IEEE Transactions Pattern Analysis and Machine Intelligence, 22, 1090–1104. CrossRefGoogle Scholar
  24. Pinto, N., Cox, D. D., & DiCarlo, J. J. (2008). Why is real-world visual object recognition hard? PLoS Computational Biology, 4, e27. CrossRefMathSciNetGoogle Scholar
  25. Prewitt, J. M. (1970). Object enhancement and extraction. In B. S. Lipkin & A. Rosenfeld (Eds.), Picture processing and psychopictoris (pp. 75–149). New York: Academic Press. Google Scholar
  26. Rodenacker, K., & Bengsson, E. (2003). A feature set for cytometry on digitized microscopic images. Analytical Cellular Pathology, 25, 1–36. Google Scholar
  27. Samaria, F., & Harter, A. C. (1994). Parameterisation of a stochastic model for human face identification. In Proceedings of the second IEEE workshop on applications of computer vision. Google Scholar
  28. Hond, D., & Spacek, L. (1997). Distinctive descriptions for face processing. In Proceedings of 8th BMVC (pp. 320–329). Google Scholar
  29. Spacek, L. (2002), University of Essex face database.
  30. Swedlow, J. R., Goldberg, I., Brauner, E., & Sorger, P. K. (2003). Image informatics and quantitative analysis of biological images. Science, 300, 100–102. CrossRefGoogle Scholar
  31. Tamura, H., Mori, S., & Yamavaki, T. (1978). Textural features corresponding to visual perception. IEEE Transactions on System, Man and Cybernetics, 8, 460–472. Google Scholar
  32. Teague, M. R. (1979). Image analysis via the general theory of moments. Journal of the Optical Society of America, 70, 920. MathSciNetCrossRefGoogle Scholar
  33. Zhao, W., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2005). Face recognition: A literature survey. ACM Computing Surveys, 35, 399–458. CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  1. 1.Laboratory of Genetics, National Institute on AgingNational Institutes of HealthBaltimoreUSA

Personalised recommendations