Analysing Facial Features Using CNNs and Computer Vision

  • Diana BorzaEmail author
  • Razvan Itu
  • Radu Danescu
  • Ioana Barbantan
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1126)


This paper presents an automatic facial analysis system which is able to perform gender detection, hair segmentation and geometry detection, color attributes extraction (hair, skin, eyebrows, eyes and lips), accessories (eyeglasses) analysis from facial images. For the more complex tasks (gender detection, hair segmentation, eyeglasses detection) we used state of the art convolutional neural networks, and for the other tasks we used classical image processing algorithms based on geometry and appearance models. When data was available, the proposed system was evaluated on public datasets. An acceptance study was also performed to assess the performance on the system in real life scenarios.


Facial attributes analysis Gender detection Hair segmentation Convolutional neural networks Color analysis 



This work was made possible with the support of the “Institute of Research, Development and Education in Computer Vision” (Asociația Institutul de Cercetare, Dezvoltare și Educație în Viziune Artificială,, Cluj-Napoca, Romania, and Tapptitude (, a product development agency, where the first iteration of the algorithm was built.


  1. 1.
    Borza, D., Itu, R., Danescu, R.: In the eye of the deceiver: analyzing eye movements as a cue to deception. J. Imaging 4(10), 120 (2018)CrossRefGoogle Scholar
  2. 2.
    Dantcheva, A., Erdogmus, N., Dugelay, J.L.: On the reliability of eye color classification as a soft biometrics trait. In: Proceedings of WACV (2011, to appear)Google Scholar
  3. 3.
    Eidinger, E., Enbar, R., Hassner, T.: Age and gender estimation of unfiltered faces. IEEE Trans. Inf. Forensics Secur. 9(12), 2170–2179 (2014)CrossRefGoogle Scholar
  4. 4.
    Everingham, M., Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The Pascal visual object classes (VOC) challenge. Int. J. Comput. Vision 88(2), 303–338 (2010)CrossRefGoogle Scholar
  5. 5.
    Hansen, D.W., Ji, Q.: In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans. Pattern Anal. Mach. Intell. 32(3), 478–500 (2010)CrossRefGoogle Scholar
  6. 6.
    Harville, M., Baker, H., Bhatti, N., Susstrunk, S.: Consistent image-based measurement and classification of skin color. In: IEEE International Conference on Image Processing 2005, vol. 2, pp. II–374. IEEE (2005)Google Scholar
  7. 7.
    Hollingsworth, K.P., Bowyer, K.W., Flynn, P.J.: The importance of small pupils: a study of how pupil dilation affects iris biometrics. In: 2008 IEEE Conference on Biometrics: Theory, Applications and Systems. pp. 1–6. IEEE (2008)Google Scholar
  8. 8.
    Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  9. 9.
    Jmal, M., Mseddi, W.S., Attia, R., Youssef, A.: Classification of human skin color and its application to face recognition. In: The Sixth International Conferences on Advances in Multimedia, MMEDIA. Citeseer (2014)Google Scholar
  10. 10.
    Kae, A., Sohn, K., Lee, H., Learned-Miller, E.: Augmenting CRFs with Boltzmann machine shape priors for image labeling (2013)Google Scholar
  11. 11.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)Google Scholar
  12. 12.
    King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)Google Scholar
  13. 13.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Large-scale CelebFaces attributes (CelebA) dataset (2018). Accessed 15 Aug 2018Google Scholar
  14. 14.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  15. 15.
    Ma, D.S., Correll, J., Wittenbrink, B.: The Chicago face database: a free stimulus set of faces and norming data. Behav. Res. Methods 47(4), 1122–1135 (2015)CrossRefGoogle Scholar
  16. 16.
    Muhammad, U.R., Svanera, M., Leonardi, R., Benini, S.: Hair detection, segmentation, and hairstyle classification in the wild. Image Vis. Comput. 71, 25–37 (2018)CrossRefGoogle Scholar
  17. 17.
    Rousset, C., Coulon, P.Y.: Frequential and color analysis for hair mask segmentation. In: 15th IEEE International Conference on Image Processing, ICIP 2008, pp. 2276–2279. IEEE (2008)Google Scholar
  18. 18.
    Shen, Y., Peng, Z., Zhang, Y.: Image based hair segmentation algorithm for the application of automatic facial caricature synthesis. Sci. World J. 2014 (2014) Google Scholar
  19. 19.
    Yacoob, Y., Davis, L.: Detection, analysis and matching of hair. In: Tenth IEEE International Conference on Computer Vision, ICCV 2005, vol. 1, pp. 741–748. IEEE (2005)Google Scholar
  20. 20.
    Yoon, S., Harville, M., Baker, H., Bhatii, N.: Automatic skin pixel selection and skin color classification. In: 2006 International Conference on Image Processing, pp. 941–944. IEEE (2006)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Diana Borza
    • 1
    Email author
  • Razvan Itu
    • 1
  • Radu Danescu
    • 1
  • Ioana Barbantan
    • 2
  1. 1.Technical University of Cluj-NapocaCluj-NapocaRomania
  2. 2.TapptitudeCluj-NapocaRomania

Personalised recommendations