Combining Deep Facial and Ambient Features for First Impression Estimation

  • Furkan Gürpınar
  • Heysem Kaya
  • Albert Ali Salah
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9915)

Abstract

First impressions influence the behavior of people towards a newly encountered person or a human-like agent. Apart from the physical characteristics of the encountered face, the emotional expressions displayed on it, as well as ambient information affect these impressions. In this work, we propose an approach to predict the first impressions people will have for a given video depicting a face within a context. We employ pre-trained Deep Convolutional Neural Networks to extract facial expressions, as well as ambient information. After video modeling, visual features that represent facial expression and scene are combined and fed to a Kernel Extreme Learning Machine regressor. The proposed system is evaluated on the ChaLearn Challenge Dataset on First Impression Recognition, where the classification target is the “Big Five” personality trait labels for each video. Our system achieved an accuracy of 90.94 % on the sequestered test set, 0.36 % points below the top system in the competition.

Keywords

Personality traits First impression Deep learning ELM 

References

  1. 1.
    Cuddy, A.J., Fiske, S.T., Glick, P.: Warmth and competence as universal dimensions of social perception: the stereotype content model and the bias map. Adv. Exp. Soc. Psychol. 40, 61–149 (2008)CrossRefGoogle Scholar
  2. 2.
    Lopez, V.P., Chen, B., Places, A., Oliu, M., Corneanu, C., Baro, X., Escalante, H.J., Guyon, I., Escalera, S.: ChaLearn LAP 2016: first round challenge on first impressions - dataset and results. In: ChaLearn Looking at People Workshop on Apparent Personality Analysis, ECCV Workshop Proceedings (2016)Google Scholar
  3. 3.
    Todorov, A., Mandisodza, A.N., Goren, A., Hall, C.C.: Inferences of competence from faces predict election outcomes. Science 308(5728), 1623–1626 (2005)CrossRefGoogle Scholar
  4. 4.
    Valente, F., Kim, S., Motlicek, P.: Annotation and recognition of personality traits in spoken conversations from the AMI meetings corpus. In: INTERSPEECH, pp. 1183–1186 (2012)Google Scholar
  5. 5.
    Madzlan, N., Han, J., Bonin, F., Campbell, N.: Towards automatic recognition of attitudes: prosodic analysis of video blogs. In: Speech Prosody, Dublin, Ireland, pp. 91–94 (2014)Google Scholar
  6. 6.
    Alam, F., Stepanov, E.A., Riccardi, G.: Personality traits recognition on social network-Facebook. In: WCPR (ICWSM-2013), Cambridge, MA, USA (2013)Google Scholar
  7. 7.
    Nowson, S., Gill, A.J.: Look! who’s talking? Projection of extraversion across different social contexts. In: Proceedings of the 2014 ACM Multimedia Workshop on Computational Personality Recognition, pp. 23–26. ACM (2014)Google Scholar
  8. 8.
    Gievska, S., Koroveshovski, K.: The impact of affective verbal content on predicting personality impressions in Youtube videos. In: Proceedings of the 2014 ACM Multimedia Workshop on Computational Personality Recognition, pp. 19–22. ACM (2014)Google Scholar
  9. 9.
    Fernando, T., et al.: Persons’ personality traits recognition using machine learning algorithms and image processing techniques. Adv. Comput. Sci.: Int. J. 5(1), 40–44 (2016)MathSciNetGoogle Scholar
  10. 10.
    Qin, R., Gao, W., Xu, H., Hu, Z.: Modern physiognomy: an investigation on predicting personality traits and intelligence from the human face. arXiv preprint arXiv:1604.07499 (2016)
  11. 11.
    Sarkar, C., Bhatia, S., Agarwal, A., Li, J.: Feature analysis for computational personality recognition using Youtube personality data set. In: Proceedings of the 2014 ACM Multimedia Workshop on Computational Personality Recognition, pp. 11–14. ACM (2014)Google Scholar
  12. 12.
    Alam, F., Riccardi, G.: Predicting personality traits using multimodal information. In: Proceedings of the 2014 ACM Multimedia Workshop on Computational Personality Recognition, pp. 15–18. ACM (2014)Google Scholar
  13. 13.
    Farnadi, G., Sushmita, S., Sitaraman, G., Ton, N., De Cock, M., Davalos, S.: A multivariate regression approach to personality impression recognition of vloggers. In: Proceedings of the 2014 ACM Multimedia Workshop on Computational Personality Recognition, pp. 1–6. ACM (2014)Google Scholar
  14. 14.
    Sidorov, M., Ultes, S., Schmitt, A.: Automatic recognition of personality traits: a multimodal approach. In: Proceedings of the 2014 Workshop on Mapping Personality Traits Challenge, pp. 11–15. ACM (2014)Google Scholar
  15. 15.
    Gosling, S.D., Rentfrow, P.J., Swann, W.B.: A very brief measure of the big-five personality domains. J. Res. Pers. 37(6), 504–528 (2003)CrossRefGoogle Scholar
  16. 16.
    Borkenau, P., Liebler, A.: Trait inferences: sources of validity at zero acquaintance. J. Pers. Soc. Psychol. 62(4), 645 (1992)CrossRefGoogle Scholar
  17. 17.
    Zebrowitz, L.A., Collins, M.A.: Accurate social perception at zero acquaintance: the affordances of a gibsonian approach. Pers. Soc. Psychol. Rev. 1(3), 204–223 (1997)CrossRefGoogle Scholar
  18. 18.
    Vinciarelli, A., Mohammadi, G.: A survey of personality computing. IEEE Trans. Affect. Comput. 5(3), 273–291 (2014)CrossRefGoogle Scholar
  19. 19.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  20. 20.
    Mathias, M., Benenson, R., Pedersoli, M., Gool, L.: Face detection without bells and whistles. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 720–735. Springer, Heidelberg (2014). doi:10.1007/978-3-319-10593-2_47 Google Scholar
  21. 21.
    Xiong, X., De la Torre, F.: Supervised descent method and its application to face alignment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539(2013)Google Scholar
  22. 22.
    Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 FPS via regressing local binary features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692 (2014)Google Scholar
  23. 23.
    Xiong, X., De la Torre, F.: Global supervised descent method. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2664–2673 (2015)Google Scholar
  24. 24.
    Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)CrossRefMATHGoogle Scholar
  25. 25.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893. IEEE (2005)Google Scholar
  26. 26.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  27. 27.
    Almaev, T.R., Valstar, M.F.: Local Gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 356–361. IEEE (2013)Google Scholar
  28. 28.
    Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection using sparse appearance descriptors in space-time video volumes. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 314–321. IEEE (2011)Google Scholar
  29. 29.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105 (2012)Google Scholar
  30. 30.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  31. 31.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015)Google Scholar
  32. 32.
    Kim, B.K., Lee, H., Roh, J., Lee, S.Y.: Hierarchical committee of deep CNNs with exponentially-weighted decision fusion for static facial expression recognition. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 427–434. ACM (2015)Google Scholar
  33. 33.
    Rothe, R., Timofte, R., Gool, L.: Dex: deep expectation of apparent age from a single image. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 10–15 (2015)Google Scholar
  34. 34.
    Liu, X., Li, S., Kan, M., Zhang, J., Wu, S., Liu, W., Han, H., Shan, S., Chen, X.: Agenet: deeply learned regressor and classifier for robust apparent age estimation. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 16–24 (2015)Google Scholar
  35. 35.
    Zhu, Y., Li, Y., Mu, G., Guo, G.: A study on apparent age estimation. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 25–31 (2015)Google Scholar
  36. 36.
    Escalera, S., Torres, M., Martinez, B., Baro, X., Jair Escalante, H., Guyon, I., Tzimiropoulos, G., Corneou, C., Oliu, M., Bagheri, M.A., Valstar, M.: Chalearn looking at people and faces of the world: Face analysis workshop and challenge 2016. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1–8, June 2016Google Scholar
  37. 37.
    Gürpınar, F., Kaya, H., Dibeklioğlu, H., Salah, A.A.: Kernel ELM and CNN based facial age estimation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Las Vegas, Nevada, USA, pp. 80–86, June 2016Google Scholar
  38. 38.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)MATHGoogle Scholar
  39. 39.
    Huang, G.B., Zhou, H., Ding, X., Zhang, R.: Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man Cybern. Part B: Cybern. 42(2), 513–529 (2012)CrossRefGoogle Scholar
  40. 40.
    Zong, W., Huang, G.B.: Face recognition based on extreme learning machine. Neurocomputing 74(16), 2541–2551 (2011)CrossRefGoogle Scholar
  41. 41.
    Mohammed, A.A., Minhas, R., Wu, Q.J., Sid-Ahmed, M.A.: Human face recognition based on multidimensional PCA and extreme learning machine. Pattern Recogn. 44(10), 2588–2597 (2011)CrossRefMATHGoogle Scholar
  42. 42.
    Utama, P., Ajie, H., et al.: A framework of human emotion recognition using extreme learning machine. In: 2014 International Conference of Advanced Informatics: Concept, Theory and Application (ICAICTA), pp. 315–320. IEEE (2014)Google Scholar
  43. 43.
    Kaya, H., Karpov, A.A., Salah, A.A.: Robust acoustic emotion recognition based on cascaded normalization and extreme learning machines. In: Cheng, L., Liu, Q., Ronzhin, A. (eds.) ISNN 2016. LNCS, vol. 9719, pp. 115–123. Springer, Heidelberg (2016). doi:10.1007/978-3-319-40663-3_14 CrossRefGoogle Scholar
  44. 44.
    An, L., Yang, S., Bhanu, B.: Efficient smile detection by extreme learning machine. Neurocomputing 149, 354–363 (2015)CrossRefGoogle Scholar
  45. 45.
    Kaya, H., Gürpınar, F., Afshar, S., Salah, A.A.: Contrasting and combining least squares based learners for emotion recognition in the wild. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 459–466. ACM (2015)Google Scholar
  46. 46.
    Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). doi:10.1007/978-3-642-42051-1_16 CrossRefGoogle Scholar
  47. 47.
    Rao, C.R., Mitra, S.K.: Generalized Inverse of Matrices and Its Applications, vol. 7. Wiley, New York (1971)MATHGoogle Scholar
  48. 48.
    Vedaldi, A., Lenc, K.: MatConvNet - convolutional neural networks for MATLAB. (2015)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Furkan Gürpınar
    • 1
  • Heysem Kaya
    • 2
  • Albert Ali Salah
    • 3
  1. 1.Program of Computational Science and EngineeringBoğaziçi UniversityIstanbulTurkey
  2. 2.Department of Computer EngineeringNamık Kemal UniversityTekirdağTurkey
  3. 3.Department of Computer EngineeringBoğaziçi UniversityIstanbulTurkey

Personalised recommendations