Multimodal Personality Trait Analysis for Explainable Modeling of Job Interview Decisions

  • Heysem Kaya
  • Albert Ali Salah
Part of the The Springer Series on Challenges in Machine Learning book series (SSCML)


Automatic analysis of job interview screening decisions is useful for establishing the nature of biases that may play a role in such decisions. In particular, assessment of apparent personality gives insights into the first impressions evoked by a candidate. Such analysis tools can be used for training purposes, if they can be configured to provide appropriate and clear feedback. In this chapter, we describe a multimodal system that analyzes a short video of a job candidate, producing apparent personality scores and a prediction about whether the candidate will be invited for a further job interview or not. This system provides a visual and textual explanation about its decision, and was ranked first in the ChaLearn 2017 Job Candidate Screening Competition. We discuss the application scenario and the considerations from a broad perspective.


Explainable machine learning Job candidate screening Multimodal affective computing Personality trait analysis 



This work is supported by Boğaziçi University Project BAP 16A01P4 and by the BAGEP Award of the Science Academy.


  1. Agerström J, Rooth DO (2011) The role of automatic obesity stereotypes in real hiring discrimination. Journal of Applied Psychology 96(4):790–805CrossRefGoogle Scholar
  2. Agerström J, Björklund F, Carlsson R, Rooth DO (2012) Warm and competent Hassan= cold and incompetent Eric: A harsh equation of real-life hiring discrimination. Basic and Applied Social Psychology 34(4):359–366CrossRefGoogle Scholar
  3. Alam F, Riccardi G (2014) Predicting personality traits using multimodal information. In: Proceedings of the 2014 ACM Multi Media on Workshop on Computational Personality Recognition, ACM, pp 15–18Google Scholar
  4. Alam F, Stepanov EA, Riccardi G (2013) Personality traits recognition on social network-Facebook. WCPR (ICWSM-13), Cambridge, MA, USAGoogle Scholar
  5. Almaev TR, Valstar MF (2013) Local Gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: Humaine Association Conference on Affective Computing and Intelligent Interaction, IEEE, pp 356–361Google Scholar
  6. Apers C, Derous E (2017) Are they accurate? recruiters’ personality judgments in paper versus video resumes. Computers in Human Behavior 73:9–19CrossRefGoogle Scholar
  7. Barezi EJ, Kampman O, Bertero D, Fung P (2018) Investigating audio, visual, and text fusion methods for end-to-end automatic personality prediction. arXiv preprint arXiv:180500705Google Scholar
  8. Breiman L (2001) Random forests. Machine learning 45(1):5–32CrossRefGoogle Scholar
  9. Celiktutan O, Gunes H (2016) Automatic prediction of impressions in time and across varying context: Personality, attractiveness and likeability. IEEE Transactions on Affective Computing 8(1):29–42, CrossRefGoogle Scholar
  10. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, vol 1, pp 886–893Google Scholar
  11. Dipboye RL (2005) Looking the part: Bias against the physically unattractive as discrimination issue. Discrimination at work: The psychological and organizational bases pp 281–301Google Scholar
  12. Dipboye RL, Macan T, Shahani-Denning C (2012) The selection interview from the interviewer and applicant perspectives: Can’t have one without the other. The Oxford handbook of personnel assessment and selection pp 323–352Google Scholar
  13. Escalante HJ, Ponce-López V, Wan J, Riegler MA, Chen B, Clapés A, Escalera S, Guyon I, Baró X, Halvorsen P, et al (2016) Chalearn joint contest on multimedia challenges beyond visual analysis: An overview. In: Pattern Recognition (ICPR), 2016 23rd International Conference on, IEEE, pp 67–73Google Scholar
  14. Escalante HJ, Guyon I, Escalera S, Jacques J, Madadi M, Baró X, Ayache S, Viegas E, Güçlütürk Y, Güçlü U, et al (2017) Design of an explainable machine learning challenge for video interviews. In: Neural Networks (IJCNN), 2017 International Joint Conference on, IEEE, pp 3688–3695Google Scholar
  15. Escalante HJ, Kaya H, Salah AA, Escalera S, Gucluturk Y, Guclu U, Baro X, Guyon I, Junior JJ, Madadi M, et al (2018) Explaining first impressions: Modeling, recognizing, and explaining apparent personality from videos. arXiv preprint arXiv:180200745Google Scholar
  16. Eyben F, Wöllmer M, Schuller B (2010) OpenSMILE: the Munich versatile and fast open-source audio feature extractor. In: ACM International Conference on Multimedia, pp 1459–1462Google Scholar
  17. Farnadi G, Sushmita S, Sitaraman G, Ton N, De Cock M, Davalos S (2014) A multivariate regression approach to personality impression recognition of vloggers. In: Proceedings of the 2014 ACM Multi Media on Workshop on Computational Personality Recognition, ACM, pp 1–6Google Scholar
  18. Fernando T, et al (2016) Persons’ personality traits recognition using machine learning algorithms and image processing techniques. Advances in Computer Science: an International Journal 5(1):40–44Google Scholar
  19. Fiske ST, Cuddy AJ, Glick P, Xu J (2002) A model of (often mixed) stereotype content: competence and warmth respectively follow from perceived status and competition. Journal of personality and social psychology 82(6):878–902CrossRefGoogle Scholar
  20. Fiske ST, Cuddy AJ, Glick P (2007) Universal dimensions of social cognition: Warmth and competence. Trends in cognitive sciences 11(2):77–83CrossRefGoogle Scholar
  21. Gievska S, Koroveshovski K (2014) The impact of affective verbal content on predicting personality impressions in youtube videos. In: Proceedings of the 2014 ACM Multi Media on Workshop on Computational Personality Recognition, ACM, pp 19–22Google Scholar
  22. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee DH, et al (2013) Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing, Springer, pp 117–124Google Scholar
  23. Güçlütürk Y, Güçlü U, van Gerven M, van Lier R (2016) Deep impression: Audiovisual deep residual networks for multimodal apparent personality trait recognition. In: ChaLearn Looking at People Workshop on Apparent Personality Analysis, ECCV Workshop Proceedings, pp 349–358Google Scholar
  24. Güçlütürk Y, Güçlü U, Baro X, Escalante HJ, Guyon I, Escalera S, van Gerven MAJ, van Lier R (2017) Multimodal first impression analysis with deep residual networks. IEEE Transactions on Affective Computing, online CrossRefGoogle Scholar
  25. Gürpınar F, Kaya H, Salah AA (2016a) Combining deep facial and ambient features for first impression estimation. In: ChaLearn Looking at People Workshop on Apparent Personality Analysis, ECCV Workshop Proceedings, pp 372–385Google Scholar
  26. Gürpınar F, Kaya H, Salah AA (2016b) Multimodal Fusion of Audio, Scene, and Face Features for First Impression Estimation. In: ChaLearn Joint Contest and Workshop on Multimedia Challenges Beyond Visual Analysis, Collocated with ICPR 2016, Cancun, MexicoGoogle Scholar
  27. Hayes TL, Macan TH (1997) Comparison of the factors influencing interviewer hiring decisions for applicants with and those without disabilities. Journal of Business and Psychology 11(3):357–371CrossRefGoogle Scholar
  28. Huang GB, Zhu QY, Siew CK (2004) Extreme Learning Machine: a new learning scheme of feedforward neural networks. In: IEEE International Joint Conference on Neural Networks, vol 2, pp 985–990Google Scholar
  29. Junior JCSJ, Güçlütürk Y, Perez M, Güçlü U, Andujar C, Baro X, Escalante HJ, Guyon I, van Gerven MAJ, van Lier R, Escalera S (2018) First impressions: A survey on computer vision-based apparent personality trait analysis. arXiv preprint arXiv:180408046 URL
  30. Kaya H, Salah AA (2014) Continuous mapping of personality traits: A novel challenge and failure conditions. In: Proceedings of the 2014 ICMI Workshop on Mapping Personality Traits Challenge, ACM, pp 17–24Google Scholar
  31. Kaya H, Gürpınar F, Salah AA (2017a) Multi-modal Score Fusion and Decision Trees for Explainable Automatic Job Candidate Screening from Video CVs. In: CVPR Workshops, Honolulu, Hawaii, USA, pp 1651–1659Google Scholar
  32. Kaya H, Gürpınar F, Salah AA (2017b) Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image and Vision Computing 65:66–75, DOI CrossRefGoogle Scholar
  33. Larkin JC, Pines HA (1979) No fat persons need apply: experimental studies of the overweight stereotype and hiring preference. Sociology of Work and Occupations 6(3):312–327CrossRefGoogle Scholar
  34. Lopez VP, Chen B, Places A, Oliu M, Corneanu C, Baro X, Escalante HJ, Guyon I, Escalera S (2016) Chalearn lap 2016: First round challenge on first impressions - dataset and results. In: ChaLearn Looking at People Workshop on Apparent Personality Analysis, ECCV Workshop Proceedings, Springer, pp 400–418Google Scholar
  35. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2):91–110MathSciNetCrossRefGoogle Scholar
  36. Madzlan N, Han J, Bonin F, Campbell N (2014) Towards automatic recognition of attitudes: Prosodic analysis of video blogs. Speech Prosody, Dublin, Ireland pp 91–94Google Scholar
  37. Nowson S, Gill AJ (2014) Look! who’s talking?: Projection of extraversion across different social contexts. In: Proceedings of the 2014 ACM Multi Media on Workshop on Computational Personality Recognition, ACM, pp 23–26Google Scholar
  38. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7):971–987CrossRefGoogle Scholar
  39. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22(10):1345–1359CrossRefGoogle Scholar
  40. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: British Machine Vision ConferenceGoogle Scholar
  41. Qin R, Gao W, Xu H, Hu Z (2016) Modern physiognomy: An investigation on predicting personality traits and intelligence from the human face. arXiv preprint arXiv:160407499Google Scholar
  42. Sarkar C, Bhatia S, Agarwal A, Li J (2014) Feature analysis for computational personality recognition using youtube personality data set. In: Proceedings of the 2014 ACM Multi Media on Workshop on Computational Personality Recognition, ACM, pp 11–14Google Scholar
  43. Schuller B, Steidl S, Batliner A, Nöth E, Vinciarelli A, Burkhardt F, Van Son R, Weninger F, Eyben F, Bocklet T, et al (2012) The INTERSPEECH 2012 speaker trait challenge. In: INTERSPEECH, pp 254–257Google Scholar
  44. Schuller B, Steidl S, Batliner A, Vinciarelli A, Scherer K, Ringeval F, Chetouani M, Weninger F, Eyben F, Marchi E, Mortillaro M, Salamin H, Polychroniou A, Valente F, Kim S (2013) The INTERSPEECH 2013 Computational Paralinguistics Challenge: Social Signals, Conflict, Emotion, Autism. In: INTERSPEECH, Lyon, France, pp 148–152Google Scholar
  45. Sidorov M, Ultes S, Schmitt A (2014) Automatic recognition of personality traits: A multimodal approach. In: Proceedings of the 2014 Workshop on Mapping Personality Traits Challenge and Workshop, ACM, pp 11–15Google Scholar
  46. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556Google Scholar
  47. Valente F, Kim S, Motlicek P (2012) Annotation and recognition of personality traits in spoken conversations from the ami meetings corpus. In: INTERSPEECH, pp 1183–1186Google Scholar
  48. Willis J, Todorov A (2006) First impressions: Making up your mind after a 100-ms exposure to a face. Psychological science 17(7):592–598CrossRefGoogle Scholar
  49. Xiong X, De la Torre F (2013) Supervised Descent Method and Its Application to Face Alignment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 532–539Google Scholar
  50. Zhang CL, Zhang H, Wei XS, Wu J (2016) Deep bimodal regression for apparent personality analysis. In: ChaLearn Looking at People Workshop on Apparent Personality Analysis, ECCV Workshop Proceedings, pp 311–324Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Department of Computer EngineeringNamik Kemal UniversityCorluTurkey
  2. 2.Department of Computer EngineeringBogazici UniversityIstanbulTurkey
  3. 3.Future Value Creation Research CenterNagoya UniversityNagoyaJapan

Personalised recommendations