Multimedia Tools and Applications

, Volume 78, Issue 1, pp 727–746 | Cite as

Understanding personality of portrait by social embedding visual features

  • Jie Nie
  • Zhiqiang Wei
  • Zhen Li
  • Yan Yan
  • Lei HuangEmail author


Image understanding achieves unprecedented performance in content recognition and emotion rating recently. However, previous research mainly focused on visual features. In this paper, inspired by human cognitive activities, we discuss how to measure portrait distance in respective of high level semantic: personality, by employing features from both visual contents and behavior contents. Firstly, a new image distance metric, named Social and Visual Portrait Distance, is designed by jointly considering visual features and human social media behavior features. In portrait images, visual features are defined globally and locally utilizing theoretical and empirical concepts from psychology theory. While in social media, behavior features are designed with considerations of demographic factors, identity claims and behavioral residue. And the new distance is estimated referred to feature reliability. Secondly, we modify the proposed distance calculation formula in order to apply it on only visual features but preserving both visual and social relations by a new Social Embedding Portrait Distance Learning method. In this manner, we could measure the social embedding visual distance of common portrait images in absence of social media information, such as web portrait, or daily photos. Comprehensive experiments are employed to investigate the effectiveness of the new portrait distance and the metric learning method in representing personality distance compared with several baselines. Moreover, the learned distance matrix reveals a reasonable explanation of social preferred visual features with contribute partitions.


Portrait Image distance Metric learning Social media Personality 



This work is supported by the National Natural Science Foundation of China (No.61402428, No.61702471, No. 61672475); Aoshan Innovation Project in Science and Technology of Qingdao National Laboratory for Marine Science and Technology (No. 2016ASKJ07); Qingdao Science and Technology Development Plan (No. 16-5-1-13-jch).


  1. 1.
    Back MD et al (2010) Facebook profiles reflect actual personality, not self-idealization. Psychol Sci 21(3):372–374CrossRefGoogle Scholar
  2. 2.
    Bhaskar H, Mihaylova L, Maskell S (2013) Articulated human body parts detection based on cluster background subtraction and foreground matching. Neurocomputing 100:58–73CrossRefGoogle Scholar
  3. 3.
    Big Five personality traits (2017) Available Accessed 29 May 2017
  4. 4.
    Buffardi LE, Campbell WK (2008) Narcissism and social networking web sites. Personal Soc Psychol Bull 34(10):1303–1314CrossRefGoogle Scholar
  5. 5.
    Cristani M, Vinciarelli A, Segalin C, Perina A (2013) Unveiling the multimedia unconscious: Implicit cognitive processes and multimedia content analysis. In: Proceedings of the 21st ACM international conference on Multimedia, pp 213–222Google Scholar
  6. 6.
    Deng J, Berg AC, Fei-fei L (2011) Hierarchical semantic indexing for large scale image retrieval. In: IEEE Conference on Computer Vision and Pattern Recognition, Colorado SpringsGoogle Scholar
  7. 7.
    Gosling SD, Ko SJ, Mannarelli T, Morris ME (2002) A room with a cue: personality judgments based on offices and bedrooms. J Pers Soc Psychol 82(3):379CrossRefGoogle Scholar
  8. 8.
    Gosling SD, Ko SJ, Mannarelli T, Morris ME (Mar. 2002) A room with a cue: personality judgments based on offices and bedrooms. J Pers Soc Psychol 82(3):379–398CrossRefGoogle Scholar
  9. 9.
    Gosling SD, Gaddis S, Vazire S et al (2007) Personality impressions based on Facebook profiles. Int Conf Weblogs Soc Media 7:1–4Google Scholar
  10. 10.
    Ivcevic Z, Ambady N (2012) Personality impressions from identity claims on Facebook. Psychol Pop Media Cult 1(1):38–45CrossRefGoogle Scholar
  11. 11.
    Jia J, Wu S, Wang X, Hu P, Cai L, Tang J (2012) Can we understand van gogh’s mood?: learning to infer affects from images in social networks. In: Proceedings of the 20th ACM international conference on Multimedia, pp 857–860Google Scholar
  12. 12.
    Kendall MG (1938) A new measure of rank correlation. Biometrika. [Online]. Available Accessed 29 May 2017
  13. 13.
    Kiapour MH, Yamaguchi K, Berg AC, Berg TL (2014) Hipster wars: discovering elements of fashion styles. In: European Conference on Computer Vision, Zurich, pp 472–488Google Scholar
  14. 14.
    Le Q, Mikolov T (2014) Distributed Representations of Sentences and Documents. In: PMLR, pp 1188–1196Google Scholar
  15. 15.
    Li F, Ding Y, Zhou M, Hao K, Chen L (2016) An affection-based dynamic leader selection model for formation control in multirobot systems. IEEE Trans Syst Man Cybern Syst PP(99):1–12CrossRefGoogle Scholar
  16. 16.
    Liu S, Cui P, Zhu W, Yang S, Tian Q (2014) Social Embedding Image Distance Learning. In: Proceedings of the 22Nd ACM International Conference on Multimedia, New York, pp 617–626Google Scholar
  17. 17.
    Liu L, Preotiuc-Pietro D, Samani ZR, Moghaddam ME, Ungar L (2016) Analyzing Personality through Social Media Profile Picture Choice. In: Tenth International AAAI Conference on Web and Social MediaGoogle Scholar
  18. 18.
    Machajdik J, Hanbury A (2010) Affective image classification using features inspired by psychology and art theory. In: Proceedings of the 18th ACM international conference on Multimedia, pp 83–92Google Scholar
  19. 19.
    Naumann LP, Vazire S, Rentfrow PJ, Gosling SD (2009) Personality judgments based on physical appearance. Personal Soc Psychol Bull 35(12):1661–1671CrossRefGoogle Scholar
  20. 20.
    Nie J, Cui P, Yan Y, Huang L, Li Z, Wei Z (2014) How your portrait impresses people? Inferring personality impressions from portrait contents. In: MM 2014 - Proceedings of the 2014 ACM Conference on Multimedia, pp 905–908Google Scholar
  21. 21.
    Nie J, Huang L, Li Z, Wei C, Hong B, Zhu W (2016) Thinking like psychologist: Learning to predict personality by using features from portrait and social media. In: 2016 4th International Conference on Cloud Computing and Intelligence Systems (CCIS), pp 21–26Google Scholar
  22. 22.
    Picard RW, Picard R (1997) Affective computing, vol 252. MIT press, CambridgeGoogle Scholar
  23. 23.
    Steele F, Evans DC, Green RK (2009) Is Your Profile Picture Worth 1000 Words? Photo Characteristics Associated with Personality Impression Agreement. In: AAAI International Conference on Weblogs & Social Media, San JoseGoogle Scholar
  24. 24.
    Todorov A, Said C, Verosky S (2011) Personality impressions from facial appearance. In: Calder A, Haxby JV, Johnson M, Rhodes G (eds) Handbook of face perception. Oxford University Press, New York, pp 631–652Google Scholar
  25. 25.
    Wang HL, Cheong L-F (2006) Affective understanding in film. IEEE Trans Circuits Syst Video Technol 16(6):689–704CrossRefGoogle Scholar
  26. 26.
    Wang W, He Q (2008) A survey on emotional semantic image retrieval. In: IEEE International Conference on Image Processing, pp 117–120Google Scholar
  27. 27.
    Weinberger KQ, Saul LK (2009) Distance metric learning for large margin nearest neighbor classification. J Mach Learn Res 10(Feb):207–244zbMATHGoogle Scholar
  28. 28.
    Weisbuch M, Ivcevic Z, Ambady N (2009) On being liked on the web and in the real world: consistency in first impressions across personal webpages and spontaneous behavior. J Exp Soc Psychol 45(3):573–576CrossRefGoogle Scholar
  29. 29.
    Yan Y, Nie J, Huang L, Li Z, Cao Q, Wei Z (2015) Is your first impression reliable? trustworthy analysis using facial traits in portraits. In: International Conference on Multimedia Modeling, Sydney, pp 148–158Google Scholar
  30. 30.
    Yan Y, Nie J, Huang L, Li Z, Cao Q, Wei Z (2016) Exploring relationship between face and trustworthy impression using mid-level facial features. In: International Conference on Multimedia Modeling, Miami, pp 540–549Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of Information Science and EngineeringOcean University of ChinaQingdaoChina

Personalised recommendations