Skip to main content

‘Expected Most of the Results, but Some Others...Surprised Me’: Personality Inference in Image Tagging Services

  • 466 Accesses

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 12724)


Image tagging APIs, offered as Cognitive Services in the movement to democratize AI, have become popular in applications that need to provide a personalized user experience. Developers can easily incorporate these services into their applications; however, little is known concerning their behavior under specific circumstances. We consider how two such services behave when predicting elements of the Big-Five personality traits from users’ profile images. We found that personality traits are not equally represented in the APIs’ output tags, with tags focusing mostly on Extraversion. The inaccurate personality prediction and the lack of vocabulary for the equal representation of all personality traits, could result in unreliable implicit user modeling, resulting in sub-optimal – or even undesirable – user experience in the application.


  • Algorithmic bias
  • Cognitive services
  • Personality
  • Image analysis

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-79840-6_12
  • Chapter length: 9 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   64.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-79840-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   84.99
Price excludes VAT (USA)


  1. 1.

  2. 2.

  3. 3.

  4. 4.

    WordNet is the most widely used English lexical database which includes nouns, verbs, adjectives, and adverbs. The words are organized and linked based on their lexical concept (set of synonyms).

  5. 5.

    It is important to remind the reader that these services are effectively “black boxes,” thus, the complete list of their tags is not publicly available, not even to the developers who are incorporating them in the software they are developing.

  6. 6.


  1. Barlas, P., Kleanthous, S., Kyriakou, K., Otterbacher, J.: Social b(eye)as in image tagging algorithms: Human and machine descriptions of people images. Proc. AAAI ICWSM 13(01), 583–591 (2019)

    Google Scholar 

  2. Buolamwini, J., Gebru, T.: Gender shades: intersectional accuracy disparities in commercial gender classification. In: Conference on Fairness. Accountability and Transparency, pp. 77–91. PMLR, New York (2018)

    Google Scholar 

  3. Celli, F., Bruni, E., Lepri, B.: Automatic personality and interaction style recognition from Facebook profile pictures. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 1101–1104. ACM, New York (2014)

    Google Scholar 

  4. Everitt, B., Hothorn, T.: An Introduction to Applied Multivariate Analysis with R (UseR!), January 2011.

  5. Goldberg, L.: An alternative “description of personality’’: the big-five factor structure. J. Pers. Soc. Psychol. 59, 1216–29 (1991)

    CrossRef  Google Scholar 

  6. Guntuku, S.C., Lin, W., Carpenter, J., Ng, W.K., Ungar, L.H., Preoţiuc-Pietro, D.: Studying personality through the content of posted and liked images on twitter. In: Proceedings of the 2017 ACM on Web Science Conference, WebSci 2017, pp. 223–227 ACM, New York (2017).

  7. Herring, S.C.: Web content analysis: expanding the paradigm. In: Hunsinger, J., Allen, M. (eds.) International Handbook of Internet Research, pp. 233–249. Springer, Dordrecht (2009).

    CrossRef  Google Scholar 

  8. Kyriakou, K., Barlas, P., Kleanthous, S., Otterbacher, J.: Fairness in proprietary image tagging algorithms: a cross-platform audit on people images. In: Proceedings of AAAI ICWSM, vol. 13, pp. 313–322. AAAI, California (2019)

    Google Scholar 

  9. Kyriakou, K., Kleanthous, S., Otterbacher, J., Papadopoulos, G.A.: Emotion-based stereotypes in image analysis services. In: Adjunct Publication of the 28th ACM UMAP Conference, pp. 252–259 (2020)

    Google Scholar 

  10. Norman, W.T.: 2800 personality trait descriptors: normative operating characteristics for a university population. University of Michigan, Ann Arbor, Michigan (1967)

    Google Scholar 

  11. Rhue, L.: Racial influence on automated perceptions of emotions. SSRN 3281765 (2018)

    Google Scholar 

  12. Silveira Jacques Junior, J.C., et al.: First impressions: a survey on vision-based apparent personality trait analysis. IEEE Trans. Affect. Comput. 1 (2019)

    Google Scholar 

Download references


This project is partially funded by the Cyprus Research and Innovation Foundation under grant EXCELLENCE/0918/0086 (DESCANT) and by the European Union’s Horizon 2020 Research and Innovation Programme under agreements No. 739578 (RISE) and 810105 (CyCAT).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Styliani Kleanthous .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Kasinidou, M., Kleanthous, S., Otterbacher, J. (2021). ‘Expected Most of the Results, but Some Others...Surprised Me’: Personality Inference in Image Tagging Services. In: Fogli, D., Tetteroo, D., Barricelli, B.R., Borsci, S., Markopoulos, P., Papadopoulos, G.A. (eds) End-User Development. IS-EUD 2021. Lecture Notes in Computer Science(), vol 12724. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-79839-0

  • Online ISBN: 978-3-030-79840-6

  • eBook Packages: Computer ScienceComputer Science (R0)