Multimedia Tools and Applications

, Volume 59, Issue 2, pp 441–462 | Cite as

Tag-based algorithms can predict human ratings of which objects a picture shows

  • Viktoria PammerEmail author
  • Barbara Kump
  • Stefanie Lindstaedt


Collaborative tagging platforms allow users to describe resources with freely chosen keywords, so called tags. The meaning of a tag as well as the precise relation between a tag and the tagged resource are left open for interpretation to the user. Although human users mostly have a fair chance at interpreting this relation, machines do not. In this paper we study the characteristics of the problem to identify descriptive tags, i.e. tags that relate to visible objects in a picture. We investigate the feasibility of using a tag-based algorithm, i.e. an algorithm that ignores actual picture content, to tackle the problem. Given the theoretical feasibility of a well-performing tag-based algorithm, which we show via an optimal algorithm, we describe the implementation and evaluation of a WordNet-based algorithm as proof-of-concept. These two investigations lead to the conclusion that even relatively simple and fast tag-based algorithms can yet predict human ratings of which objects a picture shows. Finally, we discuss the inherent difficulty both humans and machines have when deciding whether a tag is descriptive or not. Based on a qualitative analysis, we distinguish between definitional disagreement, difference in knowledge, disambiguation and difference in perception as reasons for disagreement between raters.


Semantic annotation Image tagging behaviour Descriptive tags Flickr WordNet 



The Know-Center is funded within the Austrian COMET Program—Competence Centers for Excellent Technologies—under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.


  1. 1.
    Ames M, Naaman M (2007) Why we tag: motivations for annotation in mobile and online media. In: CHI ’07: proceedings of the SIGCHI conference on Human factors in computing systems. ACM, New York, pp 971–980CrossRefGoogle Scholar
  2. 2.
    Bechhofer S, Carr L, Goble CA, Kampa S, Miles-Board T (2002) The semantics of semantic annotation. In: On the move to meaningful internet systems, 2002—DOA/CoopIS/ODBASE 2002 Confederated International Conferences DOA, CoopIS and ODBASE 2002. Springer, London, pp 1152–1167CrossRefGoogle Scholar
  3. 3.
    Blanche MT, Durrheim K, Painter D (eds) (2006) Research in practice—applied methods for the social sciences. University of Cape Town PressGoogle Scholar
  4. 4.
    Cohen J, Cohen P, West SG, Aiken LS (2003) Applied multiple regression/correlation analysis for the behavioral sciences. Lawrence Erlbaum Associates. ISBN: 0805822232Google Scholar
  5. 5.
    Gangemi A, Guarino N, Oltramari R (2001) Conceptual analysis of lexical taxonomies: the case of wordnet top-level. In: Proceedings of the international cnference on formal ontology in information systems. ACM Press, pp 285–296Google Scholar
  6. 6.
    Golder SA, Hubermann BA (2006) Usage patterns of collaborative tagging systems. J Inf Sci 32(2):198–208CrossRefGoogle Scholar
  7. 7.
    Kurasaki KS (2000) Intercoder reliability for validating conclusions drawn from open-ended interview data. Field Methods 12(3):179–194CrossRefGoogle Scholar
  8. 8.
    Mika P (2005) Ontologies are us: a unified model of social networks and semantics. In: Gil Y, Motta E, Benjamins VR, Musen MA (eds) International semantic web conference. Lecture notes in computer science, vol 3729. Springer, pp 522–536Google Scholar
  9. 9.
    Miller GA (1995) Wordnet: a lexical database for english. Commun. ACM 38(11):39–41CrossRefGoogle Scholar
  10. 10.
    Nov O, Naaman M, Ye C (2008) What drives content tagging: the case of photos on flickr. In: CHI ’08: proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems. ACM, New York, pp 1097–1100CrossRefGoogle Scholar
  11. 11.
    OpenCyc (2010) Last visited: 10 Aug 2010
  12. 12.
    Pammer V, Kump B, Lindstaedt S (2009) On the feasibility of a tag-based approach for deciding which objects a picture shows: an empirical study. In: Semantic multimedia. Proceedings of 4th International Conference on Semantic and Digital Media Technologies, SAMT 2009. Lecture notes in computer science, vol 5887/2009. Graz, Austria, 2–4 Dec 2009. Springer Berlin/Heidelberg, pp 40–51Google Scholar
  13. 13.
    Rattenbury T, Good N, Naaman M (2007) Towards automatic extraction of event and place semantics from flickr tags. In: SIGIR ’07: proceedings of the 30th annual international ACM SIGIR conference. ACM Press, New York, pp 103–110CrossRefGoogle Scholar
  14. 14.
    Schmitz P (2006) Inducing ontology from flickr tags. In: Proceedings of the collaborative web tagging workshop at WWW2006. Edinburgh, ScotlandGoogle Scholar
  15. 15.
    Sigurbjörnsson B, van Zwol R (2008) Flickr tag recommendation based on collective knowledge. In: Huai J, Chen R, Hon H-W, Liu Y, Ma W-Y, Tomkins A, Zhang X (eds) (2008) WWW. ACM, pp 327–336Google Scholar
  16. 16.
    Sun A, Bhowmick SS (2009) Image tag clarity: in search of visual-representative tags for social images. In: WSM ’09: proceedings of the first SIGMM workshop on Social media. ACM, New York, pp 19–26CrossRefGoogle Scholar
  17. 17.
    Volkmer T, Thom JA, Tahaghoghi SMM (2007) Modeling human judgment of digital imagery for multimedia retrieval. IEEE Trans Multimedia 9(5):967–974CrossRefGoogle Scholar
  18. 18.
    Wimmer RD, Dominick JR (2006) Mass media research: an introduction, 8th edn. Thomson WadsworthGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Viktoria Pammer
    • 1
    Email author
  • Barbara Kump
    • 2
  • Stefanie Lindstaedt
    • 1
  1. 1.Know-CenterGrazAustria
  2. 2.Knowledge Management Institute, Graz University of TechnologyTübingenGermany

Personalised recommendations