Multimedia Tools and Applications

, Volume 42, Issue 1, pp 73–96 | Cite as

Attributing semantics to personal photographs

  • Rodrigo F. CarvalhoEmail author
  • Sam Chapman
  • Fabio Ciravegna


A major bottleneck for the efficient management of personal photographic collections is the large gap between low-level image features and high-level semantic contents of images. This paper proposes and evaluates two methodologies for making appropriate (re)use of natural language photographic annotations for extracting references to people, location and objects and propagating any location references encountered to previously unannotated images. The evaluation identifies the strengths of each approach and shows extraction and propagation results with promising accuracy.


Photographs Semantic capture Information extraction Clustering Image Annotation 



This work was sponsored by Kodak Limited. We would also like to thank the 391 online photo sharing users who donated their photographs and respective metadata.


  1. 1.
    Ahn L, Dabbish L (2004) Labeling images with a computer game. In: CHI ’04. ACM, New York, pp 319–326CrossRefGoogle Scholar
  2. 2.
    Alp Aslandogan Y, Thier C, Yu CT, Zou J, Rishe N (1997) Using semantic contents and wordnet in image retrieval. In: SIGIR ’97: proceedings of the 20th annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 286–295CrossRefGoogle Scholar
  3. 3.
    Bang HY, Zhang C, Chen T (2004) Semantic propagation from relevance feedbacks. In: 2004 IEEE international conference on multimedia and expo. ICME ’04, 27–30 June. vol 1. IEEE, Piscataway, pp 81–84Google Scholar
  4. 4.
    Barla A, Odone F, Verri A (2003) Old fashioned state-of-the-art image classification. In: Proc. of ICIAP 2003, Mantova, 17–19 September 2003, pp 566–571Google Scholar
  5. 5.
    Budura A, Michel S, Cudre-Mauroux P, Aberer K (2008) To tag or not to tag ? harvesting adjacent metadata in large-scale tagging systems. In: The 31st annual international ACM SIGIR conference, 20–24 July 2008, SingaporeGoogle Scholar
  6. 6.
    Cooper M, Foote J, Girgensohn A, Wilcox L (2003) Temporal event clustering for digital photo collections. In MULTIMEDIA ’03: proceedings of the eleventh ACM international conference on multimedia. ACM, New York, pp 364–373CrossRefGoogle Scholar
  7. 7.
    Duda RO, Hart PE (1973) Pattern classification and scene analysis. Wiley, New York, pp 98–105zbMATHGoogle Scholar
  8. 8.
    Frohlich D, Kuchinsky A, Pering C, Don A, Ariss S (2002) Requirements for photoware. In: CSCW ’02: proceedings of the 2002 ACM conference on Computer supported cooperative work. ACM, New York, pp 166–175CrossRefGoogle Scholar
  9. 9.
    Greenwood M, Iria J, Ciravegna F (2008) Saxon: an extensible multimedia annotator. In LREC’ 08: proceedings of the 6th international conference on language resources and evaluation, Morocco, May 2008Google Scholar
  10. 10.
    Hare JS, Lewis PH (2005) Saliency-based models of image content and their application to auto-annotation by semantic propagation. In: Multimedia and the semantic web/European semantic web conference 2005, Heraklion, 29 May–1 June 2005Google Scholar
  11. 11.
    Hartigan JA, Wong MA (1979) A K-means clustering algorithm. Appl Stat 28:100–108zbMATHCrossRefGoogle Scholar
  12. 12.
    Hearst M (1992) Automatic acquisition of hyponyms from large text corpora. In: Proc. of COLING 1992, 23–28 August 1992, Nantes, pp 539–545Google Scholar
  13. 13.
    Iria J, Ireson N, Ciravegna F (2006) An experimental study on boundary classification algorithms for information extraction using svm. In: Proc. of EACL 2006, Montreal, 22–27 April 2006Google Scholar
  14. 14.
    Keyvanpour M, Asbaghi S, Fathy M (2007) A new scheme of automatic semantic propagation in the image data base using a hierarchical structure of semantics. In: DEXA ’07: proceedings of the 18th international conference on database and expert systems applications. IEEE Computer Society, Washington, DC, pp 59–63CrossRefGoogle Scholar
  15. 15.
    Manjunath B (2001) Color and texture descriptors. IEEE Trans Circuits Syst Video Technol 11:703–715CrossRefGoogle Scholar
  16. 16.
    Miller AD, Edwards KW (2007) Give and take: a study of consumer photo-sharing culture and practice. In: CHI ’07: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, New York, pp 347–356CrossRefGoogle Scholar
  17. 17.
    Naaman M, Harada S, Wang Q, Garcia-Molina H, Paepcke A (2004) Context data in geo-referenced digital photo collections. In: Proc. of ACM MM, October 2004Google Scholar
  18. 18.
    Park DK, Jeon YS, Won CS (2000) Efficient use of local edge histogram descriptor. In: MULTIMEDIA ’00: Proceedings of the 2000 ACM workshops on Multimedia. ACM, New York, pp 51–54CrossRefGoogle Scholar
  19. 19.
    Pastra K, Saggion H, Wilks Y (2002) Extracting relational facts for indexing and retrieval of crime-scene photographs. In Macintosh A, Ellis R, Coenen F (eds) Applications and innovations in intelligent systems X, British Computer Society Conference Series. Springer, Heidelberg, pp 121–134Google Scholar
  20. 20.
    Pelleg D, Moore A (2000) X-means: Extending K-means with efficient estimation of the number of clusters. In: Proc. 17th International Conf. on Machine Learning. Morgan Kaufmann, San Francisco, pp 727–734Google Scholar
  21. 21.
    Salembier P, Sikora T (2002) Introduction to MPEG-7: multimedia content description interface. Wiley, New YorkGoogle Scholar
  22. 22.
    Spyrou E, LeBorgne H, Mailis T, Cooke E, Avrithis Y, O’Connor N (2005) Fusing MPEG-7 visual descriptors for image classiffication. In: Duch W, Kacprzyk J, Oja E, Zadrozny S (eds) In: Artificial neural networks, part II: formal models and their applications, vol 3697. Springer, Heidelberg, pp 847–852Google Scholar
  23. 23.
    Srihari R (1995) Automatic indexing and content-based retrieval of captioned images. Computer 28(9):49–56CrossRefGoogle Scholar
  24. 24.
    Stauder J, Sirot J, Le Borgne H, Cooke E, O’Connor NE (2004) Relating visual and semantic image descriptors. In: EWIMT 2004—European workshop on the integration of knowledge, semantics and digital media technologyGoogle Scholar
  25. 25.
    Veltkamp R, Tanase M (2000) Content-based image retrieval systems: a survey. Technical report UU-CS-2000-34, Dept. of Computing Science, Utrecht UniversityGoogle Scholar
  26. 26.
    Zhang D, Tsotras VJ (2001) Improving min/max aggregation over spatial objects. In: ACM-GIS, pp 88–93Google Scholar
  27. 27.
    Zhang H, Chen Z, Li M, Su Z (2003) Relevance feedback and learning in content-based image search. World Wide Web 6(2):131–155CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Rodrigo F. Carvalho
    • 1
    Email author
  • Sam Chapman
    • 1
  • Fabio Ciravegna
    • 1
  1. 1.Department of Computer Science, Natural Language Processing GroupThe University of SheffieldSheffieldUK

Personalised recommendations