Overview of the Photo Annotation Task in ImageCLEF@ICPR

  • Stefanie Nowak
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6388)


The Photo Annotation Task poses the challenge for automated annotation of 53 visual concepts in Flickr photos and was organized as part of the ImageCLEF@ICPR contest. In total, 12 research teams participated in the multilabel classification challenge while initially 17 research groups were interested and got access to the data. The participants were provided with a training set of 5,000 Flickr images with annotations, a validation set of 3,000 Flickr images with annotations and the test was performed on 10,000 Flickr images. The evaluation was carried out twofold: first the evaluation per concept was conducted by utilizing the Equal Error Rate (EER) and the Area Under Curve (AUC) and second the evaluation per example was performed with the Ontology Score (OS). Summarizing the results, an average AUC of 86.5% could be achieved, including concepts with an AUC of 96%. The classification performance for each image ranged between 59% and 100% with an average score of 85%. In comparison to the results achieved in ImageCLEF 2009, the detection performance increased for the concept-based evaluation by 2.2% EER and 2.5% AUC and showed a slight decrease for the example-based evaluation.


Receiver Operating Characteristic Curve Area Under Curve Equal Error Rate Sift Feature Label Density 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Huiskes, M.J., Lew, M.S.: The MIR Flickr Retrieval Evaluation. In: Proc. of the ACM Intern. Conference on Multimedia Information Retrieval (2008)Google Scholar
  2. 2.
    Nowak, S., Dunker, P.: A Consumer Photo Tagging Ontology: Concepts and Annotations. In: Dunker, P., Liebetrau, J., Nowak, S., Müller, H., Panagiotis, V. (eds.) Proc. of THESEUS-ImageCLEF Pre-Workshop, Corfu, Greece (2009)Google Scholar
  3. 3.
    Manning, C., Raghavan, P., Schütze, H.: An Introduction to Information Retrieval [Draft], ch. 8. Cambridge University Press, Cambridge (2009)Google Scholar
  4. 4.
    Tsoumakas, G., Vlahavas, I.: Random k-labelsets: An ensemble method for multilabel classification. In: Proc. of the European Conference on Machine Learning (2007)Google Scholar
  5. 5.
    Everingham, M., Zisserman, A., Williams, C., Van Gool, L., Allan, M., et al.: The 2005 PASCAL Visual Object Classes Challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 117–176. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  6. 6.
    Deselaers, T., Hanbury, A.: The visual concept detection task in ImageCLEF 2008. In: Peters, C., Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, T., Peñas, A., Petras, V. (eds.) Evaluating Systems for Multilingual and Multimodal Information Access. LNCS, vol. 5706, pp. 531–538. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  7. 7.
    Nowak, S., Lukashevich, H., Dunker, P., Rüger, S.: Performance measures for multilabel evaluation: a case study in the area of image classification. In: MIR 2010: Proceedings of the International Conference on Multimedia Information Retrieval, New York, USA, pp. 35–44 (2010)Google Scholar
  8. 8.
    Tahir, M.A., Fei, Y., Barnard, M., Awais, M., Mikolajczyk, K., Kittler, J.: The University of Surrey Visual Concept Detection System at ImageCLEF 2010: Working Notes. In: Ünay, D., Çataltepe, Z., Aksoy, S. (eds.) ICPR 2010. LNCS, vol. 6388, pp. 164–171. Springer, Heidelberg (2010)Google Scholar
  9. 9.
    van de Sande, K., Gevers, T., Smeulders, A.: The University of Amsterdam’s Concept Detection System at ImageCLEF 2009. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T. (eds.) CLEF 2009. LNCS, vol. 6242, pp. 261–268. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Dumont, E., Zhao, Z.Q., Glotin, H., Paris, S.: A new TFIDF Bag of Visual Words for Concept Detection. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T. (eds.) CLEF 2009. LNCS, vol. 6242. Springer, Heidelberg (2010)Google Scholar
  11. 11.
    Iftene, A., Vamanu, L., Croitoru, C.: UAIC at ImageCLEF 2009 Photo Annotation Task. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T. (eds.) CLEF 2009. LNCS, vol. 6242, pp. 283–286. Springer, Heidelberg (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Stefanie Nowak
    • 1
  1. 1.Audio-Visual SystemsFraunhofer IDMTIlmenauGermany

Personalised recommendations