Evaluation Platform for Content-Based Image Retrieval Systems

  • Petra Budikova
  • Michal Batko
  • Pavel Zezula
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6966)


In all subfields of information retrieval, test datasets and ground truth data are important tools for testing and comparison of new search methods. This is also reflected by the image retrieval community where several benchmarking activities have been created in past years. However, the number of available test collections is still rather small and the existing ones are often limited in size or accessible only to the participants of benchmarking competitions. In this work, we present a new freely-available large-scale dataset for evaluation of content-based image retrieval systems. The dataset consists of 20 million high-quality images with five visual descriptors and rich and systematic textual annotations, a set of 100 test query objects and a semi-automatically collected ground truth data verified by users. Furthermore, we provide services that enable exploitation and collaborative expansion of the ground truth.


large-scale image dataset visual and textual annotation ground truth collaboration service 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Batko, M., Falchi, F., Lucchese, C., Novak, D., Perego, R., Rabitti, F., Sedmidubský, J., Zezula, P.: Building a web-scale image similarity search system. Multimedia Tools Appl. 47(3), 599–629 (2010)CrossRefGoogle Scholar
  2. 2.
    Batko, M., Kohoutkova, P., Zezula, P.: Combining metric features in large collections. In: ICDE Workshops, pp. 370–377. IEEE Computer Society, Los Alamitos (2008)Google Scholar
  3. 3.
    Batko, M., Novak, D., Zezula, P.: MESSIF: Metric similarity search implementation framework. In: Thanos, C., Borri, F., Candela, L. (eds.) Digital Libraries: Research and Development. LNCS, vol. 4877, pp. 1–10. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Budikova, P., Batko, M., Zezula, P.: Similarity query postprocessing by ranking. In: 8th International Workshop on Adaptive Multimedia Retrieval (2010)Google Scholar
  5. 5.
    Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2) (2008)Google Scholar
  6. 6.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: CVPR 2009 (2009)Google Scholar
  7. 7.
    Fellbaum, C. (ed.): WordNet: An Electronic Lexical Database. The MIT Press, Cambridge (1998)MATHGoogle Scholar
  8. 8.
    Fluhr, C., Moëllic, P.A., Hède, P.: Usage-oriented multimedia information retrieval technological evaluation. In: Multimedia Information Retrieval, pp. 301–306 (2006)Google Scholar
  9. 9.
    Huiskes, M.J., Lew, M.S.: The MIR Flickr retrieval evaluation. In: Proc. of the Multimedia Information Retrieval. ACM, New York (2008)Google Scholar
  10. 10.
    Jain, R., Sinha, P.: Content without context is meaningless. In: ACM Multimedia, pp. 1259–1268 (2010)Google Scholar
  11. 11.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  12. 12.
    Marchand-Maillet, S., Worring, M.: Benchmarking image and video retrieval: an overview. In: Multimedia Information Retrieval, pp. 297–300 (2006)Google Scholar
  13. 13.
    Morrison, D., Marchand-Maillet, S., Bruno, E.: TagCaptcha: annotating images with CAPTCHAs. In: Proc. of the ACM Multimedia, pp. 1557–1558 (2010)Google Scholar
  14. 14.
    MPEG-7: Multimedia content description interfaces. Part 3: Visual. ISO/IEC 15938-3:2002 (2002)Google Scholar
  15. 15.
    Müller, H., Müller, W., Marchand-Maillet, S., Pun, T., Squire, D.M.: A framework for benchmarking in CBIR. Multimedia Tools Appl. 21(1), 55–73 (2003)CrossRefGoogle Scholar
  16. 16.
    Natsev, A., Haubold, A., Tesic, J., Xie, L., Yan, R.: Semantic concept-based query expansion and re-ranking for multimedia retrieval. In: ACM Multimedia, pp. 991–1000 (2007)Google Scholar
  17. 17.
    Novak, D., Batko, M., Zezula, P.: Generic similarity search engine demonstrated by an image retrieval application. In: Proceedings of SIGIR 2009, p. 840 (2009)Google Scholar
  18. 18.
    Popescu, A., Tsikrika, T., Kludas, J.: Overview of the Wikipedia Retrieval Task at ImageCLEF 2010. In: CLEF (Notebook Papers/LABs/Workshops) (2010)Google Scholar
  19. 19.
    Westerveld, T., van Zwol, R.: The INEX 2006 multimedia track. In: Fuhr, N., Lalmas, M., Trotman, A. (eds.) INEX 2006. LNCS, vol. 4518, pp. 331–344. Springer, Heidelberg (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Petra Budikova
    • 1
  • Michal Batko
    • 1
  • Pavel Zezula
    • 1
  1. 1.Masaryk UniversityBrnoCzech Republic

Personalised recommendations