Information Retrieval

, Volume 4, Issue 3, pp 275–293

A Test Collection for the Evaluation of Content-Based Image Retrieval Algorithms—A User and Task-Based Approach

  • Marjo Markkula
  • Marius Tico
  • Bemmu Sepponen
  • Katja Nirkkonen
  • Eero Sormunen
Article

DOI: 10.1023/A:1011954407169

Cite this article as:
Markkula, M., Tico, M., Sepponen, B. et al. Information Retrieval (2001) 4: 275. doi:10.1023/A:1011954407169

Abstract

Content-based image retrieval (CBIR) algorithms have been seen as a promising access method for digital photograph collections. Unfortunately, we have very little evidence of the usefulness of these algorithms in real user needs and contexts. In this paper, we introduce a test collection for the evaluation of CBIR algorithms. In the test collection, the performance testing is based on photograph similarity perceived by end-users in the context of realistic illustration tasks and environment. The building process and the characteristics of the resulting test collection are outlined, including a typology of similarity criteria expressed by the subjects judging the similarity of photographs. A small-scale study on the consistency of similarity assessments is presented. A case evaluation of two CBIR algorithms is reported. The results show clear correlation between the subjects' similarity assessments and the functioning of feature parameters of the tested algorithms.

content-based image retrieval evaluation test collections user needs 

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • Marjo Markkula
    • 1
  • Marius Tico
    • 2
  • Bemmu Sepponen
    • 3
  • Katja Nirkkonen
    • 3
  • Eero Sormunen
    • 1
  1. 1.Department of Information StudiesUniversity of TampereFinland
  2. 2.Digital Media Institute, TampereUniversity of TechnologyFinland
  3. 3.Department of Information ServicesUniversity of TampereFinland