Evaluation of visual information indexing and retrieval

Abstract

Comparative evaluation of methods and systems is of primarily importance in the domain of visual indexing and retrieval. As in many other domains, it is generally organized by institutions like NIST or by research networks like Pascal or PetaMedia. Evaluations are carried out in the context of periodical campaigns, or benchmarks. In these, one or more visual indexing or retrieval tasks are defined, each with a data collection, relevance judgments, performance measures and an experimentation protocol. Participants submit results computed automatically and blindly and the organizers return the measured performances. These evaluation campaigns are generally concluded by a workshop in which the participants explain how they performed the tasks. The chapter will give an overview of the major evaluation campaigns in the domain and present in detail the tasks, the data collection, the metrics and the protocols used. The state of the art performance in recent campaigns and the lessons learned from these campaigns will also be presented.