Encyclopedia of Database Systems

2018 Edition
| Editors: Ling Liu, M. Tamer Özsu

Test Collection

  • Ben Carterette
Reference work entry
DOI: https://doi.org/10.1007/978-1-4614-8265-9_5052

Synonyms

Corpus

Definition

A test collection is a standard set of data used to measure search engine performance. It comprises a set of queries, ideally randomly sampled from some space, a set of documents to be searched, and a set of judgments indicating the relevance of each document to each query in the set.

Key Points

The use of test collections for performance evaluation began with Cleverdon and Mills [1] and is today known as the Cranfield methodology. Test collections today are much larger than Cleverdon’s Cranfield collection, consisting of millions of documents and tens of thousands of relevance judgments. The advantage of having standardized test collections is that experimental results can be compared across research groups and over time.

The National Institute of Standards and Technology (NIST), through their annual Text REtrieval Conferences (TREC), has led the way in providing test collections for information retrieval research. NIST has assembled large-scale test...

This is a preview of subscription content, log in to check access.

Recommended Reading

  1. 1.
    Voorhees EM, Harman DK, editors. TREC: experiment and evaluation in information retrieval. Cambridge: MIT; 2005.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of Massachusetts AmherstAmherstUSA