Information System Evaluation

Abstract

Evaluation of information retrieval systems and specific information retrieval system technologies is becoming more important as organizations begin to build more complex and fully functional information retrieval systems for their corporate data. Growth in interest in information retrieval system technologies driven by the original TREC conferences has spurred the creation of many different test databases and forums to evaluate information retrieval technologies. It also has lead to a lot more thought and development on techniques used in the evaluation of specific technologies. The availability of large test sets has provided the basis for researchers and developers to validate that their algorithms are scalable and will operate in real world situations. One of the challenges in evaluation is the ambiguity of the definition of relevant and disagreement between evaluators. Algorithms such as the Kappa Coefficient can be used in adjusting for differences. The other challenge is how to evaluate an operational system you are changing versus doing your evaluation at a conference. The major different options for information retrieval system evaluation are described.