Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-Maximization

  • Ning Gao
  • William Webber
  • Douglas W. Oard
Conference paper

DOI: 10.1007/978-3-319-06028-6_1

Part of the Lecture Notes in Computer Science book series (LNCS, volume 8416)
Cite this paper as:
Gao N., Webber W., Oard D.W. (2014) Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-Maximization. In: de Rijke M. et al. (eds) Advances in Information Retrieval. ECIR 2014. Lecture Notes in Computer Science, vol 8416. Springer, Cham

Abstract

Relevance judgments are often the most expensive part of information retrieval evaluation, and techniques for comparing retrieval systems using fewer relevance judgments have received significant attention in recent years. This paper proposes a novel system comparison method using an expectation-maximization algorithm. In the expectation step, real-valued pseudo-judgments are estimated from a set of system results. In the maximization step, new system weights are learned from a combination of a limited number of actual human judgments and system pseudo-judgments for the other documents. The method can work without any human judgments, and is able to improve its accuracy by incrementally adding human judgments. Experiments using TREC Ad Hoc collections demonstrate strong correlations with system rankings using pooled human judgments, and comparison with existing baselines indicates that the new method achieves the same comparison reliability with fewer human judgments.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Ning Gao
    • 1
  • William Webber
    • 2
  • Douglas W. Oard
    • 1
  1. 1.College of Information Studies/UMIACSUniversity of MarylandCollege ParkUSA
  2. 2.William Webber ConsultingUSA

Personalised recommendations