Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge

  • Jimmy Lin
  • Matt Crane
  • Andrew Trotman
  • Jamie Callan
  • Ishan Chattopadhyaya
  • John Foley
  • Grant Ingersoll
  • Craig Macdonald
  • Sebastiano Vigna
Conference paper

DOI: 10.1007/978-3-319-30671-1_30

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9626)
Cite this paper as:
Lin J. et al. (2016) Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge. In: Ferro N. et al. (eds) Advances in Information Retrieval. ECIR 2016. Lecture Notes in Computer Science, vol 9626. Springer, Cham

Abstract

The Open-Source IR Reproducibility Challenge brought together developers of open-source search engines to provide reproducible baselines of their systems in a common environment on Amazon EC2. The product is a repository that contains all code necessary to generate competitive ad hoc retrieval baselines, such that with a single script, anyone with a copy of the collection can reproduce the submitted runs. Our vision is that these results would serve as widely accessible points of comparison in future IR research. This project represents an ongoing effort, but we describe the first phase of the challenge that was organized as part of a workshop at SIGIR 2015. We have succeeded modestly so far, achieving our main goals on the Gov2 collection with seven open-source search engines. In this paper, we describe our methodology, share experimental results, and discuss lessons learned as well as next steps.

Keywords

ad hoc retrieval Open-source search engines 

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Jimmy Lin
    • 1
  • Matt Crane
    • 1
  • Andrew Trotman
    • 2
  • Jamie Callan
    • 3
  • Ishan Chattopadhyaya
    • 4
  • John Foley
    • 5
  • Grant Ingersoll
    • 4
  • Craig Macdonald
    • 6
  • Sebastiano Vigna
    • 7
  1. 1.University of WaterlooWaterlooCanada
  2. 2.eBay Inc.San JoseUSA
  3. 3.Carnegie Mellon UniversityPittsburghUSA
  4. 4.LucidworksRedwood CityUSA
  5. 5.University of Massachusetts AmherstAmherstUSA
  6. 6.University of GlasgowGlasgowUK
  7. 7.Università degli Studi di MilanoMilanItaly

Personalised recommendations