Skip to main content
SpringerLink
Log in

Rethinking How to Extend Average Precision to Graded Relevance

  • Conference paper
Book cover Information Access Evaluation. Multilinguality, Multimodality, and Interaction (CLEF 2014)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 8685))

Abstract

We present two new measures of retrieval effectiveness, inspired by Graded Average Precision(GAP), which extends Average Precision(AP) to graded relevance judgements. Starting from the random choice of a user, we define Extended Graded Average Precision(xGAP) and Expected Graded Average Precision(eGAP), which are more accurate than GAP in the case of a small number of highly relevant documents with high probability to be considered relevant by the users. The proposed measures are then evaluated on TREC 10, TREC 14, and TREC 21 collections showing that they actually grasp a different angle from GAP and that they are robust when it comes to incomplete judgments and shallow pools.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Buckley, C., Voorhees, E.M.: Retrieval Evaluation with Incomplete Information. In: SIGIR 2007, pp. 25–32. ACM Press, USA (2004)

    Google Scholar 

  2. Buckley, C., Voorhees, E.M.: Retrieval System Evaluation. In: TREC. Experiment and Evaluation in Information Retrieval, pp. 53–78. MIT Press, USA (2005)

    Google Scholar 

  3. Clarke, C.L.A., Craswell, N., Voorhees, H.: Overview of the TREC 2012 Web Track. In: TREC 2012, pp. 1–8. NIST, Special Publication 500-298, USA (2013)

    Google Scholar 

  4. Hawking, D., Craswell, N.: Overview of the TREC-2001 Web Track. In TREC 2001, pp. 61–67. NIST, Special Publication 500-250, USA (2001)

    Google Scholar 

  5. Järvelin, K., Kekäläinen, J.: Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems (TOIS) 20(4), 422–446 (2002)

    Article  Google Scholar 

  6. Kendall, M.G.: Rank correlation methods. Griffin, Oxford (1948)

    MATH  Google Scholar 

  7. Moffat, A., Zobel, J.: Rank-biased Precision for Measurement of Retrieval Effectiveness. ACM Transactions on Information Systems (TOIS) 27(1), 2:1–2:27 (2008)

    Google Scholar 

  8. Resnick, S.I.: A Probability Path. Birkhäuser, Boston (2005)

    Book  Google Scholar 

  9. Robertson, S.E., Kanoulas, E., Yilmaz, E.: Extending Average Precision to Graded Relevance Judgments. In: SIGIR 2010, pp. 603–610. ACM Press, USA (2010)

    Google Scholar 

  10. Voorhees, E.: Evaluation by Highly Relevant Documents. In: SIGIR 2001, pp. 74–82. ACM Press, USA (2001)

    Google Scholar 

  11. Voorhees, E.M.: Overview of the TREC 2005 Robust Retrieval Track. In: TREC 2005. NIST, Special Pubblication 500-266, USA (2005)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dept. of Mathematics, University of Padua, Italy

    Marco Ferrante

  2. Dept. of Information Engineering, University of Padua, Italy

    Nicola Ferro & Maria Maistro

Authors
  1. Marco Ferrante
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicola Ferro
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maria Maistro
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Google Inc., Brandschenkestraße 110, 8002, Zurich, Switzerland

    Evangelos Kanoulas

  2. Institute of Software Technology and Interactive Systems, Vienna University of Technology, Favoritenstrasse 9-11, 1040, Vienna, Austria

    Mihai Lupu

  3. Information School, University of Sheffield, Sheffield, UK

    Paul Clough

  4. Department of Computer Science and IT, RMIT University, 3000, Melbourne, VIC, Australia

    Mark Sanderson

  5. Department of Computing, Edge Hill University, L39 4QP, Ormskirk, Lancashire, UK

    Mark Hall

  6. Vienna University of Technology, Austria

    Allan Hanbury

  7. Information School, University of Sheffield, Regent Court, 211 Portobello, S1 4DP, Sheffield, UK

    Elaine Toms

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Ferrante, M., Ferro, N., Maistro, M. (2014). Rethinking How to Extend Average Precision to Graded Relevance. In: Kanoulas, E., et al. Information Access Evaluation. Multilinguality, Multimodality, and Interaction. CLEF 2014. Lecture Notes in Computer Science, vol 8685. Springer, Cham. https://doi.org/10.1007/978-3-319-11382-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-11382-1_3

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-11381-4

  • Online ISBN: 978-3-319-11382-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation