Encyclopedia of Database Systems

2018 Edition
| Editors: Ling Liu, M. Tamer Özsu

Evaluation Metrics for Structured Text Retrieval

  • Jovan Pehcevski
  • Benjamin Piwowarski
Reference work entry
DOI: https://doi.org/10.1007/978-1-4614-8265-9_152

Synonyms

Evaluation of XML retrieval effectiveness; Performance metrics

Definition

Anevaluation metric is used to evaluate the effectiveness of information retrieval systems and to justify theoretical and/or pragmatic developments of these systems. It consists of a set of measures that follow a common underlying evaluation methodology.

There are many metrics that can be used to evaluate the effectiveness of structured text retrieval systems. These metrics are based on different evaluation assumptions, incorporate different hypotheses of the expected user behavior, and implement their own evaluation methodologies to handle the level of overlap among the units of retrieval.

Historical Background

Over the past 5 years, the initiative for the evaluation of XML retrieval (INEX) has investigated various aspects of structured text retrieval, by particularly focusing on XML retrieval. Major advances, both in terms of approaches to XML retrieval and evaluation of XML retrieval, have been made...

This is a preview of subscription content, log in to check access.

Recommended Reading

  1. 1.
    Gövert N, Fuhr N, Lalmas M, Kazai G. Evaluating the effectiveness of content-oriented XML retrieval methods. Inf Retr. 2006;9(6):699–722.CrossRefGoogle Scholar
  2. 2.
    Järvelin K, Kekäläinen J. Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst. 2002;20(4):422–46.CrossRefGoogle Scholar
  3. 3.
    Kazai G. Choosing an ideal recall-base for the evaluation of the focused task: sensitivity analysis of the XCG evaluation measures. In: Proceedings of the Comparative Evaluation of XML Information Retrieval Systems: Fifth Workshop of the Initiative for the Evaluation of XML Retrieval; 2007. p. 35–44.Google Scholar
  4. 4.
    Kazai G, Lalmas M. Notes on what to measure in INEX. In: Proceedings of the INEX 2005 Workshop on Element Retrieval Methodology; 2005. p. 22–38.Google Scholar
  5. 5.
    Kazai G, Lalmas M. eXtended Cumulated Gain measures for the evaluation of content-oriented XML retrieval. ACM Trans Inf Syst. 2006;24(4):503–42.CrossRefGoogle Scholar
  6. 6.
    Kazai G, Lalmas M, de Vries AP. The overlap problem in content-oriented XML retrieval evaluation. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 2004. p. 72–9.Google Scholar
  7. 7.
    Pehcevski J. Evaluation of effective XML information retrieval. RMIT University, Melbourne, Australia, Ph.D. thesis, 2006.Google Scholar
  8. 8.
    Pehcevski J, Thom JA. HiXEval: highlighting XML retrieval evaluation. In: Advances in XML Information Retrieval and Evaluation: Proceedings of the Fourth Workshop of the Initiative for the Evaluation of XML Retrieval; 2006. p. 43–57.Google Scholar
  9. 9.
    Piwowarski B, Dupret G. Evaluation in (XML) information retrieval: expected precision-recall with user modelling (EPRUM). In: Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 2006. p. 260–7.Google Scholar
  10. 10.
    Piwowarski B, Gallinari P. Expected ratio of relevant units: a measure for structured information retrieval. In: Proceedings of the 2nd International Workshop of the Initiative for the Evaluation of XML Retrieval; 2003. p. 158–66.Google Scholar
  11. 11.
    Piwowarski B, Gallinari P, Dupret G. Precision recall with user modelling (PRUM): application to structured information retrieval. ACM Trans Inf Syst. 2007;25(1):1–37.CrossRefGoogle Scholar
  12. 12.
    Raghavan V, Bollmann P, Jung G. A critical investigation of recall and precision. ACM Trans Inf Syst. 1989;7(3):205–29.CrossRefGoogle Scholar
  13. 13.
    de Vries A, Kazai G, Lalmas M. Tolerance to irrelevance: a user-effort evaluation of retrieval systems without predefined retrieval unit. In: Proceedings of the 7th International Conference on Computer-Assisted Information Retrieval; 2004. p. 463–73.Google Scholar
  14. 14.
    Woodley A. and Geva S. XCG overlap at INEX 2004. In: Proceedings of the 4th International Workshop of the Initiative for the Evaluation of XML Retrieval; 2005. p. 25–39.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.INRIA Paris-RocquencourtLe Chesnay CedexFrance
  2. 2.University of GlasgowGlasgowUK

Section editors and affiliations

  • Jaap Kamps
    • 1
  1. 1.University of AmsterdamAmsterdamThe Netherlands