A Classification of IR Effectiveness Metrics

  • Gianluca Demartini
  • Stefano Mizzaro
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3936)


Effectiveness is a primary concern in the information retrieval (IR) field. Various metrics for IR effectiveness have been proposed in the past; we take into account all the 44 metrics we are aware of, classifying them into a two-dimensional grid. The classification is based on the notions of relevance, i.e., if (or how much) a document is relevant, and retrieval, i.e., if (how much) a document is retrieved. To our knowledge, no similar classification has been proposed so far.


Information Retrieval Relevance Judgment Binary Relevance Information Retrieval Technique Ranking Relevance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Belew, R.: Finding Out About. Cambridge Univ. Press, Cambridge (2000)zbMATHGoogle Scholar
  2. 2.
    Borlund, P., Ingwersen, P.: Measures of relative relevance and ranked half-life: Performance indicators for interactive IR. In: 21st SIGIR, pp. 324–331 (1998)Google Scholar
  3. 3.
    Brache, R.: Personal communication (2005)Google Scholar
  4. 4.
    Buckley, C., Voorhees, E.: Retrieval evaluation with incomplete information. In: 27th SIGIR, pp. 25–32 (2004)Google Scholar
  5. 5.
    Cooper, W.S.: Expected search length: A single measure of retrieval effectiveness based on weak ordering action of retrieval systems. JASIST 19, 30–41 (1968)Google Scholar
  6. 6.
    de Vries, A., Kazai, G., Lalmas, M.: Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit. In: RIAO 2004 Conference Proceedings, pp. 463–473 (2004)Google Scholar
  7. 7.
    Della Mea, V., Mizzaro, S.: Measuring retrieval effectiveness: A new proposal and a first experimental validation. JASIST 55(6), 530–543 (2004)CrossRefGoogle Scholar
  8. 8.
    Frei, H., Schauble, P.: Determining the effectiveness of retrieval algorithms. IPM 27(2), 153–164 (1991)Google Scholar
  9. 9.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. TOIS 20, 422–446 (2002)CrossRefGoogle Scholar
  10. 10.
    Kando, N., Kuriyama, K., Yoshioka, M.: Information retrieval system evaluation using multi-grade relevance judgments. In: IPSJ SIGNotes (2001)Google Scholar
  11. 11.
    Kazai, G.: Report of the INEX 2003 metrics working group. In: Proceedings of the 2nd INEX Workshop, pp. 184–190 (2004)Google Scholar
  12. 12.
    Kazai, G., Lalmas, M.: INEX, evaluation metrics (2005),
  13. 13.
    Korfhage, R.R.: Information Storage and Retrieval. John Wiley & Sons, Chichester (1997)Google Scholar
  14. 14.
    Losee, R.M.: Upper bounds for retrieval performance and their use measuring performance and generating optimal boolean queries. Can it get any better than this? IPM 30(2), 193–204 (1994)Google Scholar
  15. 15.
    Mizzaro, S.: Relevance: The whole history. JASIS 48(9), 810–832 (1997)CrossRefGoogle Scholar
  16. 16.
    Piwowarski, B., Gallinari, P.: Expected ratio of relevant units: A measure for structured information retrieval. In: Proceedings of INEX 2003, pp. 158–166 (2004)Google Scholar
  17. 17.
    Sakai, T.: New performance metrics based on multigrade relevance: Their application to question answering. In: NTCIR 4 Meeting Working Notes (2004)Google Scholar
  18. 18.
    Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1984)zbMATHGoogle Scholar
  19. 19.
    van Rijsbergen, C.J.: Information Retrieval. Butterworths, 2nd edn. (1979)Google Scholar
  20. 20.
    Yao, Y.Y.: Measuring retrieval effectiveness based on user preference of documents. JASIS 46(2), 133–145 (1995)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Gianluca Demartini
    • 1
  • Stefano Mizzaro
    • 1
  1. 1.Dept. of Mathematics and Computer ScienceUniversity of UdineUdineItaly

Personalised recommendations