A Formal Approach to Effectiveness Metrics for Information Access: Retrieval, Filtering, and Clustering

  • Enrique Amigó
  • Julio Gonzalo
  • Stefano Mizzaro
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9022)

Abstract

In this tutorial we present a formal account of evaluation metrics for three of the most salient information related tasks: Retrieval, Clustering, and Filtering. We focus on the most popular metrics and, by exploiting measurement theory, we show some constraints for suitable metrics in each of the three tasks. We also systematically compare metrics according to how they satisfy such constraints, we provide criteria to select the most adequate metric for each specific information access task, and we discuss how to combine and weight metrics.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Amigó, E., Gonzalo, J., Artiles, J., Verdejo, F.: A comparison of extrinsic clustering evaluation metrics based on formal constraints. Inf. Retr. 12(4), 461–486 (2009)CrossRefGoogle Scholar
  2. 2.
    Amigó, E., Gonzalo, J., Artiles, J., Verdejo, F.: Combining evaluation metrics via the unanimous improvement ratio and its application to clustering tasks. J. Artif. Int. Res. 42(1), 689–718 (2011)MATHGoogle Scholar
  3. 3.
    Amigó, E., Gonzalo, J., Mizzaro, S.: A general account of effectiveness metrics for information tasks: retrieval, filtering, and clustering. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1289–1289. ACM (2014)Google Scholar
  4. 4.
    Amigó, E., Gonzalo, J., Verdejo, F.: A general evaluation measure for document organization tasks. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2013, pp. 643–652 (2013)Google Scholar
  5. 5.
    Busin, L., Mizzaro, S.: Axiometrics: An axiomatic approach to information retrieval effectiveness metrics. In: Proceedings of ICTIR 2013: 4th International Conference on the Theory of Information Retrieval, pp. 22–29. ACM, New York (2013)CrossRefGoogle Scholar
  6. 6.
    Carterette, B.: System effectiveness, user models, and user utility: a conceptual framework for investigation. In: Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2011, pp. 903–912. ACM, New York (2011)Google Scholar
  7. 7.
    Demartini, G., Mizzaro, S.: A classification of IR effectiveness metrics. In: Lalmas, M., MacFarlane, A., Rüger, S.M., Tombros, A., Tsikrika, T., Yavlinsky, A. (eds.) ECIR 2006. LNCS, vol. 3936, pp. 488–491. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  8. 8.
    Dom, B.E., Dom, B.E.: An information-theoretic external cluster-validity measure. Technical report, Research Report RJ 10219, IBM (2001)Google Scholar
  9. 9.
    Maddalena, E., Mizzaro, S.: Axiometrics: Axioms of information retrieval effectiveness metrics. In: Proceedings of the Sixth International Workshop on Evaluating Information Access (EVIA 2014), pp. 17–24 (December 9, 2014)Google Scholar
  10. 10.
    Maddalena, E., Mizzaro, S.: The Axiometrics Project. In: Basili, R., Crestani, F., Pennacchiotti, M. (eds.) Proceedings of the 5th Italian Information Retrieval Workshop, Roma, Italy, January 20-21. CEUR Workshop Proceedings, vol. 1127, pp. 11–15. CEUR-WS.org (2014)Google Scholar
  11. 11.
    Meila, M.: Comparing clusterings. In: Proc. of COLT 2003 (2003)Google Scholar
  12. 12.
    Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst. 27(1), 2:1–2:27 (2008)Google Scholar
  13. 13.
    Smucker, M.D., Clarke, C.L.: Time-based calibration of effectiveness measures. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2012, pp. 95–104. ACM, New York (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Enrique Amigó
    • 1
  • Julio Gonzalo
    • 1
  • Stefano Mizzaro
    • 2
  1. 1.nlp.uned.es, E.T.S.I. Informática, UNEDMadridSpain
  2. 2.Department of Mathematics and Computer ScienceUniversity of UdineUdineItaly

Personalised recommendations