On the Evaluation of Entity Profiles

  • Maarten de Rijke
  • Krisztian Balog
  • Toine Bogers
  • Antal van den Bosch
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6360)

Abstract

Entity profiling is the task of identifying and ranking descriptions of a given entity. The task may be viewed as one where the descriptions being sought are terms that need to be selected from a knowledge source (such as an ontology or thesaurus). In this case, entity profiling systems can be assessed by means of precision and recall values of the descriptive terms produced. However, recent evidence suggests that more sophisticated metrics are needed that go beyond mere lexical matching of system-produced descriptors against a ground truth, allowing for graded relevance and rewarding diversity in the list of descriptors returned. In this note, we motivate and propose such a metric.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Al-Maskari, A., Sanderson, M., Clough, P.: The relationship between IR effectivenss measures and user satisfaction. In: SIGIR 2007, pp. 773–774. ACM, New York (2007)Google Scholar
  2. 2.
    Balog, K., de Rijke, M.: Determining expert profiles (with an application to expert finding). In: IJCAI 2007, pp. 2657–2662 (2007)Google Scholar
  3. 3.
    Balog, K., Bogers, T., Azzopardi, L., de Rijke, M., van den Bosch, A.: Broad expertise retrieval in sparse data environments. In: SIGIR 2007, pp. 551–558. ACM, New York (2007)Google Scholar
  4. 4.
    Balog, K., Azzopardi, L., de Rijke, M.: A language modeling framework for expert finding. Inf. Proc. & Manag. 45(1), 1–19 (2009)CrossRefGoogle Scholar
  5. 5.
    Balog, K., Bogers, T., van den Bosch, A., de Rijke, M.: Expertise profiling in a knowledge-intensive organization (submitted, 2010)Google Scholar
  6. 6.
    Bogers, T., Balog, K.: UvT Expert Collection documentation. Technical report, ILK Research Group Technical Report Series no. 07-06 (July 2007)Google Scholar
  7. 7.
    Brooks, T.A.: Topical subject expertise and the semantic distance model of relevance assessment. Journal of Documentation 51, 370–387 (1995)CrossRefGoogle Scholar
  8. 8.
    Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: SIGIR 2008, pp. 659–666. ACM, New York (2008)Google Scholar
  9. 9.
    Dang, H.T., Lin, J., Kelly, D.: Overview of the TREC 2006 question answering track. In: TREC 2006. NIST (2006)Google Scholar
  10. 10.
    Hofmann, K., Tsagkias, E., Meij, E.J., de Rijke, M.: The impact of document structure on keyphrase extraction. In: CIKM 2009, Hong Kong. ACM, New York (November 2009)Google Scholar
  11. 11.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)CrossRefGoogle Scholar
  12. 12.
    Jijkoun, V., de Rijke, M.: Overview of the WiQA task at CLEF 2006. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 265–274. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  13. 13.
    Lin, J., Demner-Fushman, D.: Will pyramids built of nuggets topple over. In: HLT 2006, Morristown, NJ, USA, pp. 383–390. Association for Computational Linguistics (2006)Google Scholar
  14. 14.
    Meij, E.J., Bron, M., Huurnink, B., Hollink, L., de Rijke, M.: Learning semantic query suggestions. In: Bernstein, A., Karger, D.R., Heath, T., Feigenbaum, L., Maynard, D., Motta, E., Thirunarayan, K. (eds.) ISWC 2009. LNCS, vol. 5823, Springer, Heidelberg (2009)CrossRefGoogle Scholar
  15. 15.
    Voorhees, E.M.: Overview of the TREC 2004 question answering track. In: TREC 2004. NIST (2005)Google Scholar
  16. 16.
    Voorhees, E.M., Dang, H.T.: Overview of the TREC 2005 question answering track. In: TREC 2005. NIST (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Maarten de Rijke
    • 1
  • Krisztian Balog
    • 1
  • Toine Bogers
    • 2
  • Antal van den Bosch
    • 3
  1. 1.ISLAUniversity of AmsterdamThe Netherlands
  2. 2.IIIARoyal School of Library & Information ScienceDenmark
  3. 3.ILK/Tilburg Centre for Creative ComputingTilburg UniversityThe Netherlands

Personalised recommendations