Abstract
Entity profiling is the task of identifying and ranking descriptions of a given entity. The task may be viewed as one where the descriptions being sought are terms that need to be selected from a knowledge source (such as an ontology or thesaurus). In this case, entity profiling systems can be assessed by means of precision and recall values of the descriptive terms produced. However, recent evidence suggests that more sophisticated metrics are needed that go beyond mere lexical matching of system-produced descriptors against a ground truth, allowing for graded relevance and rewarding diversity in the list of descriptors returned. In this note, we motivate and propose such a metric.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Al-Maskari, A., Sanderson, M., Clough, P.: The relationship between IR effectivenss measures and user satisfaction. In: SIGIR 2007, pp. 773–774. ACM, New York (2007)
Balog, K., de Rijke, M.: Determining expert profiles (with an application to expert finding). In: IJCAI 2007, pp. 2657–2662 (2007)
Balog, K., Bogers, T., Azzopardi, L., de Rijke, M., van den Bosch, A.: Broad expertise retrieval in sparse data environments. In: SIGIR 2007, pp. 551–558. ACM, New York (2007)
Balog, K., Azzopardi, L., de Rijke, M.: A language modeling framework for expert finding. Inf. Proc. & Manag. 45(1), 1–19 (2009)
Balog, K., Bogers, T., van den Bosch, A., de Rijke, M.: Expertise profiling in a knowledge-intensive organization (submitted, 2010)
Bogers, T., Balog, K.: UvT Expert Collection documentation. Technical report, ILK Research Group Technical Report Series no. 07-06 (July 2007)
Brooks, T.A.: Topical subject expertise and the semantic distance model of relevance assessment. Journal of Documentation 51, 370–387 (1995)
Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: SIGIR 2008, pp. 659–666. ACM, New York (2008)
Dang, H.T., Lin, J., Kelly, D.: Overview of the TREC 2006 question answering track. In: TREC 2006. NIST (2006)
Hofmann, K., Tsagkias, E., Meij, E.J., de Rijke, M.: The impact of document structure on keyphrase extraction. In: CIKM 2009, Hong Kong. ACM, New York (November 2009)
Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst. 20(4), 422–446 (2002)
Jijkoun, V., de Rijke, M.: Overview of the WiQA task at CLEF 2006. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 265–274. Springer, Heidelberg (2007)
Lin, J., Demner-Fushman, D.: Will pyramids built of nuggets topple over. In: HLT 2006, Morristown, NJ, USA, pp. 383–390. Association for Computational Linguistics (2006)
Meij, E.J., Bron, M., Huurnink, B., Hollink, L., de Rijke, M.: Learning semantic query suggestions. In: Bernstein, A., Karger, D.R., Heath, T., Feigenbaum, L., Maynard, D., Motta, E., Thirunarayan, K. (eds.) ISWC 2009. LNCS, vol. 5823, Springer, Heidelberg (2009)
Voorhees, E.M.: Overview of the TREC 2004 question answering track. In: TREC 2004. NIST (2005)
Voorhees, E.M., Dang, H.T.: Overview of the TREC 2005 question answering track. In: TREC 2005. NIST (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
de Rijke, M., Balog, K., Bogers, T., van den Bosch, A. (2010). On the Evaluation of Entity Profiles. In: Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A. (eds) Multilingual and Multimodal Information Access Evaluation. CLEF 2010. Lecture Notes in Computer Science, vol 6360. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15998-5_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-15998-5_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15997-8
Online ISBN: 978-3-642-15998-5
eBook Packages: Computer ScienceComputer Science (R0)