Evaluating Entity Summarization Using a Game-Based Ground Truth

  • Andreas Thalhammer
  • Magnus Knuth
  • Harald Sack
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7650)


In recent years, strategies for Linked Data consumption have caught attention in Semantic Web research. For direct consumption by users, Linked Data mashups, interfaces, and visualizations have become a popular research area. Many approaches in this field aim to make Linked Data interaction more user friendly to improve its accessibility for non-technical users. A subtask for Linked Data interfaces is to present entities and their properties in a concise form. In general, these summaries take individual attributes and sometimes user contexts and preferences into account. But the objective evaluation of the quality of such summaries is an expensive task. In this paper we introduce a game-based approach aiming to establish a ground truth for the evaluation of entity summarization. We exemplify the applicability of the approach by evaluating two recent summarization approaches.


entity summarization property ranking evaluation linked data games with a purpose 


  1. 1.
    Amigó, E., Gonzalo, J., Peñas, A., Verdejo, F.: Qarla: a framework for the evaluation of text summarization systems. In: Proc. of the 43rd Annual Meeting on Association for Computational Linguistics, ACL 2005, pp. 280–289. Association for Computational Linguistics, Stroudsburg (2005)CrossRefGoogle Scholar
  2. 2.
    Cantador, I., Brusilovsky, P., Kuflik, T.: 2nd ws. on information heterogeneity and fusion in recommender systems (hetrec 2011). In: Proc. of 5th ACM Conf. on Recommender systems, RecSys 2011. ACM, New York (2011)Google Scholar
  3. 3.
    Cheng, G., Tran, T., Qu, Y.: RELIN: Relatedness and Informativeness-Based Centrality for Entity Summarization. In: Aroyo, L., Welty, C., Alani, H., Taylor, J., Bernstein, A., Kagal, L., Noy, N., Blomqvist, E. (eds.) ISWC 2011, Part I. LNCS, vol. 7031, pp. 114–129. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  4. 4.
    comScore. comscore reports global search market growth of 46 percent in 2009 (2010),
  5. 5.
    Dunning, T.: Accurate methods for the statistics of surprise and coincidence. Computational Linguistics 19(1), 61–74 (1993)Google Scholar
  6. 6.
    Luhn, H.P.: The automatic creation of literature abstracts. IBM J. Res. Dev. 2(2), 159–165 (1958)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Item-based collaborative filtering recommendation algorithms. In: Proc. of the 10th Int. Conf. on World Wide Web, WWW 2001, pp. 285–295. ACM, New York (2001)Google Scholar
  8. 8.
    Singhal, A.: Introducing the knowledge graph: things, not strings (2012),
  9. 9.
    Thalhammer, A., Toma, I., Roa-Valverde, A.J., Fensel, D.: Leveraging usage data for linked data movie entity summarization. In: Proc. of the 2nd Int. Ws. on Usage Analysis and the Web of Data (USEWOD 2012) co-located with WWW 2012, Lyon, France, vol. abs/1204.2718 (2012)Google Scholar
  10. 10.
    Waitelonis, J., Ludwig, N., Knuth, M., Sack, H.: WhoKnows? – evaluating linked data heuristics with a quiz that cleans up DBpedia. Int. Journal of Interactive Technology and Smart Education (ITSE) 8(3), 236–248 (2011)CrossRefGoogle Scholar
  11. 11.
    Waitelonis, J., Sack, H.: Towards exploratory video search using linked data. Multimedia Tools and Applications 59, 645–672 (2012), 10.1007/s11042-011-0733-1CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Andreas Thalhammer
    • 1
  • Magnus Knuth
    • 2
  • Harald Sack
    • 2
  1. 1.University of InnsbruckInnsbruckAustria
  2. 2.Hasso Plattner Institute PotsdamPotsdamGermany

Personalised recommendations