A Framework to Build Games with a Purpose for Linked Data Refinement

  • Gloria Re Calegari
  • Andrea Fiano
  • Irene CelinoEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11137)


With the rise of linked data and knowledge graphs, the need becomes compelling to find suitable solutions to increase the coverage and correctness of datasets, to add missing knowledge and to identify and remove errors. Several approaches – mostly relying on machine learning and NLP techniques – have been proposed to address this refinement goal; they usually need a partial gold standard, i.e. some “ground truth” to train automatic models. Gold standards are manually constructed, either by involving domain experts or by adopting crowdsourcing and human computation solutions.

In this paper, we present an open source software framework to build Games with a Purpose for linked data refinement, i.e. web applications to crowdsource partial ground truth, by motivating user participation through fun incentive. We detail the impact of this new resource by explaining the specific data linking “purposes” supported by the framework (creation, ranking and validation of links) and by defining the respective crowdsourcing tasks to achieve those goals.

To show this resource’s versatility, we describe a set of diverse applications that we built on top of it; to demonstrate its reusability and extensibility potential, we provide references to detailed documentation, including an entire tutorial which in a few hours guides new adopters to customize and adapt the framework to a new use case.



This work was partially supported by the STARS4ALL project (H2020-688135) co-funded by the European Commission.


  1. 1.
    Acosta, M., Zaveri, A., Simperl, E., Kontokostas, D., Auer, S., Lehmann, J.: Crowdsourcing linked data quality assessment. In: Alani, H., Kagal, L., Fokoue, A., Groth, P., Biemann, C., Parreira, J.X., Aroyo, L., Noy, N., Welty, C., Janowicz, K. (eds.) ISWC 2013. LNCS, vol. 8219, pp. 260–276. Springer, Heidelberg (2013). Scholar
  2. 2.
    Brovelli, M.A., Celino, I., Fiano, A., Molinari, M.E., Venkatachalam, V.: A crowdsourcing-based game for land cover validation. Appl. Geomat. 10(1), 1–11 (2018)CrossRefGoogle Scholar
  3. 3.
    Celino, I., et al.: Urbanopoly - A social and location-based game with a purpose to crowdsource your urban data. In: International Conference on Privacy, Security, Risk and Trust (PASSAT), and International Confernece on Social Computing (SocialCom), pp. 910–913. IEEE (2012)Google Scholar
  4. 4.
    Celino, I., et al.: Linking smart cities datasets with human computation – the Case of UrbanMatch. In: Cudré-Mauroux, P. (ed.) ISWC 2012. LNCS, vol. 7650, pp. 34–49. Springer, Heidelberg (2012). Scholar
  5. 5.
    Celino, I., Della Valle, E., Gualandris, R.: On the effectiveness of a mobile puzzle game UI to crowdsource linked data management tasks. In: 1st International Workshop on User Interfaces for Crowdsourcing and Human Computation (2014)Google Scholar
  6. 6.
    Celino, I., Fiano, A., Fino, R.: Analysis of a cultural heritage game with a purpose with an educational incentive. In: Bozzon, A., Cudre-Maroux, P., Pautasso, C. (eds.) ICWE 2016. LNCS, vol. 9671, pp. 422–430. Springer, Cham (2016). Scholar
  7. 7.
    Chamberlain, J., Poesio, M., Kruschwitz, U.: Phrase detectives: a web-based collaborative annotation game. In: Proceedings of the International Conference on Semantic Systems (I-Semantics 2008), pp. 42–49 (2008)Google Scholar
  8. 8.
    Dong, X., et al.: Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 601–610. ACM (2014)Google Scholar
  9. 9.
    Fellegi, I.P., Sunter, A.B.: A theory for record linkage. J. Am. Stat. Assoc. 64(328), 1183–1210 (1969)CrossRefGoogle Scholar
  10. 10.
    Ferrara, A., Nikolov, A., Scharffe, F.: Data linking for the semantic web. In: Semantic Web: Ontology and Knowledge Base Enabled Tools, Services, and Applications, 169 (2013)Google Scholar
  11. 11.
    Fürber, C., Hepp, M.: Using SPARQL and SPIN for data quality management on the semantic web. In: Abramowicz, W., Tolksdorf, R. (eds.) BIS 2010. LNBIP, vol. 47, pp. 35–46. Springer, Heidelberg (2010). Scholar
  12. 12.
    Guéret, C., Groth, P., Stadler, C., Lehmann, J.: Assessing linked data mappings using network measures. In: Simperl, E., Cimiano, P., Polleres, A., Corcho, O., Presutti, V. (eds.) ESWC 2012. LNCS, vol. 7295, pp. 87–102. Springer, Heidelberg (2012). Scholar
  13. 13.
    Hees, J., Roth-Berghofer, T., Biedert, R., Adrian, B., Dengel, A.: BetterRelations: using a game to rate linked data triples. In: Bach, J., Edelkamp, S. (eds.) KI 2011. LNCS (LNAI), vol. 7006, pp. 134–138. Springer, Heidelberg (2011). Scholar
  14. 14.
    Howe, J.: The rise of crowdsourcing. Wired Mag. 14(6), 1–4 (2006)Google Scholar
  15. 15.
    Law, E., Ahn, L.V.: Human computation. Synth. Lect. Artif. Intell. Mach. Learn. 5(3), 1–121 (2011)CrossRefGoogle Scholar
  16. 16.
    Paulheim, H.: Knowledge graph refinement: a survey of approaches and evaluation methods. Semant. Web 8(3), 489–508 (2017)CrossRefGoogle Scholar
  17. 17.
    Paulheim, H., Bizer, C.: Improving the quality of linked data using statistical distributions. Int. J. Semant. Web Inf. Syst. (IJSWIS) 10(2), 63–86 (2014)CrossRefGoogle Scholar
  18. 18.
    Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 1403–1412. ACM (2011)Google Scholar
  19. 19.
    Re Calegari, G., Nasi, G., Celino, I.: Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification. Hum. Comput. J. 5(1), 13–30 (2018). Scholar
  20. 20.
    Settles, B.: Active learning. Synth. Lect. Artif. Intell. Mach. Learn. 6(1), 1–114 (2012)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Simperl, E., Norton, B., Vrandečić, D.: Crowdsourcing tasks in linked data management. In: Proceedings of the Second International Conference on Consuming Linked Data, Vol. 782, pp. 61–72. CEUR-WS. org (2011)Google Scholar
  22. 22.
    Siorpaes, K., Hepp, M.: Games with a purpose for the semantic web. IEEE Intell. Syst., 23(3) (2008)CrossRefGoogle Scholar
  23. 23.
    Sleeman, J., Finin, T.: Type prediction for efficient coreference resolution in heterogeneous semantic graphs. In: IEEE Seventh International Conference on Semantic Computing (ICSC), pp. 78–85. IEEE (2013)Google Scholar
  24. 24.
    Sleeman, J., Finin, T., Joshi, A.: Topic Modeling for RDF Graphs. In: LD4IE at ISWC, pp. 48–62 (2015)Google Scholar
  25. 25.
    Thaler, S., Simperl, E.P.B., Siorpaes, K.: SpotTheLink: a game for ontology alignment. Wissensmanagement 182, 246–253 (2011)Google Scholar
  26. 26.
    Ul Hassan, U., O’Riain, S., Curry, E.: Effects of expertise assessment on the quality of task routing in human computation. In: Proceedings of the 2nd International Workshop on Social Media for Crowdsourcing and Human Computation, Paris, France (2013)Google Scholar
  27. 27.
    Von Ahn, L., Dabbish, L.: Designing games with a purpose. Commun. ACM 51(8), 58–67 (2008)Google Scholar
  28. 28.
    Waitelonis, J., Ludwig, N., Knuth, M., Sack, H.: WhoKnows? evaluating linked data heuristics with a quiz that cleans up DBpedia. Interact. Technol. Smart Educ. 8(4), 236–248 (2011)CrossRefGoogle Scholar
  29. 29.
    Wieser, C., Bry, F., Bérard, A., Lagrange, R.: ARTigo: building an artwork search engine with games and higher-order latent semantic analysis. In: First AAAI Conference on Human Computation and Crowdsourcing (2013)Google Scholar
  30. 30.
    Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., Auer, S.: Quality assessment for linked data: a survey. Semant. Web 7(1), 63–93 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Gloria Re Calegari
    • 1
  • Andrea Fiano
    • 1
  • Irene Celino
    • 1
    Email author
  1. 1.CefrielMilanoItaly

Personalised recommendations