Synthesizing Knowledge Graphs for Link and Type Prediction Benchmarking

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10249)


Despite the growing amount of research in link and type prediction in knowledge graphs, systematic benchmark datasets are still scarce. In this paper, we propose a synthesis model for the generation of benchmark datasets for those tasks. Synthesizing data is a way of having control over important characteristics of the data, and allows the study of the impact of such characteristics on the performance of different methods. The proposed model uses existing knowledge graphs to create synthetic graphs with similar characteristics, such as distributions of classes, relations, and instances. As a first step, we replicate already existing knowledge graphs in order to validate the synthesis model. To do so, we perform extensive experiments with different link and type prediction methods. We show that we can systematically create knowledge graph benchmarks which allow for quantitative measurements of the result quality and scalability of link and type prediction methods.


Knowledge graphs Link prediction Type prediction Benchmarking 



The work presented in this paper has been partly supported by the Ministry of Science, Research and the Arts Baden-Württemberg in the project SyKo\(^2\)W\(^2\) (Synthesis of Completion and Correction of Knowledge Graphs on the Web).


  1. 1.
    Albuquerque, G., Löwe, T., Magnor, M.: Synthetic generation of high-dimensional datasets. IEEE Trans. Vis. Comput. Graph. 17(12), 2317–2324 (2011). (TVCG, Proc. Visualization / InfoVis)CrossRefGoogle Scholar
  2. 2.
    Angles, R., Boncz, P., Larriba-Pey, J., Fundulaki, I., Neumann, T., Erling, O., Neubauer, P., Martinez-Bazan, N., Kotsev, V., Toma, I.: The linked data benchmark council: a graph and rdf industry benchmarking effort. SIGMOD Rec. 43(1), 27–31 (2014)CrossRefGoogle Scholar
  3. 3.
    Palmero Aprosio, A., Giuliano, C., Lavelli, A.: Automatic expansion of DBpedia exploiting wikipedia cross-language information. In: Cimiano, P., Corcho, O., Presutti, V., Hollink, L., Rudolph, S. (eds.) ESWC 2013. LNCS, vol. 7882, pp. 397–411. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38288-8_27CrossRefGoogle Scholar
  4. 4.
    Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 26, pp. 2787–2795. Curran Associates, Inc. (2013)Google Scholar
  5. 5.
    Chawla, S., Gionis, A.: k-means-: a unified approach to clustering and outlier detection. In: Proceedings of the 13th SIAM International Conference on Data Mining, Austin, Texas, USA, pp. 189–197. SIAM (2013)CrossRefGoogle Scholar
  6. 6.
    Cheatham, M., Dragisic, Z., Euzenat, J., Faria, D., Ferrara, A., Flouris, G., Fundulaki, I., Granada, R., Ivanova, V., Jiménez-Ruiz, E., et al.: Results of the ontology alignment evaluation initiative 2015. In: 10th ISWC Workshop on Ontology Matching (OM), pp. 60–115 (2015)Google Scholar
  7. 7.
    Clare, A., King, R.D.: Knowledge discovery in multi-label phenotype data. In: Raedt, L., Siebes, A. (eds.) PKDD 2001. LNCS (LNAI), vol. 2168, pp. 42–53. Springer, Heidelberg (2001). doi: 10.1007/3-540-44794-6_4CrossRefzbMATHGoogle Scholar
  8. 8.
    van Erp, M., Mendes, P., Paulheim, H., Ilievski, F., Plu, J., Rizzo, G., Waitelonis, J.: Evaluating entity linking: an analysis of current benchmark datasets and a roadmap for doing a better job. In: Proceedings of the Language Resources and Evaluation Conference, ELRA (2016)Google Scholar
  9. 9.
    Galárraga, L.A., Teflioudi, C., Hose, K., Suchanek, F.M.: AMIE: association rule mining under incomplete evidence in ontological knowledge bases. In: WWW 2013, Rio de Janeiro, Brazil, pp. 413–422. ACM (2013)Google Scholar
  10. 10.
    Gangemi, A., Nuzzolese, A.G., Presutti, V., Draicchio, F., Musetti, A., Ciancarini, P.: Automatic typing of DBpedia entities. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012. LNCS, vol. 7649, pp. 65–81. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-35176-1_5CrossRefGoogle Scholar
  11. 11.
    Goethals, B., Bussche, J.: Relational association rules: getting Warmer. In: Hand, D.J., Adams, N.M., Bolton, R.J. (eds.) Pattern Detection and Discovery. LNCS (LNAI), vol. 2447, pp. 125–139. Springer, Heidelberg (2002). doi: 10.1007/3-540-45728-3_10CrossRefGoogle Scholar
  12. 12.
    Guo, Y., Pan, Z., Heflin, J.: LUBM: a benchmark for owl knowledge base systems. Web Semant. 3(2–3), 158–182 (2005)CrossRefGoogle Scholar
  13. 13.
    Lao, N., Cohen, W.W.: Relational retrieval using a combination of path-constrained random walks. Mach. Learn. 81(1), 53–67 (2010)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Lehmann, J.: Dl-learner: learning concepts in description logics. J. Mach. Learn. Res. 10, 2639–2642 (2009)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Lopez, V., Unger, C., Cimiano, P., Motta, E.: Evaluating question answering over linked data. Web Semant. Sci. Serv. Agents World Wide Web 21, 3–13 (2013)CrossRefGoogle Scholar
  16. 16.
    Melo, A., Paulheim, H., Völker, J.: Type prediction in RDF knowledge bases using hierarchical multilabel classification. In: Proceedings of the International Conference on Web Intelligence, Mining and Semantics, WIMS 2016, Nîmes, France, pp. 14:1–14:10 (2016)Google Scholar
  17. 17.
    Melo, A., Theobald, M., Völker, J.: Correlation-based refinement of rules with numerical attributes. In: Proceedings of the International Florida Artificial Intelligence Research Society Conference, FLAIRS, Pensacola, Florida (2014).
  18. 18.
    Morsey, M., Lehmann, J., Auer, S., Ngonga Ngomo, A.-C.: DBpedia SPARQL benchmark – performance assessment with real queries on real data. In: Aroyo, L., Welty, C., Alani, H., Taylor, J., Bernstein, A., Kagal, L., Noy, N., Blomqvist, E. (eds.) ISWC 2011. LNCS, vol. 7031, pp. 454–469. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-25073-6_29CrossRefGoogle Scholar
  19. 19.
    Muggleton, S.: Learning from positive data. In: Muggleton, S. (ed.) ILP 1996. LNCS, vol. 1314, pp. 358–376. Springer, Heidelberg (1997). doi: 10.1007/3-540-63494-0_65CrossRefzbMATHGoogle Scholar
  20. 20.
    Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. Proc. IEEE 104(1), 11–33 (2016)CrossRefGoogle Scholar
  21. 21.
    Nickel, M., Rosasco, L., Poggio, T.A.: Holographic embeddings of knowledge graphs. CoRR abs/1510.04935 (2015)Google Scholar
  22. 22.
    Nickel, M., Tresp, V., Kriegel, H.P.: A three-way model for collective learning on multi-relational data. In: Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 809–816. ACM (2011)Google Scholar
  23. 23.
    Nuzzolese, A.G., Gangemi, A., Presutti, V., Ciancarini, P.: Type inference through the analysis of Wikipedia links. In: WWW 2012 Workshop on Linked Data on the Web, Lyon, France. CEUR Workshop Proceedings, vol. 937 (2012)Google Scholar
  24. 24.
    Paulheim, H.: Knowledge graph refinement: a survey of approaches and evaluation methods. Semant. Web 8(3), 489–508 (2017)CrossRefGoogle Scholar
  25. 25.
    Paulheim, H., Bizer, C.: Type inference on noisy RDF data. In: Alani, H., et al. (eds.) ISWC 2013. LNCS, vol. 8218, pp. 510–525. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-41335-3_32CrossRefGoogle Scholar
  26. 26.
    Paulheim, H., Bizer, C.: Improving the quality of linked data using statistical distributions. Int. J. Semant. Web Inf. Syst. 10(2), 63–86 (2014)CrossRefGoogle Scholar
  27. 27.
    Raedt, L., Frasconi, P., Kersting, K., Muggleton, S. (eds.): Probabilistic Inductive Logic Programming. LNCS (LNAI), vol. 4911. Springer, Heidelberg (2008)zbMATHGoogle Scholar
  28. 28.
    Ristoski, P., Vries, G.K.D., Paulheim, H.: A collection of benchmark datasets for systematic evaluations of machine learning on the semantic web. In: Groth, P., Simperl, E., Gray, A., Sabou, M., Krötzsch, M., Lecue, F., Flöck, F., Gil, Y. (eds.) ISWC 2016. LNCS, vol. 9982, pp. 186–194. Springer, Cham (2016). doi: 10.1007/978-3-319-46547-0_20CrossRefGoogle Scholar
  29. 29.
    Samadi, B., Cipolone, A., Lin, P.J., Xiao, R., Jeske, D.R., Holt, D., Rend, C., Cox, S.: Development of a synthetic data set generator for building and testing information discovery systems. In: Third International Conference on Information Technology, pp. 707–712 (2006)Google Scholar
  30. 30.
    Schmidt, M., Hornung, T., Lausen, G., Pinkel, C.: Sp2bench: a SPARQL performance benchmark. CoRR abs/0806.4627 (2008)Google Scholar
  31. 31.
    Theodoridis, Y., Nascimento, M.A.: Generating spatiotemporal datasets on the WWW. SIGMOD Rec. 29(3), 39–43 (2000)CrossRefGoogle Scholar
  32. 32.
    Yosef, M.A., Bauer, S., Hoffart, J., Spaniol, M., Weikum, G.: HYENA: hierarchical type classification for entity names. In: COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Posters, Mumbai, India, pp. 1361–1370 (2012)Google Scholar
  33. 33.
    Zhang, M.L., Zhou, Z.H.: Multilabel neural networks with applications to functional genomics and text categorization. IEEE Trans. Knowl. Data Eng. 18(10), 1338–1351 (2006)CrossRefGoogle Scholar
  34. 34.
    Zhang, M.L., Zhou, Z.H.: ML-KNN: a lazy learning approach to multi-label learning. Pattern Recogn. 40(7), 2038–2048 (2007)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of MannheimMannheimGermany

Personalised recommendations