Abstract
Knowledge graphs have become much large and complex during past several years due to its wide applications in knowledge discovery. Many knowledge graphs were built using automated construction tools and via crowdsourcing. The graph may contain significant amount of syntax and semantics errors that great impact its quality. A low quality knowledge graph produce low quality application that is built on it. Therefore, evaluating quality of knowledge graph is necessary for building high quality applications. Many frameworks were proposed for systematic evaluation of knowledge graphs, but they are either too complex to be practical or lacking of scalability to large scale knowledge graphs. In this paper, we conducted a comprehensive study of existing frameworks and proposed a practical framework for evaluating quality on “fit for purpose” of knowledge graphs. We first selected a set of quality dimensions and their corresponding metrics based on the requirements of knowledge discovery based on knowledge graphs through systematic investigation of representative published applications. Then we recommended an approach for evaluating each metric considering its feasibility and scalability. The framework can be used for checking the essential quality requirements of knowledge graphs for serving the purpose of knowledge discovery.
This research is partially supported by United States NSF award #1852249, China NSSFC 2019–2022 project 19BTQ075.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, Y., Kuang, J., Cheng, D., Zheng, J., Gao, M., Zhou, A.: AgriKG: an agricultural knowledge graph and its applications. In: Li, G., Yang, J., Gama, J., Natwichai, J., Tong, Y. (eds.) DASFAA 2019. LNCS, vol. 11448, pp. 533–537. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-18590-9_81
Deng, S., Zhang, N., Zhang, W., Chen, J., Pan, J.Z., Chen, H.: Knowledge-driven stock trend prediction and explanation via temporal convolutional network. In: Companion Proceedings of The 2019 World Wide Web Conference, WWW 2019, pp. 678–685. ACM, New York (2019). https://doi.org/10.1145/3308560.3317701
Dietz, L., Kotov, A., Meij, E.: Utilizing knowledge graphs for text-centric information retrieval. In: The 41st International ACM SIGIR Conference on Research & #38; Development in Information Retrieval, SIGIR 2018, pp. 1387–1390. ACM, New York (2018). https://doi.org/10.1145/3209978.3210187
Freebase: Data Dumps, June 2013. https://developers.google.com/freebase/
Gao, J., Li, X., Xu, Y.E., Sisman, B., Dong, X.L., Yang, J.: Efficient knowledge graph accuracy evaluation. Technical Report (2019). https://users.cs.duke.edu/~jygao/KG_eval_vldb_full.pdf. Accessed 25 May 2019
Gao, Y., Liang, J., Han, B., Yakout, M., Mohamed, A.: Building a large-scale, accurate and fresh knowledge graph. In: KDD-2018, Tutorial T39, August 2018. https://kdd2018tutorialt39.azurewebsites.net/
Hixon, B., Clark, P., Hajishirzi, H.: Learning knowledge graphs for question answering through conversational dialog. In: HLT-NAACL (2015)
Huang, X., Zhang, J., Li, D., Li, P.: Knowledge graph embedding based question answering. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, pp. 105–113. ACM, New York (2019). https://doi.org/10.1145/3289600.3290956
Kontokostas, D., et al.: Test-driven evaluation of linked data quality. In: Proceedings of the 23rd International Conference on World Wide Web, WWW 2014, pp. 747–758 (2014)
Lukovnikov, D., Fischer, A., Lehmann, J., Auer, S.: Neural network-based question answering over knowledge graphs on word and character level. In: Proceedings of the 26th WWW, pp. 1211–1220 (2017). https://doi.org/10.1145/3038912.3052675
Keller, R.M.: Building a knowledge graph for the air traffic management community. In: Companion Proceedings of The 2019 WWW, pp. 700–704 (2019). https://doi.org/10.1145/3308560.3317706
Muppalla, R., Lalithsena, S., Banerjee, T., Sheth, A.: A knowledge graph framework for detecting traffic events using stationary cameras. In: Proceedings of WebSci 2017, pp. 431–436 (2017). https://doi.org/10.1145/3091478.3162384
Ostapuk, N., Yang, J., Cudre-Mauroux, P.: ActiveLink: deep active learning for link prediction in knowledge graphs. In: The World Wide Web Conference, WWW 2019, pp. 1398–1408. ACM, New York (2019). https://doi.org/10.1145/3308558.3313620
Paulheim, H.: Knowledge graph refinement: a survey of approaches and evaluation methods. Semant. Web 8, 489–508 (2016)
Ping, P., Watson, K., Han, J., Bui, A.: Individualized knowledge graph: a viable informatics path to precision medicine. Circ. Res. 120(7), 1078–1080 (2017). https://doi.org/10.1161/CIRCRESAHA.116.310024
Radhakrishnan, P., Talukdar, P., Varma, V.: ELDEN: improved entity linking using densified knowledge graphs. In: Proceedings of the 2018 Conference of NACCAL: Human Language Technologies, Volume 1, pp. 1844–1853, June 2018. https://doi.org/10.18653/v1/N18-1167
Singhal, A.: Introducing the knowledge graph: things, not strings. Google Blog, August 2012. https://www.blog.google/products/search/introducing-knowledge-graph-things-not/
Stvilia, B., Gasser, L., Twidale, M.B., Smith, L.C.: A framework for information quality assessment. J. Am. Soc. Inf. Sci. Technol. 58(12), 1720–1733 (2007)
Wang, H., et al.: Ripple network: propagating user preferences on the knowledge graph for recommender systems. CoRR abs/1803.03467 (2018). http://arxiv.org/abs/1803.03467
Wang, H., et al.: Knowledge graph convolutional networks for recommender systems with label smoothness regularization. arXiv preprint arXiv:1905.04413 (2019)
Wang, R., et al.: AceKG: a large-scale knowledge graph for academic data mining. CoRR abs/1807.08484 (2018), http://arxiv.org/abs/1807.08484
Wang, R.Y., Strong, D.M.: Beyond accuracy: what data quality means to data consumers. J. Manag. Inf. Syst. 12, 5–33 (1996)
Zaveri, A., et al.: User-driven quality evaluation of DBpedia. In: Proceedings of the 9th International Conference on Semantic Systems, I-SEMANTICS 2013, pp. 97–104 (2013)
Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., Auer, S.: Quality assessment for linked data: a survey. Semant. Web 7, 63–93 (2016)
Zhang, Y., Dai, H., Kozareva, Z., Smola, A.J., Song, L.: Variational reasoning for question answering with knowledge graph. CoRR abs/1709.04071 (2017). http://arxiv.org/abs/1709.04071
Zhao, Q., Li, Q., Wen, J.: Construction and application research of knowledge graph in aviation risk field. In: MATEC Web of Conferences, vol. 151, p. 05003, January 2018. https://doi.org/10.1051/matecconf/201815105003
Zheng, W., Yu, J.X., Zou, L., Cheng, H.: Question answering over knowledge graphs: question understanding via template decomposition. Proc. VLDB Endow. 11(11), 1373–1386 (2018). https://doi.org/10.14778/3236187.3236192
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Chen, H., Cao, G., Chen, J., Ding, J. (2019). A Practical Framework for Evaluating the Quality of Knowledge Graph. In: Zhu, X., Qin, B., Zhu, X., Liu, M., Qian, L. (eds) Knowledge Graph and Semantic Computing: Knowledge Computing and Language Understanding. CCKS 2019. Communications in Computer and Information Science, vol 1134. Springer, Singapore. https://doi.org/10.1007/978-981-15-1956-7_10
Download citation
DOI: https://doi.org/10.1007/978-981-15-1956-7_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1955-0
Online ISBN: 978-981-15-1956-7
eBook Packages: Computer ScienceComputer Science (R0)