Skip to main content
Book cover

European Semantic Web Conference

ESWC 2022: The Semantic Web pp 217–235Cite as

Improving Question Answering Quality Through Language Feature-Based SPARQL Query Candidate Validation

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13261)

Abstract

Question Answering systems are on the rise and on their way to become one of the standard user interfaces. However, in conversational user interfaces, the information quantity needs to be kept low as users expect a limited number of precise answers (often it is 1) – similar to human-human communication. The acceptable number of answers in a result list is a key differentiator from search engines where showing more answers (10–100) to the user is widely accepted. Hence, the quality of Question Answering is crucial for the wide acceptance of such systems. The adaptation of natural-language user interfaces for satisfying the information needs of humans requires high-quality and not-redundant answers. However, providing compact and correct answers to the users’ questions is a challenging task. In this paper, we consider a certain class of Question Answering systems that work over Knowledge Graphs. We developed a system-agnostic approach for optimizing the ranked lists of SPARQL query candidates produced by the Knowledge Graph Question Answering system that are used to retrieve an answer to a given question. We call this a SPARQL query validation process. For the evaluation of our approach, we used two well-known Knowledge Graph Question Answering benchmarks. Our results show a significant improvement in the Question Answering quality. As the approach is system-agnostic, it can be applied to any Knowledge Graph Question Answering system that produces query candidates.

Keywords

  • Question Answering over Knowledge Graphs
  • Query Validation
  • Query Candidate Filtering

A. Gashkov and A. Perevalov—Shared first authorship–these authors contributed equally to this work.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-06981-9_13
  • Chapter length: 19 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   69.99
Price excludes VAT (USA)
  • ISBN: 978-3-031-06981-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   89.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.

Notes

  1. 1.

    https://www.w3.org/TR/rdf-sparql-query/.

  2. 2.

    https://doi.org/10.6084/m9.figshare.19434515.

  3. 3.

    Measuring the impact on the verbalization regarding the QV results would be part of additional research.

  4. 4.

    Example: Assuming a user asks for the red or green wire to be cut for defusing a bomb, then a guessed answer by the QA system might have a devastating result in real life.

  5. 5.

    A query candidate is considered as correct if \(F1 \; score(y_{pred}, y_{true}) = 1\), where \(y_{pred}\) – is the set of answers obtained with query candidate and \(y_{true}\) is the “gold standard” answer set.

  6. 6.

    https://www.w3.org/TR/turtle/.

  7. 7.

    BERT was consciously trained to understand relationships between two consecutive sentences if the second sentence follows the first one (e.g., “[CLS] the man went to [MASK] store [SEP] he bought a gallon [MASK] milk [SEP]) because many important downstream tasks such as QA and Natural Language Inference (NLI) are based on understanding the relationship between two sentences” [11].

  8. 8.

    The trained Query Validators are available online;

    LC-QuAD: https://huggingface.co/perevalov/query-validation-lcquad,

    RuBQ: https://huggingface.co/perevalov/query-validation-rubq.

  9. 9.

    In our previous study, we already compared QV ’s quality using different query candidate verbalization methods [19].

  10. 10.

    http://gerbil-qa.aksw.org/gerbil/, version 0.2.3.

  11. 11.

    Our results are available online. LC-QuAD 2.0:

    http://gerbil-qa.aksw.org/gerbil/experiment?id=202112080001 and

    http://gerbil-qa.aksw.org/gerbil/experiment?id=202112080002;

    RuBQ 2.0: http://gerbil-qa.aksw.org/gerbil/experiment?id=202112090005 and

    http://gerbil-qa.aksw.org/gerbil/experiment?id=202112090006.

References

  1. Abdiansah, A., Azhari, A., Sari, A.K.: Survey on answer validation for Indonesian question answering system (IQAS). Int. J. Intell. Syst. Appl. 10, 68–78 (2018). https://doi.org/10.5815/ijisa.2018.04.08

    CrossRef  Google Scholar 

  2. Asai, A., Choi, E.: Challenges in information seeking QA: unanswerable questions and paragraph retrieval. arXiv preprint arXiv:2010.11915 (2020)

  3. Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.: DBpedia: a nucleus for a web of open data. In: Aberer, K., et al. (eds.) ASWC/ISWC-2007. LNCS, vol. 4825, pp. 722–735. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-76298-0_52

  4. Azmy, M., Shi, P., Lin, J., Ilyas, I.: Farewell freebase: migrating the simplequestions dataset to DBpedia. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 2093–2103 (2018)

    Google Scholar 

  5. Babych, S., Henn, A., Pawellek, J., Padó, S.: Dependency-based answer validation for German. In: Petras, V., Forner, P., Clough, P.D. (eds.) CLEF 2011 Labs and Workshop, Notebook Papers, 19–22 September 2011, Amsterdam, The Netherlands. CEUR Workshop Proceedings, vol. 1177. CEUR-WS.org (2011)

    Google Scholar 

  6. Berners-Lee, T., Hendler, J., Lassila, O.: The semantic web. Sci. Am. 284(5), 34–43 (2001)

    CrossRef  Google Scholar 

  7. Biswas, D., Dubey, M., Rony, M.R.A.H., Lehmann, J.: VANiLLa: verbalized answers in natural language at large scale. CoRR abs/2105.11407 (2021)

    Google Scholar 

  8. Both, A., Diefenbach, D., Singh, K., Shekarpour, S., Cherix, D., Lange, C.: Qanary – a methodology for vocabulary-driven open question answering systems. In: Sack, H., Blomqvist, E., d’Aquin, M., Ghidini, C., Ponzetto, S.P., Lange, C. (eds.) ESWC 2016. LNCS, vol. 9678, pp. 625–641. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-34129-3_38

    CrossRef  Google Scholar 

  9. Both, A., Gashkov, A., Eltsova, M.: Similarity detection of natural-language questions and answers using the VANiLLa dataset. J. Phys: Conf. Ser. 1886(1), 012017 (2021). https://doi.org/10.1088/1742-6596/1886/1/012017

    CrossRef  Google Scholar 

  10. Burtsev, M., et al.: DeepPavlov: open-source library for dialogue systems. In: Proceedings of ACL 2018, System Demonstrations, pp. 122–127. Association for Computational Linguistics, Melbourne (2018)

    Google Scholar 

  11. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Long and Short Papers), vol. 1, pp. 4171–4186. Association for Computational Linguistics, Minneapolis (2019). https://doi.org/10.18653/v1/N19-1423

  12. Diefenbach, D., Both, A., Singh, K., Maret, P.: Towards a question answering system over the semantic web. Semantic Web 11, 421–439 (2020)

    CrossRef  Google Scholar 

  13. Diefenbach, D., Giménez-García, J., Both, A., Singh, K., Maret, P.: QAnswer KG: designing a portable question answering system over RDF data. In: Harth, A., Kirrane, S., Ngonga Ngomo, A.-C., Paulheim, H., Rula, A., Gentile, A.L., Haase, P., Cochez, M. (eds.) ESWC 2020. LNCS, vol. 12123, pp. 429–445. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49461-2_25

    CrossRef  Google Scholar 

  14. Diefenbach, D., Lopez, V., Singh, K., Maret, P.: Core techniques of question answering systems over knowledge bases: a survey. Knowl. Inf. Syst. 55(3), 529–569 (2017). https://doi.org/10.1007/s10115-017-1100-y

    CrossRef  Google Scholar 

  15. Diefenbach, D., Migliatti, P.H., Qawasmeh, O., Lully, V., Singh, K., Maret, P.: QAnswer: a question answering prototype bridging the gap between a considerable part of the LOD cloud and end-users. In: Liu, L., et al. (eds.) The World Wide Web Conference, WWW 2019, San Francisco, May 13–17, 2019, pp. 3507–3510. ACM (2019). https://doi.org/10.1145/3308558.3314124

  16. Dimitrakis, E., Sgontzos, K., Tzitzikas, Y.: A survey on question answering systems over linked data and documents. J. Intell. Inf. Syst. 55(2), 233–259 (2019). https://doi.org/10.1007/s10844-019-00584-7

    CrossRef  Google Scholar 

  17. Dubey, M., Banerjee, D., Abdelkawi, A., Lehmann, J.: LC-QuAD 2.0: a large dataset for complex question answering over Wikidata and DBpedia. In: Ghidini, C., Hartig, O., Maleshkova, M., Svátek, V., Cruz, I., Hogan, A., Song, J., Lefrançois, M., Gandon, F. (eds.) ISWC 2019. LNCS, vol. 11779, pp. 69–78. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30796-7_5

    CrossRef  Google Scholar 

  18. Erxleben, F., Günther, M., Krötzsch, M., Mendez, J., Vrandečić, D.: Introducing Wikidata to the linked data web. In: Mika, P., Tudorache, T., Bernstein, A., Welty, C., Knoblock, C., Vrandečić, D., Groth, P., Noy, N., Janowicz, K., Goble, C. (eds.) ISWC 2014. LNCS, vol. 8796, pp. 50–65. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11964-9_4

    CrossRef  Google Scholar 

  19. Gashkov, A., Perevalov, A., Eltsova, M., Both, A.: Improving the question answering quality using answer candidate filtering based on natural-language features. In: 16th International Conference on Intelligent Systems and Knowledge Engineering (ISKE 2021) (2021)

    Google Scholar 

  20. Godin, F., Kumar, A., Mittal, A.: Learning when not to answer: a ternary reward structure for reinforcement learning based question answering. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Industry Papers), Vol. 2, pp. 122–129. Association for Computational Linguistics, Minneapolis (2019). https://doi.org/10.18653/v1/N19-2016

  21. Gómez-Adorno, H., Pinto, D., Vilariño, D.: A question answering system for reading comprehension tests. In: Carrasco-Ochoa, J.A., Martínez-Trinidad, J.F., Rodríguez, J.S., di Baja, G.S. (eds.) MCPR 2013. LNCS, vol. 7914, pp. 354–363. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38989-4_36

    CrossRef  Google Scholar 

  22. Grappy, A., Grau, B., Falco, M., Ligozat, A., Robba, I., Vilnat, A.: Selecting answers to questions from web documents by a robust validation process. In: 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, vol. 1, pp. 55–62 (2011). https://doi.org/10.1109/WI-IAT.2011.210

  23. Hu, M., Wei, F., Peng, Y., Huang, Z., Yang, N., Li, D.: Read+verify: machine reading comprehension with unanswerable questions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6529–6537 (2019)

    Google Scholar 

  24. Korablinov, V., Braslavski, P.: RuBQ: a Russian dataset for question answering over Wikidata. In: Pan, J.Z., Tamma, V., d’Amato, C., Janowicz, K., Fu, B., Polleres, A., Seneviratne, O., Kagal, L. (eds.) ISWC 2020. LNCS, vol. 12507, pp. 97–110. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62466-8_7

    CrossRef  Google Scholar 

  25. Lin, Y., Zhang, M., Zhang, R., Zou, L.: Deep-gAnswer: a knowledge based question answering system. In: U, L.H., Spaniol, M., Sakurai, Y., Chen, J. (eds.) APWeb-WAIM 2021. LNCS, vol. 12859, pp. 434–439. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85899-5_33

    CrossRef  Google Scholar 

  26. Magnini, B., Negri, M., Prevete, R., Tanev, H.: Is it the right answer? Exploiting web redundancy for answer validation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 425–432. Association for Computational Linguistics, Philadelphia (2002). https://doi.org/10.3115/1073083.1073154

  27. Maheshwari, G., Trivedi, P., Lukovnikov, D., Chakraborty, N., Fischer, A., Lehmann, J.: Learning to rank query graphs for complex question answering over knowledge graphs. In: Ghidini, C., Hartig, O., Maleshkova, M., Svátek, V., Cruz, I., Hogan, A., Song, J., Lefrançois, M., Gandon, F. (eds.) ISWC 2019. LNCS, vol. 11778, pp. 487–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30793-6_28

    CrossRef  Google Scholar 

  28. Miller, G.A.: WordNet: An Electronic Lexical Database. MIT Press (1998)

    Google Scholar 

  29. Napolitano, G., Usbeck, R., Ngomo, A.-C.N.: The scalable question answering over linked data (SQA) challenge 2018. In: Buscaldi, D., Gangemi, A., Reforgiato Recupero, D. (eds.) SemWebEval 2018. CCIS, vol. 927, pp. 69–75. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00072-1_6

    CrossRef  Google Scholar 

  30. Pakray, P., Barman, U., Bandyopadhyay, S., Gelbukh, A.: Semantic answer validation using universal networking language. Int. J. Comput. Sci. Inf. Technol. 3(4), 4927–4932 (2012)

    Google Scholar 

  31. Parikh, A.P., Täckström, O., Das, D., Uszkoreit, J.: A decomposable attention model for natural language inference. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2249–2255. Association for Computational Linguistics (2016)

    Google Scholar 

  32. Pellissier Tanon, T., de Assunção, M.D., Caron, E., Suchanek, F.M.: Demoing Platypus – a multilingual question answering platform for Wikidata. In: Gangemi, A., et al. (eds.) ESWC 2018. LNCS, vol. 11155, pp. 111–116. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98192-5_21

  33. Powers, D.M.W.: Evaluation: from precision, recall and F-factor to ROC, informedness, markedness & correlation. J. Mach. Learn. Technol. 2(1), 37–63 (2011)

    MathSciNet  Google Scholar 

  34. Rodrigo, A., Pérez-Iglesias, J., Peñas, A., Garrido, G., Araujo, L.: A question answering system based on information retrieval and validation. In: CLEF 2010 LABs and Workshops, Notebook Papers (2010)

    Google Scholar 

  35. Rybin, I., Korablinov, V., Efimov, P., Braslavski, P.: RuBQ 2.0: an innovated Russian question answering dataset. In: Verborgh, R., Hose, K., Paulheim, H., Champin, P.-A., Maleshkova, M., Corcho, O., Ristoski, P., Alam, M. (eds.) ESWC 2021. LNCS, vol. 12731, pp. 532–547. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77385-4_32

    CrossRef  Google Scholar 

  36. Saha, A., Pahuja, V., Khapra, M.M., Sankaranarayanan, K., Chandar, S.: Complex sequential question answering: towards learning to converse over linked question answer pairs with a knowledge graph. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  37. Schütze, H., Manning, C.D., Raghavan, P.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)

    MATH  Google Scholar 

  38. Singh, K., et al.: Why reinvent the wheel: let’s build question answering systems together. In: Proceedings of the 2018 World Wide Web Conference, pp. 1247–1256 (2018)

    Google Scholar 

  39. Solovyev, A.: Dependency-based algorithms for answer validation task in Russian question answering. In: Gurevych, I., Biemann, C., Zesch, T. (eds.) GSCL 2013. LNCS (LNAI), vol. 8105, pp. 199–212. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40722-2_20

    CrossRef  Google Scholar 

  40. Tan, C., Wei, F., Zhou, Q., Yang, N., Lv, W., Zhou, M.: I know there is no answer: modeling answer validation for machine reading comprehension. In: Zhang, M., Ng, V., Zhao, D., Li, S., Zan, H. (eds.) NLPCC 2018. LNCS (LNAI), vol. 11108, pp. 85–97. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99495-6_8

    CrossRef  Google Scholar 

  41. Trivedi, P., Maheshwari, G., Dubey, M., Lehmann, J.: LC-QuAD: a corpus for complex question answering over knowledge graphs. In: d’Amato, C., Fernandez, M., Tamma, V., Lecue, F., Cudré-Mauroux, P., Sequeda, J., Lange, C., Heflin, J. (eds.) ISWC 2017. LNCS, vol. 10588, pp. 210–218. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68204-4_22

    CrossRef  Google Scholar 

  42. Usbeck, R., Gusmita, R.H., Ngomo, A.N., Saleem, M.: 9th challenge on question answering over linked data (QALD-9). In: Joint proceedings of the 4th Workshop on Semantic Deep Learning (SemDeep-4) and NLIWoD4: Natural Language Interfaces for the Web of Data (NLIWOD-4) and 9th Question Answering over Linked Data challenge (QALD-9) co-located with 17th International Semantic Web Conference (ISWC 2018), Monterey, 8th–9th October 2018, pp. 58–64 (2018)

    Google Scholar 

  43. Usbeck, R., et al.: GERBIL - general entity annotation benchmark framework. In: 24th WWW Conference (2015)

    Google Scholar 

  44. Yen, A.Z., Huang, H.H., Chen, H.H.: Unanswerable question correction in question answering over personal knowledge base. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14266–14275 (2021)

    Google Scholar 

  45. Yih, S.W., Chang, M.W., He, X., Gao, J.: Semantic parsing via staged query graph generation: question answering with knowledge base. In: Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP (2015)

    Google Scholar 

  46. Yu, M., Yin, W., Hasan, K.S., Santos, C.D., Xiang, B., Zhou, B.: Improved neural relation detection for knowledge base question answering. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Long Papers), Vol. 1, pp. 1321–1331. Association for Computational Linguistics (2017)

    Google Scholar 

  47. Zamanov, I., Kraeva, M., Hateva, N., Yovcheva, I., Nikolova, I., Angelova, G.: Voltron: a hybrid system for answer validation based on lexical and distance features. In: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). pp. 242–246. Association for Computational Linguistics, Denver (2015). https://doi.org/10.18653/v1/S15-2043

Download references

Author information

Authors and Affiliations

  1. Perm National Research Polytechnic University, Perm, Russia

    Aleksandr Gashkov & Maria Eltsova

  2. Anhalt University of Applied Sciences, Köthen, Germany

    Aleksandr Perevalov

  3. Leipzig University of Applied Sciences, Leipzig, Germany

    Aleksandr Perevalov & Andreas Both

  4. DATEV eG, Nuremberg, Germany

    Andreas Both

Authors
  1. Aleksandr Gashkov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aleksandr Perevalov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maria Eltsova
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andreas Both
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Both .

Editor information

Editors and Affiliations

  1. University of Amsterdam, Amsterdam, Noord-Holland, The Netherlands

    Paul Groth

  2. Universidad Simón Bolívar, Leibniz Information Centre for Science and Technology, Hannover, Niedersachsen, Germany

    Maria-Esther Vidal

  3. Institut Polytechnique de Paris "DIG", Télécom ParisTech, Palaiseau, France

    Fabian Suchanek

  4. University of Southern California, Marina del Rey, CA, USA

    Pedro Szekley

  5. IBM Research - Thomas J. Watson Research, Yorktown Heights, NY, USA

    Pavan Kapanipathi

  6. LaSIGE, Fac de Ciencias,Edif C6, Pis0 3, Universidade de Lisboa, Lisbon, Portugal

    Dr. Catia Pesquita

  7. University of Nantes, Nantes, France

    Hala Skaf-Molli

  8. Aalto University, Espoo, Finland

    Minna Tamper

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Gashkov, A., Perevalov, A., Eltsova, M., Both, A. (2022). Improving Question Answering Quality Through Language Feature-Based SPARQL Query Candidate Validation. In: , et al. The Semantic Web. ESWC 2022. Lecture Notes in Computer Science, vol 13261. Springer, Cham. https://doi.org/10.1007/978-3-031-06981-9_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06981-9_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06980-2

  • Online ISBN: 978-3-031-06981-9

  • eBook Packages: Computer ScienceComputer Science (R0)