Measuring Semantic Coherence of a Conversation

  • Svitlana Vakulenko
  • Maarten de Rijke
  • Michael Cochez
  • Vadim Savenkov
  • Axel PolleresEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11136)


Conversational systems have become increasingly popular as a way for humans to interact with computers. To be able to provide intelligent responses, conversational systems must correctly model the structure and semantics of a conversation. We introduce the task of measuring semantic (in)coherence in a conversation with respect to background knowledge, which relies on the identification of semantic relations between concepts introduced during a conversation. We propose and evaluate graph-based and machine learning-based approaches for measuring semantic coherence using knowledge graphs, their vector space embeddings and word embedding models, as sources of background knowledge. We demonstrate how these approaches are able to uncover different coherence patterns in conversations on the Ubuntu Dialogue Corpus.



This work is supported by the project 855407 “Open Data for Local Communities” (CommuniData) of the Austrian Federal Ministry of Transport, Innovation and Technology (BMVIT) under the program “ICT of the Future.” Svitlana Vakulenko was supported by the EU H2020 programme under the MSCA-RISE agreement 645751 (RISE_BPM). Axel Polleres was supported under the Distinguished Visiting Austrian Chair Professors program hosted by The Europe Center of Stanford University. Maarten de Rijke was supported by Ahold Delhaize, Amsterdam Data Science, the Bloomberg Research Grant program, the China Scholarship Council, the Criteo Faculty Research Award program, Elsevier, the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement nr 312827 (VOX-Pol), the Google Faculty Research Awards program, the Microsoft Research Ph.D. program, the Netherlands Institute for Sound and Vision, the Netherlands Organisation for Scientific Research (NWO) under project nrs CI-14-25, 652.002.001, 612.001.551, 652.001.003, and Yandex. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.


  1. 1.
    Athreya, R.G., Ngonga, A., Usbeck, R.: Enhancing community interactions with data-driven chatbots - the DBpedia chatbot. In: WWW 2018 Companion. ACM (2018)Google Scholar
  2. 2.
    Barzilay, R., Lapata, M.: Modeling local coherence: an entity-based approach. Comput. Linguist. 34(1), 1–34 (2008)CrossRefGoogle Scholar
  3. 3.
    Blanco, R., Ottaviano, G., Meij, E.: Fast and space-efficient entity linking for queries. In: WDSM 2015, pp. 179–188. ACM (2015)Google Scholar
  4. 4.
    Chomsky, N.: Syntactic Structures. Mouton and Co., The Hague (1957)zbMATHGoogle Scholar
  5. 5.
    Cochez, M., Ristoski, P., Ponzetto, S.P., Paulheim, H.: Biased graph walks for RDF graph embeddings. In: WIMS 2017, pp. 21:1–21:12 (2017)Google Scholar
  6. 6.
    Cochez, M., Ristoski, P., Ponzetto, S.P., Paulheim, H.: Global RDF vector space embeddings. In: d’Amato, C., et al. (eds.) ISWC 2017. LNCS, vol. 10587, pp. 190–207. Springer, Cham (2017). Scholar
  7. 7.
    Cui, B., Li, Y., Zhang, Y., Zhang, Z.: Text coherence analysis based on deep neural network. In: CIKM 2017, pp. 2027–2030. ACM (2017)Google Scholar
  8. 8.
    Daiber, J., Jakob, M., Hokamp, C., Mendes, P.N.: Improving efficiency and accuracy in multilingual entity extraction. In: I-SEMANTICS 2013, pp. 121–124 (2013)Google Scholar
  9. 9.
    De Beaugrande, R., Dressler, W.: Textlinguistics. Longman, Harlow (1981)Google Scholar
  10. 10.
    Elsner, M., Charniak, E.: Extending the entity grid with entity-specific features. In: ACL 2011, pp. 125–129. ACL (2011)Google Scholar
  11. 11.
    Fernández, J.D., Martínez-Prieto, M.A., Gutiérrez, C., Polleres, A., Arias, M.: Binary RDF representation for publication and exchange (HDT). JWS 19, 22–41 (2013)CrossRefGoogle Scholar
  12. 12.
    Galley, M., McKeown, K., Fosler-Lussier, E., Jing, H.: Discourse segmentation of multi-party conversation. In: ACL 2003, pp. 562–569 (2003)Google Scholar
  13. 13.
    Hasibi, F., Balog, K., Garigliotti, D., Zhang, S.: Nordlys: a toolkit for entity-oriented and semantic search. In: SIGIR 2017, pp. 1289–1292 (2017)Google Scholar
  14. 14.
    Kim, Y.: Convolutional neural networks for sentence classification. CoRR abs/1408.5882 (2014)Google Scholar
  15. 15.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014)Google Scholar
  16. 16.
    Lapata, M.: Probabilistic text structuring: experiments with sentence ordering. In: ACL 2003, pp. 545–552 (2003)Google Scholar
  17. 17.
    Lowe, R., Pow, N., Serban, I., Pineau, J.: The ubuntu dialogue corpus: a large dataset for research in unstructured multi-turn dialogue systems. In: SIGDIAL 2015, pp. 285–294 (2015)Google Scholar
  18. 18.
    Lowe, R.T., Pow, N., Serban, I.V., Charlin, L., Liu, C., Pineau, J.: Training end-to-end dialogue systems with the ubuntu dialogue corpus. D&D 8(1), 31–65 (2017)Google Scholar
  19. 19.
    Lukovnikov, D., Fischer, A., Lehmann, J., Auer, S.: Neural network-based question answering over knowledge graphs on word and character level. In: WWW 2017, pp. 1211–1220. ACM (2017)Google Scholar
  20. 20.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NIPS 2013, pp. 3111–3119 (2013)Google Scholar
  21. 21.
    Mohammad, S., Hirst, G.: Distributional measures as proxies for semantic relatedness. CoRR abs/1203.1 (2012)Google Scholar
  22. 22.
    Nguyen, D.T., Joty, S.R.: A neural local coherence model. In: ACL 2017, pp. 1320–1330 (2017)Google Scholar
  23. 23.
    Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: EMNLP 2014, pp. 1532–1543. ACL (2014)Google Scholar
  24. 24.
    Petöfi, J.S.: Semantics, pragmatics, text theory. Università di Urbino (1974)Google Scholar
  25. 25.
    Savenkov, V., Mehmood, Q., Umbrich, J., Polleres, A.: Counting to k or how SPARQL1.1 property paths can be extended to top-k path queries. In: SEMANTICS 2017, pp. 97–103 (2017)Google Scholar
  26. 26.
    Silva, V.S., Freitas, A., Handschuh, S.: Recognizing and justifying text entailment through distributional navigation on definition graphs. In: AAAI 2018 (2018)Google Scholar
  27. 27.
    Thoma, S., Rettinger, A., Both, F.: Towards holistic concept representations: embedding relational knowledge, visual attributes, and distributional word semantics. In: d’Amato, C., et al. (eds.) ISWC 2017. LNCS, vol. 10587, pp. 694–710. Springer, Cham (2017). Scholar
  28. 28.
    Usbeck, R., Ngomo, A.N., Haarmann, B., Krithara, A., Röder, M., Napolitano, G.: 7th open challenge on question answering over linked data (QALD-7). In: 4th SemWebEval Challenge at ESWC 2017, pp. 59–69 (2017)CrossRefGoogle Scholar
  29. 29.
    Wilcke, X., Bloem, P., de Boer, V.: The knowledge graph as the default data model for learning on heterogeneous knowledge. Data Sci. 1(1–2), 39–57 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Svitlana Vakulenko
    • 1
  • Maarten de Rijke
    • 2
  • Michael Cochez
    • 3
    • 4
    • 5
  • Vadim Savenkov
    • 1
  • Axel Polleres
    • 1
    • 6
    • 7
    Email author
  1. 1.Vienna University of Economics and BusinessViennaAustria
  2. 2.University of AmsterdamAmsterdamThe Netherlands
  3. 3.Fraunhofer FITSankt AugustinGermany
  4. 4.Informatik 5, RWTH University AachenAachenGermany
  5. 5.Faculty of Information TechnologyUniversity of JyvaskylaJyvaskylaFinland
  6. 6.Complexity Science Hub ViennaViennaAustria
  7. 7.Stanford UniversityStanfordUSA

Personalised recommendations