Sentence entailment in compositional distributional semantics

  • Mehrnoosh Sadrzadeh
  • Dimitri Kartsaklis
  • Esma Balkır
Open Access
Article
  • 13 Downloads

Abstract

Distributional semantic models provide vector representations for words by gathering co-occurrence frequencies from corpora of text. Compositional distributional models extend these from words to phrases and sentences. In categorical compositional distributional semantics, phrase and sentence representations are functions of their grammatical structure and representations of the words therein. In this setting, grammatical structures are formalised by morphisms of a compact closed category and meanings of words are formalised by objects of the same category. These can be instantiated in the form of vectors or density matrices. This paper concerns the applications of this model to phrase and sentence level entailment. We argue that entropy-based distances of vectors and density matrices provide a good candidate to measure word-level entailment, show the advantage of density matrices over vectors for word level entailments, and prove that these distances extend compositionally from words to phrases and sentences. We exemplify our theoretical constructions on real data and a toy entailment dataset and provide preliminary experimental evidence.

Keywords

Distributional semantics Compositional distributional semantics Distributional inclusion hypothesis Density matrices Entailment Entropy 

Mathematics Subject Classification (2010)

03B65 

Notes

Acknowledgements

Sadrzadeh is supported by EPSRC CAF grant EP/J002607/1 and Kartsaklis by AFOSR grant FA9550-14-1-0079. Balkır was supported by a Queen Mary Vice Principal scholarship, when contributing to this project.

References

  1. 1.
    Balkır, E.: Using Density Matrices in a Compositional Distributional Model of Meaning. Master’s thesis, University of Oxford (2014)Google Scholar
  2. 2.
    Balkır, E., Sadrzadeh, M., Coecke, B.: Distributional sentence entailment using density matrices. In: FTP-ENTC Proceedings of the First International Conference on Theoretical Topics in Computer Science (TTCS), vol. 9541, pp. 1–22 (2015)Google Scholar
  3. 3.
    Balkir, E., Kartsaklis, D., Sadrzadeh, M.: Sentence entailment in compositional distributional semantics. In: Fourteenth International Symposium on Artificial Intelligence and Mathematics. arXiv:1512.04419 (2016)
  4. 4.
    Bankova, D., Coecke, B., Lewis, M., Marsden, D.: Graded entailment for compositional distributional semantics, arXiv:1601.04908 (2016)
  5. 5.
    Baroni, M., Zamparelli, R.: Nouns are vectors, adjectives are matrices: representing adjective-noun constructions in semantic space. In: Conference on Empirical Methods in Natural Language Processing (EMNLP-10). Cambridge (2010)Google Scholar
  6. 6.
    Baroni, M., Bernardi, R., Do, N.Q., Shan, C.C.: Entailment above the word level in distributional semantics. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 23–32. Association for Computational Linguistics (2012)Google Scholar
  7. 7.
    Blacoe, W., Kashefi, E., Lapata, M.: A quantum-theoretic approach to distributional semantics. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 847–857 (2013)Google Scholar
  8. 8.
    Clark, S., Pulman, S.: Combining symbolic and distributional models of meaning. In: Proceedings of the AAAI Spring Symposium on Quantum Interaction, pp. 52–55 (2007)Google Scholar
  9. 9.
    Clarke, D.: Context-theoretic semantics for natural language: an overview. In: Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pp. 112–119. Association for Computational Linguistics (2009)Google Scholar
  10. 10.
    Coecke, B., Paquette, E.O.: Categories for the practising physicist. Springer, Berlin (2010)CrossRefMATHGoogle Scholar
  11. 11.
    Coecke, B., Sadrzadeh, M., Clark, S.: Mathematical foundations for a compositional distributional model of meaning. Linguist. Anal. 36, 345–384 (2010)Google Scholar
  12. 12.
    Dagan, I., Lee, L., Pereira, F.C.N.: Similarity-based models of word cooccurrence probabilities. Mach. Learn. 34(1–3), 43–69 (1999)CrossRefMATHGoogle Scholar
  13. 13.
    Dagan, I., Glickman, O., Magnini, B.: The pascal recognising textual entailment challenge. In: Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pp. 177–190. Springer, Berlin (2006)Google Scholar
  14. 14.
    Firth, J.R.: A Synopsis of Linguistic Theory, 1930–1955. Studies in Linguistic Analysis, pp. 1–32 (1957)Google Scholar
  15. 15.
    Grefenstette, E., Sadrzadeh, M.: Experimental support for a categorical compositional distributional model of meaning. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 1394–1404. Association for Computational Linguistics (2011)Google Scholar
  16. 16.
    Grefenstette, E., Sadrzadeh, M.: Concrete models and empirical evaluations for the categorical compositional distributional model of meaning. Comput. Linguist. 41, 71–118 (2015)CrossRefGoogle Scholar
  17. 17.
    Harris, Z.: Distributional structure. Word 10, 146–162 (1954)CrossRefGoogle Scholar
  18. 18.
    Hedges, J., Sadrzadeh, M.: A generalised quantifier theory of natural language in categorical compositional distributional semantics with bialgebras. In: EPTCS Proceedings of the 13th International Conference on Quantum Physics and Logic, to appear (2016)Google Scholar
  19. 19.
    Herbelot, A., Ganesalingam, M.: Measuring semantic content in distributional vectors. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 440–445. Association for Computational Linguistics (2013)Google Scholar
  20. 20.
    Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 655–665. Association for Computational Linguistics (2014)Google Scholar
  21. 21.
    Kartsaklis, D.: Compositional Distributional Semantics with Compact Closed Categories and Frobenius Algebras. Ph.D. thesis, University of Oxford (2015)Google Scholar
  22. 22.
    Kartsaklis, D.: Coordination in categorical compositional distributional semantics. In: EPTCS Proceedings of the Workshop on Semantic Spaces at the Intersection of NLP, Physics and Cognitive Science, to appear (2016)Google Scholar
  23. 23.
    Kartsaklis, D., Sadrzadeh, M.: Prior disambiguation of word tensors for constructing sentence vectors. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNL), pp. 1590–1601 (2013)Google Scholar
  24. 24.
    Kartsaklis, D., Sadrzadeh, M.: A Frobenius model of information structure in categorical compositional distributional semantics. In: Proceedings of the 14th Meeting on Mathematics of Language (2015)Google Scholar
  25. 25.
    Kartsaklis, D., Sadrzadeh, M., Pulman, S.: A unified sentence space for categorical distributional-compositional semantics: theory and experiments. In: COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Posters, 8–15 December 2012, Mumbai, India, pp. 549–558 (2012)Google Scholar
  26. 26.
    Kelly, G., Laplaza, M.: Coherence for compact closed categories. J. Pure Appl. Algebra 19(0), 193–213 (1980) http://www.sciencedirect.com/science/article/pii/0022404980901012 MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Kotlerman, L., Dagan, I., Szpektor, I., Zhitomirsky-Geffet, M.: Directional distributional similarity for lexical inference. Nat. Lang. Eng. 16(04), 359–389 (2010)CrossRefGoogle Scholar
  28. 28.
    Lambek, J.: Type grammars as pregroups. Grammars 4(1), 21–39 (2001)CrossRefMATHGoogle Scholar
  29. 29.
    Lee, L.: Measures of distributional similarity. In: Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics, pp. 25–32 (1999)Google Scholar
  30. 30.
    MacCartney, B., Manning, C.D.: Natural logic for textual inference. In: ACL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics (2007)Google Scholar
  31. 31.
    MacLane, S.: Categories for the Working Mathematician. Springer, Berlin (1971)MATHGoogle Scholar
  32. 32.
    Milajevs, D., Kartsaklis, D., Sadrzadeh, M., Purver, M.: Evaluating neural word representations in tensor-based compositional settings. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, pp. 708–719 (2014). http://www.aclweb.org/anthology/D14-1079
  33. 33.
    Mitchell, J., Lapata, M.: Composition in distributional models of semantics. Cogn. Sci. 34(8), 1388–1439 (2010)CrossRefGoogle Scholar
  34. 34.
    Piedeleu, R.: Ambiguity in Categorical Models of Meaning. Master’s thesis, University of Oxford (2014)Google Scholar
  35. 35.
    Piedeleu, R., Kartsaklis, D., Coecke, B., Sadrzadeh, M.: Open system categorical quantum semantics in natural language processing. In: Proceedings of the 6th Conference on Algebra and Coalgebra in Computer Science. Nijmegen, Netherlands (2015)Google Scholar
  36. 36.
    Preller, A.: From Sentence to Concept, a Linguistic Quantum Logic. Tech. Rep. RR-11019. LIRMM (2011). http://hal-lirmm.ccsd.cnrs.fr/lirmm-00600428
  37. 37.
    Reddy, S., McCarthy, D., Manandhar, S.: An empirical study on compositionality in compound nouns. In: Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP-11) (2011)Google Scholar
  38. 38.
    Rimell, L.: Distributional lexical entailment by topic coherence. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26–30, 2014, Gothenburg, Sweden, pp. 511–519 (2014)Google Scholar
  39. 39.
    Rubenstein, H., Goodenough, J.: Contextual correlates of synonymy. Commun. ACM 8(10), 627–633 (1965)CrossRefGoogle Scholar
  40. 40.
    Sadrzadeh, M., Clark, S., Coecke, B.: Frobenius anatomy of word meanings i: subject and object relative pronouns. J. Log. Comput. 23, 1293–1317 (2013)MathSciNetCrossRefMATHGoogle Scholar
  41. 41.
    Sadrzadeh, M., Clark, S., Coecke, B.: Frobenius anatomy of word meanings 2: possessive relative pronouns. J. Log. Comput. 26, 785–815 (2016)MathSciNetCrossRefMATHGoogle Scholar
  42. 42.
    Salton, G., Wong, A., Yang, C.S.: A vector space model for automatic indexing. Commun. ACM 18, 613–620 (1975)CrossRefMATHGoogle Scholar
  43. 43.
    Schütze, H.: Automatic word sense discrimination. Comput. Linguist. 24, 97–123 (1998)Google Scholar
  44. 44.
    Selinger, P.: Dagger compact closed categories and completely positive maps. Electronic Notes in Theoretical Computer Science 170, 139–163 (2007)CrossRefMATHGoogle Scholar
  45. 45.
    Socher, R., Huval, B., Manning, C., Ng, AY: Semantic compositionality through recursive matrix-vector spaces. In: Conference on Empirical Methods in Natural Language Processing 2012 (2012)Google Scholar
  46. 46.
    Turney, P.D.: Similarity of semantic relations. Comput. Linguist. 32(3), 379–416 (2006)CrossRefMATHGoogle Scholar
  47. 47.
    Weeds, J., Weir, D., McCarthy, D.: Characterising measures of lexical distributional similarity. In: Proceedings of the 20th international conference on Computational Linguistics. No. 1015, Association for Computational Linguistics (2004)Google Scholar
  48. 48.
    Widdows, D.: Geometry and meaning. Center for the Study of Language and Information/SRI (2004)Google Scholar

Copyright information

© The Author(s) 2018

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Mehrnoosh Sadrzadeh
    • 1
  • Dimitri Kartsaklis
    • 1
  • Esma Balkır
    • 1
  1. 1.School of Electronic Engineering and Computer ScienceQueen Mary University of LondonLondonUK

Personalised recommendations