Sentence entailment in compositional distributional semantics

Abstract

Distributional semantic models provide vector representations for words by gathering co-occurrence frequencies from corpora of text. Compositional distributional models extend these from words to phrases and sentences. In categorical compositional distributional semantics, phrase and sentence representations are functions of their grammatical structure and representations of the words therein. In this setting, grammatical structures are formalised by morphisms of a compact closed category and meanings of words are formalised by objects of the same category. These can be instantiated in the form of vectors or density matrices. This paper concerns the applications of this model to phrase and sentence level entailment. We argue that entropy-based distances of vectors and density matrices provide a good candidate to measure word-level entailment, show the advantage of density matrices over vectors for word level entailments, and prove that these distances extend compositionally from words to phrases and sentences. We exemplify our theoretical constructions on real data and a toy entailment dataset and provide preliminary experimental evidence.

References

  1. 1.

    Balkır, E.: Using Density Matrices in a Compositional Distributional Model of Meaning. Master’s thesis, University of Oxford (2014)

  2. 2.

    Balkır, E., Sadrzadeh, M., Coecke, B.: Distributional sentence entailment using density matrices. In: FTP-ENTC Proceedings of the First International Conference on Theoretical Topics in Computer Science (TTCS), vol. 9541, pp. 1–22 (2015)

  3. 3.

    Balkir, E., Kartsaklis, D., Sadrzadeh, M.: Sentence entailment in compositional distributional semantics. In: Fourteenth International Symposium on Artificial Intelligence and Mathematics. arXiv:1512.04419 (2016)

  4. 4.

    Bankova, D., Coecke, B., Lewis, M., Marsden, D.: Graded entailment for compositional distributional semantics, arXiv:1601.04908 (2016)

  5. 5.

    Baroni, M., Zamparelli, R.: Nouns are vectors, adjectives are matrices: representing adjective-noun constructions in semantic space. In: Conference on Empirical Methods in Natural Language Processing (EMNLP-10). Cambridge (2010)

  6. 6.

    Baroni, M., Bernardi, R., Do, N.Q., Shan, C.C.: Entailment above the word level in distributional semantics. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 23–32. Association for Computational Linguistics (2012)

  7. 7.

    Blacoe, W., Kashefi, E., Lapata, M.: A quantum-theoretic approach to distributional semantics. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 847–857 (2013)

  8. 8.

    Clark, S., Pulman, S.: Combining symbolic and distributional models of meaning. In: Proceedings of the AAAI Spring Symposium on Quantum Interaction, pp. 52–55 (2007)

  9. 9.

    Clarke, D.: Context-theoretic semantics for natural language: an overview. In: Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pp. 112–119. Association for Computational Linguistics (2009)

  10. 10.

    Coecke, B., Paquette, E.O.: Categories for the practising physicist. Springer, Berlin (2010)

    Google Scholar 

  11. 11.

    Coecke, B., Sadrzadeh, M., Clark, S.: Mathematical foundations for a compositional distributional model of meaning. Linguist. Anal. 36, 345–384 (2010)

    Google Scholar 

  12. 12.

    Dagan, I., Lee, L., Pereira, F.C.N.: Similarity-based models of word cooccurrence probabilities. Mach. Learn. 34(1–3), 43–69 (1999)

    Article  MATH  Google Scholar 

  13. 13.

    Dagan, I., Glickman, O., Magnini, B.: The pascal recognising textual entailment challenge. In: Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pp. 177–190. Springer, Berlin (2006)

  14. 14.

    Firth, J.R.: A Synopsis of Linguistic Theory, 1930–1955. Studies in Linguistic Analysis, pp. 1–32 (1957)

  15. 15.

    Grefenstette, E., Sadrzadeh, M.: Experimental support for a categorical compositional distributional model of meaning. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 1394–1404. Association for Computational Linguistics (2011)

  16. 16.

    Grefenstette, E., Sadrzadeh, M.: Concrete models and empirical evaluations for the categorical compositional distributional model of meaning. Comput. Linguist. 41, 71–118 (2015)

    Article  Google Scholar 

  17. 17.

    Harris, Z.: Distributional structure. Word 10, 146–162 (1954)

    Article  Google Scholar 

  18. 18.

    Hedges, J., Sadrzadeh, M.: A generalised quantifier theory of natural language in categorical compositional distributional semantics with bialgebras. In: EPTCS Proceedings of the 13th International Conference on Quantum Physics and Logic, to appear (2016)

  19. 19.

    Herbelot, A., Ganesalingam, M.: Measuring semantic content in distributional vectors. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 440–445. Association for Computational Linguistics (2013)

  20. 20.

    Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 655–665. Association for Computational Linguistics (2014)

  21. 21.

    Kartsaklis, D.: Compositional Distributional Semantics with Compact Closed Categories and Frobenius Algebras. Ph.D. thesis, University of Oxford (2015)

  22. 22.

    Kartsaklis, D.: Coordination in categorical compositional distributional semantics. In: EPTCS Proceedings of the Workshop on Semantic Spaces at the Intersection of NLP, Physics and Cognitive Science, to appear (2016)

  23. 23.

    Kartsaklis, D., Sadrzadeh, M.: Prior disambiguation of word tensors for constructing sentence vectors. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNL), pp. 1590–1601 (2013)

  24. 24.

    Kartsaklis, D., Sadrzadeh, M.: A Frobenius model of information structure in categorical compositional distributional semantics. In: Proceedings of the 14th Meeting on Mathematics of Language (2015)

  25. 25.

    Kartsaklis, D., Sadrzadeh, M., Pulman, S.: A unified sentence space for categorical distributional-compositional semantics: theory and experiments. In: COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Posters, 8–15 December 2012, Mumbai, India, pp. 549–558 (2012)

  26. 26.

    Kelly, G., Laplaza, M.: Coherence for compact closed categories. J. Pure Appl. Algebra 19(0), 193–213 (1980) http://www.sciencedirect.com/science/article/pii/0022404980901012

    MathSciNet  Article  MATH  Google Scholar 

  27. 27.

    Kotlerman, L., Dagan, I., Szpektor, I., Zhitomirsky-Geffet, M.: Directional distributional similarity for lexical inference. Nat. Lang. Eng. 16(04), 359–389 (2010)

    Article  Google Scholar 

  28. 28.

    Lambek, J.: Type grammars as pregroups. Grammars 4(1), 21–39 (2001)

    Article  MATH  Google Scholar 

  29. 29.

    Lee, L.: Measures of distributional similarity. In: Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics, pp. 25–32 (1999)

  30. 30.

    MacCartney, B., Manning, C.D.: Natural logic for textual inference. In: ACL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics (2007)

  31. 31.

    MacLane, S.: Categories for the Working Mathematician. Springer, Berlin (1971)

    Google Scholar 

  32. 32.

    Milajevs, D., Kartsaklis, D., Sadrzadeh, M., Purver, M.: Evaluating neural word representations in tensor-based compositional settings. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, pp. 708–719 (2014). http://www.aclweb.org/anthology/D14-1079

  33. 33.

    Mitchell, J., Lapata, M.: Composition in distributional models of semantics. Cogn. Sci. 34(8), 1388–1439 (2010)

    Article  Google Scholar 

  34. 34.

    Piedeleu, R.: Ambiguity in Categorical Models of Meaning. Master’s thesis, University of Oxford (2014)

  35. 35.

    Piedeleu, R., Kartsaklis, D., Coecke, B., Sadrzadeh, M.: Open system categorical quantum semantics in natural language processing. In: Proceedings of the 6th Conference on Algebra and Coalgebra in Computer Science. Nijmegen, Netherlands (2015)

  36. 36.

    Preller, A.: From Sentence to Concept, a Linguistic Quantum Logic. Tech. Rep. RR-11019. LIRMM (2011). http://hal-lirmm.ccsd.cnrs.fr/lirmm-00600428

  37. 37.

    Reddy, S., McCarthy, D., Manandhar, S.: An empirical study on compositionality in compound nouns. In: Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP-11) (2011)

  38. 38.

    Rimell, L.: Distributional lexical entailment by topic coherence. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26–30, 2014, Gothenburg, Sweden, pp. 511–519 (2014)

  39. 39.

    Rubenstein, H., Goodenough, J.: Contextual correlates of synonymy. Commun. ACM 8(10), 627–633 (1965)

    Article  Google Scholar 

  40. 40.

    Sadrzadeh, M., Clark, S., Coecke, B.: Frobenius anatomy of word meanings i: subject and object relative pronouns. J. Log. Comput. 23, 1293–1317 (2013)

    MathSciNet  Article  MATH  Google Scholar 

  41. 41.

    Sadrzadeh, M., Clark, S., Coecke, B.: Frobenius anatomy of word meanings 2: possessive relative pronouns. J. Log. Comput. 26, 785–815 (2016)

    MathSciNet  Article  MATH  Google Scholar 

  42. 42.

    Salton, G., Wong, A., Yang, C.S.: A vector space model for automatic indexing. Commun. ACM 18, 613–620 (1975)

    Article  MATH  Google Scholar 

  43. 43.

    Schütze, H.: Automatic word sense discrimination. Comput. Linguist. 24, 97–123 (1998)

    Google Scholar 

  44. 44.

    Selinger, P.: Dagger compact closed categories and completely positive maps. Electronic Notes in Theoretical Computer Science 170, 139–163 (2007)

    Article  MATH  Google Scholar 

  45. 45.

    Socher, R., Huval, B., Manning, C., Ng, AY: Semantic compositionality through recursive matrix-vector spaces. In: Conference on Empirical Methods in Natural Language Processing 2012 (2012)

  46. 46.

    Turney, P.D.: Similarity of semantic relations. Comput. Linguist. 32(3), 379–416 (2006)

    Article  MATH  Google Scholar 

  47. 47.

    Weeds, J., Weir, D., McCarthy, D.: Characterising measures of lexical distributional similarity. In: Proceedings of the 20th international conference on Computational Linguistics. No. 1015, Association for Computational Linguistics (2004)

  48. 48.

    Widdows, D.: Geometry and meaning. Center for the Study of Language and Information/SRI (2004)

Download references

Acknowledgements

Sadrzadeh is supported by EPSRC CAF grant EP/J002607/1 and Kartsaklis by AFOSR grant FA9550-14-1-0079. Balkır was supported by a Queen Mary Vice Principal scholarship, when contributing to this project.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mehrnoosh Sadrzadeh.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sadrzadeh, M., Kartsaklis, D. & Balkır, E. Sentence entailment in compositional distributional semantics. Ann Math Artif Intell 82, 189–218 (2018). https://doi.org/10.1007/s10472-017-9570-x

Download citation

Keywords

  • Distributional semantics
  • Compositional distributional semantics
  • Distributional inclusion hypothesis
  • Density matrices
  • Entailment
  • Entropy

Mathematics Subject Classification (2010)

  • 03B65