Abstract
This chapter discusses representations for larger structures in natural language. The primary focus is on the sentence level. However, many of the techniques also apply to sub-sentence structures (phrases), and super-sentence structures (documents). The three main types of representations discussed here are: unordered models, such as sum of word embeddings; sequential models, such as recurrent neural networks; and structured models, such as recursive autoencoders.
A sentence is a group of words expressing a complete thought
English Composition and Literature,
Webster,1923
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python. O’Reilly Media, Inc.
Blei, David M., Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. The Journal of Machine Learning Research 3: 993–1022.
Bowman, Samuel R., Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016a. A fast unified model for parsing and sentence understanding. arXiv:1603.06021.
Bowman, Samuel R., Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. 2016b. Generating sentences from a continuous space. In International conference on learning representations (ICLR) Workshop.
Cho, Kyunghyun, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 1724–1734. Doha, Qatar: Association for Computational Linguistics.
Dumais, Susan T., George W. Furnas, Thomas K. Landauer, Scott Deerwester, and Richard Harshman. 1988. Using latent semantic analysis to improve access to textual information. In Proceedings of the SIGCHI conference on Human factors in computing systems, 281–285. ACM.
Goller, Christoph and Andreas Kuchler. 1996. Learning task-dependent distributed representations by back propagation through structure. In IEEE international conference on neural networks, 1996, vol. 1, 347–352. IEEE.
Hofmann, Thomas. 2000. Learning the similarity of documents: An information geometric approach to document retrieval and categorization. In Advances in neural information processing systems, 914–920.
Honnibal, Matthew and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 conference on empirical methods in natural language processing, 1373–1378. Lisbon, Portugal: Association for Computational Linguistics.
Horvat, Matic and William Byrne. 2014. A graph-based approach to string regeneration. In EACL, 85–95.
Iyyer, Mohit, Jordan Boyd-Graber, and Hal Daumé III. 2014a. Generating sentences from semantic vector space representations. In NIPS workshop on learning semantics.
Iyyer, Mohit, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. 2014b. A neural network for factoid question answering over para-graphs. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 633–644.
Kingma, D. P. and M. Welling. 2014. Auto-encoding variational bayes. In The international conference on learning representations (ICLR). arXiv:1312.6114 [stat.ML].
Kiros, Ryan, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In CoRR. arXiv:1506.06726.
Lau, Jey Han, and Timothy Baldwin. 2016. An empirical evaluation of doc2vec with practical insights into document embedding generation. In ACL, 78.
Le, Quoc and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st international conference on machine learning (ICML-14), 1188–1196.
Li, Bofang, Tao Liu, Zhe Zhao, Puwei Wang, and Xiaoyong Du. 2017. Neural bag-of-ngrams. In AAAI, 3067–3074.
Manning, C.D. and H. Schütze. 1999. Foundations of statistical natural language processing. Cambridge: MIT Press. ISBN: 9780262133609.
Manning, Christopher D., Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for computational linguistics (ACL) system demonstrations, 55–60.
Mesnil, Grégoire, Tomas Mikolov, Marc’Aurelio Ranzato, and Yoshua Bengio. 2014. Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews. arXiv:1412.5335.
Mitchell, Jeff and Mirella Lapata. 2008. Vector-based models of semantic composition. In ACL, 236–244.
Pollack, Jordan B. 1990. Recursive distributed representations. Artificial Intelligence, 46 (1): 77–105. ISSN: 0004-3702. https://doi.org/10.1016/0004-3702(90)90005-K.
Rehůrek, Radim and Petr Sojka. 2010. Software framework for topic modelling with large corpora. English. In Proceedings of the LREC 2010 workshop on new challenges for NLP frameworks, 45–50. Valletta, Malta: ELRA. http://is.muni.cz/publication/884893/en
Ritter, Samuel, Cotie Long, Denis Paperno, Marco Baroni, Matthew Botvinick, and Adele Goldberg. 2015. Leveraging preposition ambiguity to assess compositional distributional models of semantics. In The fourth joint conference on lexical and computational semantics.
Socher, Richard. 2014. Recursive deep learning for natural language processing and computer vision. Ph.D. thesis. Stanford University.
Socher, Richard, Christopher D. Manning, and Andrew Y. Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 deep learning and unsupervised feature learning workshop, 1–9.
Socher, Richard, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in neural information processing systems, 24.
Socher, Richard, Cliff C Lin, Chris Manning, and Andrew Y Ng. 2011b. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11), 129–136.
Socher, Richard, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011c. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the 2011 conference on empirical methods in natural language processing (EMNLP).
Socher, Richard, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, 1201–1211. Association for Computational Linguistics.
Socher, Richard, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Y.Ng Andrew. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics 2: 207–218.
Stenetorp, Pontus. 2013. Transition-based dependency parsing using recursive neural networks. In Deep learning workshop at the, 2013. Conference on neural information processing systems (NIPS). Nevada, USA: Lake Tahoe.
Wang, Sida and Christopher D. Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th annual meeting of the association for computational linguistics: Short papers, vol. 2, 90–94. Association for Computational Linguistics.
Wang, Rui, Wei Liu, and Chris McDonald. 2017. A matrix-vector recurrent unit model for capturing compositional semantics in phrase embeddings. In International conference on information and knowledge management.
White, Lyndon, Roberto Togneri, Wei Liu, and Mohammed Bennamoun. 2015. How well sentence embeddings capture meaning. In Proceedings of the 20th Australasian document computing symposium. ADCS ’15, 9:1–9:8. Parramatta, NSW, Australia: ACM. ISBN: 978-1-4503-4040-3, https://doi.org/10.1145/2838931.2838932.
White, Lyndon, Roberto Togneri, Wei Liu, and Mohammed Bennamoun. 2016a. Generating bags of words from the sums of their word embeddings. In 17th international conference on intelligent text processing and computational linguistics (CICLing).
White, Lyndon, Roberto Togneri, Wei Liu, and Mohammed Bennamoun. 2016b. Modelling sentence generation from sum of word embedding vectors as a mixed integer programming problem. In IEEE international conference on data mining: High dimensional data mining workshop (ICDM: HDM). https://doi.org/10.1109/ICDMW.2016.0113.
White, L., R. Togneri, W. Liu, and M. Bennamoun. 2017. Learning distributions of meant color. arXiv:1709.09360 [cs.CL].
Zhang, Jiajun, Shujie Liu, Mu Li, Ming Zhou, and Chengqing Zong. 2014. Bilingually constrained phrase embeddings for machine translation. In ACL.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
White, L., Togneri, R., Liu, W., Bennamoun, M. (2019). Sentence Representations and Beyond. In: Neural Representations of Natural Language. Studies in Computational Intelligence, vol 783. Springer, Singapore. https://doi.org/10.1007/978-981-13-0062-2_5
Download citation
DOI: https://doi.org/10.1007/978-981-13-0062-2_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-0061-5
Online ISBN: 978-981-13-0062-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)