Conclusions

Abstract

In this book, we aimed at providing a high-level introduction to various types of embeddings used in NLP. We covered early works in word embeddings and more recent contextualized embeddings based on large pre-trained language models. The currently celebrated contextualized embeddings are the product of a long path of evolution. Since early works on lexical semantics, the distributional hypothesis has been the dominating basis for the field of semantic representation and prevailed even for recent models, however, the way of constructing representations has gone under a lot of change. The initial stage of this path is characterized by models that explicitly collected co-occurrence statistics, an approach that often required a subsequent dimensionality reduction step (Chapter 3). Together with the revival of neural networks and deep learning, the field of semantic representation experienced a massive boost. Neural networks provided an efficient way for processing large amounts of texts and for directly computing dense compact representations. Since then, the term representation has been almost fully substituted by their dense version, called embeddings. This development path has revolutionalized other fields of research such as graph embedding (Chapter 4) or resulted in the emergence of other fields of research, such as sense embedding (Chapter 5) and sentence embedding (Chapter 7).