Contextualized Embeddings

Abstract

This chapter provides an introduction to contextualized word embeddings which can be considered the new generation of word (and sense) embeddings. The distinguishing factor here is the sensitivity of a word’s representation to the context: a target word’s embedding can change depending on the context in which it appears. These dynamic embeddings alleviate many of the issues associated with static word embeddings and provide reliable means for capturing semantic and syntactic properties of word usage in context. Despite their young age, contextualized word embeddings have provided significant gains in almost any downstream NLP task to which they have been applied.