Combining Neural Language Models for Word Sense Induction

Abstract

Word sense induction (WSI) is the problem of grouping occurrences of an ambiguous word according to the expressed sense of this word. Recently a new approach to this task was proposed, which generates possible substitutes for the ambiguous word in a particular context using neural language models, and then clusters sparse bag-of-words vectors built from these substitutes. In this work, we apply this approach to the Russian language and improve it in two ways. First, we propose methods of combining left and right contexts, resulting in better substitutes generated. Second, instead of fixed number of clusters for all ambiguous words we propose a technique for selecting individual number of clusters for each word. Our approach established new state-of-the-art level, improving current best results of WSI for the Russian language on two RUSSE 2018 datasets by a large margin.

Appendices

A Examples of Substitutes Generated

Table 5 provides examples of discriminative substitutes with their relative frequencies for each of two most frequent senses of several words. A substitute is called discriminative if it is frequently generated for one sense of an ambigusous word, but rarely for another. Formally, we take substitutes with the largest $\frac{P(w|sense_1)}{P(w|sense_2)}$, where $P(w|sense_i)$ is estimated using add-one smoothing:

$$P(w|sense_i) = \frac{cnt(w|sense_i) + 1}{cnt(sense_i)+|vocab|}$$

Additionally, we leave only substitutes which were generated at lest 10 times for one of the senses.

Table 5. Discriminative substitutes for several words from bts-rnc train

Table 6 lists ten most probable substitutes according to the combined distribution and according to the forward and the backward LM distributions separately for several examples. Substitutes from unidirectional distributions are very sensitive to the position of the target word. When either left or right context doesn’t contain enough information at least halve of the substitutes will be not related to the target word. Combined distribution provides more relevant substitutes.

Table 6. Substitutes generated for randomly selected examples.

B The Number of Clusters Selected

Figure 4 plots distributions of the differences between the true number of senses, the number of clusters in submissions and the optimal number of clusters. Silhouette score gives the number of clusters, which is usually larger than the number of senses, but is near the optimum with respect to ARI and given our vectors. The previous best submissions better estimate the true number of senses.

Fig. 4.

Comparison of the number of clusters in our (silnc) and previous best submissions (prev_best_nc) with the true number of senses (true_nc) and the optimal number of clusters (max_ari_nc).

Table 7. Selected hyperparameters

C Hyperparameters

Table 7 shows the selected hyperparameters for the methods described in Sect. 3. For bts-rnc and active-dict datasets hyperparameters were selected using grid search on corresponding train sets. For wiki-wiki we used the hyperparameters from bts-rnc due to very small size of wiki-wiki train set. We selected the following hyperparameters.

• 1. Add bias (True/False). Ignoring bias in the softmax layer of the LM was proposed by [3] to improve substitutes, because adding bias results in prediction of frequency words instead of rare but relevant substitutes.

• 2. Normalize output embeddings (True/False). Similarly to ignoring bias, this may result in prediction of more relevant substitutes.

• 3. K (10–400). The number of substitutes from each distribution.

• 4. Exclude Target (True/False). We want the substitutes for different senses of the target word to be non-overlapping. Thus, it may be beneficial to exclude the target word from the substitutes.

• 5. TFIDF (True/False). Applying TFIDF transformation to bag-of-words vectors of substitutes sometimes improve performance.

• 6. S (=20). The number of representatives for each example. It didn’t affect the performance so we use the value from [3].

• 7. L (4–30). The number of substitutes to sample from top K predictions.

• 8. z (1.0–3.0). The parameter of Zipf distribution.

• 9. $\varvec{\beta }$ (0.1–0.5). Relative length of the left or the right context after which the discounting of the corresponding LM begins.