On the Replicability of Combining Word Embeddings and Retrieval Models

The last 5 years have seen proof that neural network-based word embedding models provide term representations that are a useful information source for a variety of tasks in natural language processing. In information retrieval (IR), “traditional” models remain a high baseline to beat, particularly when considering efficiency in addition to effectiveness [6]. Combining the word embedding models with the traditional IR models is therefore very attractive and several papers have attempted to improve the baseline by adding in, in a more or less ad-hoc fashion, word-embedding information. Onal et al. [10] summarized the various developments of the last half-decade in the field of neural IR and group the methods in two categories: aggregate and learn. The first one, also known as compositional distributional semantics, starts from term representations and uses some function to combine them into a document representation (a simple example is a weighted sum). The second method uses the word embedding as a first layer of another neural network to output a document representation.

The advantage of the first type of methods is that they often distill down to a linear combination (perhaps via a kernel), from which an explanation about the representation of the document is easier to induce than from the neural network layers built on top of a word embedding. Recently, the issue of explainability in IR and recommendation is generating a renewed interest [15].

In this sense, Zhang et al. [14] introduced a new model for combining high-dimensional vectors, using a mixture model of von Mises-Fisher (VMF) instead of Gaussian distributions previously suggested by Clinchant and Perronnin [3]. This is an attractive hypothesis because the Gaussian Mixture Model (GMM) works on Euclidean distance, while the mixture of von Mises-Fisher (moVMF) model works on cosine distances—the typical distance function in IR.

In the following sections, we set up to replicate the experiments described by Zhang et al. [14]. They are grouped in three sets: classification, clustering, and information retrieval, and compare “standard” embedding methods with the novel moVMF representation.

In general, we follow the experimental setup of the original paper and, for lack of space, we do not repeat here many details, if they are clearly explained there.

Each of the following experiments is conceptually divided in three phases. First, text processing (e.g. tokenisation); second, creating a fixed-length vector representation for every document; finally, the third phase is determined by the goal to be achieved, i.e. classification, clustering, and retrieval.

For the first phase the same pre-processing is applied to all datasets. In the original paper, this phase was only briefly described as tokenisation and stop-word removal. It is not given what tokeniser, linguistic filters (stemming, lemmatisation, etc.), or stop word list were used. Knowing that the gensim library was used, we took all standard parameters (see provided code⁶). Gensim however does not come with a pre-defined stopword list, and therefore, based on our own experience, we used the one provided in the NLTK library⁷ for English.

For the second phase, transforming terms and documents to vectors, Zhang et al. [14] specify that all trained models are 50 dimensional. We have additionally experimented with dimensionality 20 (used by Clinchant and Perronnin [3] for clustering) and 100, as we hypothesized that 50 might be too low. The TF-IDF model is 5000 dimensional (i.e. only the top 5000 terms based on their tf-idf value are used), while the Fischer-Kernel models are \(15 \times d\) dimensional, where \(d = \{ 20, 50, 100 \}\), as just explained. In what follows, d refers to the dimensionality of LSI, LDA, cBow, and PV models.

The cBoW and PV models are trained using a default window size of 5, keeping both low and high-frequency terms, again following the setup of the original experiment. The LDA model is trained using a chunk size of 1000 documents and for a number of iterations over the corpus ranging from 20 to 100. For the FK methods, both fitting procedures (GMM and moVMF) are independently initialised 10 times and the best fitting model is kept.

For the third phase, parameters are explained in the following sections.

3.1 Classification

Logistic regression is used for classification in Zhang et al., and therefore also used here. The results of our experiments, for \(d = 50\) and 100-dimensional feature vectors, are summarised in Table 1. For all the methods, we perform a parameter scan of the (inverse) regularisation strength of the logistic regression classifier, as shown in Fig. 1(a) and (b). Additionally, the learning algorithms are trained for a different number of epochs and the resulting classification accuracy assessed, cf. Fig. 1(c) and (d).

Table 1.

Results of classification experiments on the subj and sent datasets. Shown are the mean accuracy and standard deviation, under 10-fold cross-validation, for optimally chosen hyperparameters (i.e. top values in Fig. 1).

Model	50-dim.		100-dim.
	Subj	Sent	Subj	Sent
TF-IDF	89.3 ± 0.7	75.9 ± 1.3	89.3 ± 0.7	75.9 ± 1.3
LSI	82.5 ± 1.0	63.2 ± 0.9	84.3 ± 1.0	65.4 ± 1.6
LDA	62.2 ± 1.7	56.5 ± 1.6	62.1 ± 1.0	57.1 ± 1.5
cBow	89.0 ± 1.1	70.1 ± 1.3	89.2 ± 1.6	71.4 ± 1.1
PV-DBOW	88.7 ± 1.1	65.8 ± 1.4	89.4 ± 0.8	68.3 ± 1.3
PV-DM	84.0 ± 1.5	68.6 ± 1.3	88.0 ± 0.9	70.3 ± 1.1
FV-GMM	89.1 ± 0.9	68.5 ± 1.5	88.7 ± 0.9	72.4 ± 1.1
FV-moVMF	89.5 ± 0.9	70.1 ± 0.9	88.7 ± 1.4	71.2 ± 1.5

Open image in new window

Figure 1(a) indicates that cBow, FV-GMM, FV-moVMF, and the simple TF-IDF, when properly tuned, exhibit a very similar accuracy on subj—the given confidence intervals do not indeed allow us to identify a single, best model. Surprisingly, TF-IDF outperforms all the others on the sent dataset (Fig. 1(b)). Increasing the dimensionality of the feature vectors, from \(d = 50\) to 100, has the effect of reducing the gap between TF-IDF and the rest of the models on the sent dataset (see Table 1).

We replicated previously reported experiments that presented evidence that a new mixture model, based on von Mises-Fisher distributions, outperformed a series of other models in three tasks (classification, clustering, and retrieval—when combined with standard retrieval models).

Since the source code was not released in the original paper, important implementation and formulation details were omitted, and the authors never replied to our request for information, a significant effort has been devoted to reverse engineer the experiments. In general, for none of the tasks were we able to confirm the conclusions of the previous experiments: we do not have enough evidence to conclude that FV-moVMF outperforms the other methods. The situation is rather different when considering the effectiveness of these document representations for clustering purposes: we find indeed that the FV-moVMF significantly underperforms, contradicting previous conclusions. In the case of retrieval, although Zhang et al.’s proposed method (FV-moVMF) indeed boosts BM25, it does not outperform most of the other models it was compared to.