Interpretable Topic Extraction and Word Embedding Learning Using Row-Stochastic DEDICOM

Abstract

The DEDICOM algorithm provides a uniquely interpretable matrix factorization method for symmetric and asymmetric square matrices. We employ a new row-stochastic variation of DEDICOM on the pointwise mutual information matrices of text corpora to identify latent topic clusters within the vocabulary and simultaneously learn interpretable word embeddings. We introduce a method to efficiently train a constrained DEDICOM algorithm and a qualitative evaluation of its topic modeling and word embedding performance.

L. Hillebrand and D. Biesner—First author equal contribution.

Appendix

Table 3. Overview of our semi-artifical dataset. Each synthetic sample consists of the corresponding Wikipedia articles 1–3. We differentiate between different articles, i.e. articles that have little thematical overlap (for example a person and a city, a fish and an insect or a ball game and a combat sport), and similar articles, i.e. articles with large thematical overlap (for example European countries, tech companies or aquatic animals). We group our dataset into different samples (3 articles that are pairwise different), similar samples (3 articles that are all similar) and mixed samples (2 similar articles, 1 different).

Table 4. For each evaluated matrix factorization method we display the top 10 words for each topic and the 5 most similar words based on cosine similarity for the 2 top words from each topic.

Table 5. Top half lists the top 10 representative words per dimension of the basis matrix A, bottom half lists the 5 most similar words based on cosine similarity for the 2 top words from each topic.

Fig. 5.

(a) 2-dimensional representation of word embeddings $\textit{\textbf{A}}'$ colored by topic assignment. (b) 2-dimensional representation of word embeddings $\textit{\textbf{A}}'$ colored by original Wikipedia article assignment (words that occur in more than one article are excluded). (c) Colored heatmap of affinity matrix $\textit{\textbf{R}}$.

Table 6. For each evaluated matrix factorization method we display the top 10 words for each topic and the 5 most similar words based on cosine similarity for the 2 top words from each topic.

Table 7. Top half lists the top 10 representative words per dimension of the basis matrix A, bottom half lists the 5 most similar words based on cosine similarity for the 2 top words from each topic.

Fig. 6.

(a) 2-dimensional representation of word embeddings $\textit{\textbf{A}}'$ colored by topic assignment. (b) 2-dimensional representation of word embeddings $\textit{\textbf{A}}'$ colored by original Wikipedia article assignment (words that occur in more than one article are excluded). (c) Colored heatmap of affinity matrix $\textit{\textbf{R}}$.

Table 8. For each evaluated matrix factorization method we display the top 10 words for each topic and the 5 most similar words based on cosine similarity for the 2 top words from each topic.