Weakly supervised prototype topic model with discriminative seed words: modifying the category prior by self-exploring supervised signals

Abstract

Dataless text classification, i.e., a new paradigm of weakly supervised learning, refers to the task of learning with unlabeled documents and a few predefined representative words of categories, known as seed words. The recent generative dataless methods construct document-specific category priors by using seed word occurrences only; however, such category priors often contain very limited and even noisy supervised signals. To remedy this problem, in this paper, we propose a novel formulation of category prior. First, for each document, we consider its label membership degree by not only counting seed word occurrences, but also using a novel prototype scheme, which captures pseudo-nearest neighboring categories. Second, for each label, we consider its frequency prior knowledge of the corpus, which is also a discriminative knowledge for classification. By incorporating the proposed category prior into the previous generative dataless method, we suggest a novel generative dataless method, namely Weakly Supervised Prototype Topic Model. The experimental results on real-world datasets demonstrate that Wsptm outperforms the existing baseline methods.

Appendix

In this Appendix, we derive the key update equations of Wsptm.

The posterior of topic assignment of each word token, i.e., Eqs.10and 11: By holding $\{\theta ,{\widehat{\theta }}, \phi , {\widehat{\phi }}\}$ fixed, for each word token $w_{dn}$, the joint distributions of the word token and topic assignments are given as follows:

$$\begin{aligned} p\left( w_{dn}, c_{dn}=1, \, z_{dn}=k\right)= & {} \theta _d^T \gamma _{w_{dn}} \, \theta _{dk} \phi _{kw_{dn}}, \end{aligned}$$

(18)

$$\begin{aligned} p\left( w_{dn},c_{dn}=0, \, {{\widehat{z}}}_{dn}=g\right)= & {} \left( 1-\theta _d^T\gamma _{w_{dn}}\right) {\widehat{\theta }}_{dg} {\widehat{\phi }}_{gw_{dn}} \end{aligned}$$

(19)

The marginal probabilities are given as follows:

$$\begin{aligned} p\left( w_{dn}, c_{dn}=1\right)= & {} \theta _d^T\gamma _{w_{dn}} \sum \limits _{i=1}^K\theta _{di}\phi _{iw_{dn}}, \end{aligned}$$

(20)

$$\begin{aligned} p\left( w_{dn},c_{dn}=0\right)= & {} \left( 1-\theta _d^T\gamma _{w_{dn}}\right) \sum \limits _{i=1}^G{\widehat{\theta }}_{di}{\widehat{\phi }}_{iw_{dn}} \end{aligned}$$

(21)

With the above formulas, we can estimate the posteriors of topic assignments by applying the Bayes rule:

$$\begin{aligned} p\left( z_{dn}=k\right)= & {} \theta _d^T\gamma _{w_{dn}} \frac{\theta _{dk}\phi _{kw_{dn}}}{\sum \nolimits _{i=1}^K \theta _{di}\phi _{iw_{dn}}} \buildrel \Delta \over = N_{dnk}, \end{aligned}$$

(22)

$$\begin{aligned} p\left( {{\widehat{z}}}_{dn}=g\right)= & {} \left( 1-\theta _d^T\gamma _{w_{dn}}\right) \frac{\widehat{\theta }_{dg}{\widehat{\phi }}_{gw_{dn}}}{\sum \nolimits _{i=1}^G \widehat{\theta }_{di}\phi _{iw_{dn}}} \buildrel \Delta \over = {{\widehat{N}}}_{dng} \end{aligned}$$

(23)

We kindly emphasize that we omit the notations $c_{dn}=1$ and $c_{dn}=0$ in Eqs.22 and 23 to make equations simple.

The update equations of $\{\widehat{\theta }, \phi , {\widehat{\phi }}\}$, i.e., Eqs.12, 13, 14, and the “initialization” equation of $\theta$, i.e., Eq. 15: Given the current $\{N_{dnk}\}_{k=1}^K, \{\widehat{N}_{dng}\}_{g=1}^G$ of all word tokens and pre-computed word weights $\pi$, we can use them to generate soft occurrence numbers of distribution-specific samples. We take ${\widehat{\theta }}$ as an example. For each document d, we can regard $\{\sum \nolimits _{n=1}^{N_d}\pi (w_{dn}){{\widehat{N}}}_{dng}\}_{g=1}^G$ as the soft occurrence numbers of background topics, while we known that ${\widehat{\theta }}$ is drawn from a Dirichlet prior ${\widehat{\alpha }}$. Accordingly, we can regard this as a Dirichlet-Multinomial estimation, directly giving the update equation of ${\widehat{\theta }}$ as follows:

$$\begin{aligned} {\widehat{\theta }}_{dg}= & {} \frac{\sum \nolimits _{n=1}^{N_d}\pi (w_{dn}){{\widehat{N}}}_{dng}+\widehat{\alpha }}{\sum \nolimits _{i=1}^G \sum \nolimits _{n=1}^{N_d}\pi (w_{dn}){{\widehat{N}}}_{dni}+ G{\widehat{\alpha }}} \end{aligned}$$

(24)

The formulas of $\{\theta , \phi , {\widehat{\phi }}\}$ are similar to that of ${\widehat{\theta }}$, so we omit details.