An effective and interpretable method for document classification

Abstract

As the number of documents has been rapidly increasing in recent time, automatic text categorization is becoming a more important and fundamental task in information retrieval and text mining. Accuracy and interpretability are two important aspects of a text classifier. While the accuracy of a classifier measures the ability to correctly classify unseen data, interpretability is the ability of the classifier to be understood by humans and provide reasons why each data instance is assigned to a label. This paper proposes an interpretable classification method by exploiting the Dirichlet process mixture model of von Mises–Fisher distributions for directional data. By using the labeled information of the training data explicitly and determining automatically the number of topics for each class, the learned topics are coherent, relevant and discriminative. They help interpret as well as distinguish classes. Our experimental results showed the advantages of our approach in terms of separability, interpretability and effectiveness in classification task of datasets with high dimension and complex distribution. Our method is highly competitive with state-of-the-art approaches.

Appendices

Appendix

1.1 Lower bound function

The log likelihood of the corpus is bounded by lower bound when using Jensens inequality

$$\begin{aligned}&\log P\left( \mathbf X ^v,\mathbf Z ^v|\alpha _{v},\zeta _{v},\{\kappa _{v,t}\}_{v=1,t=1}^{V,T},\rho _{v}\right) \nonumber \\&\quad \ge E_{q}[\log P(\mathbf X ^v,\mathbf Z ^v,\mathbf U _v,\varvec{\mu }_v)] - E_{q}[\log q((\mathbf U _v,\varvec{\mu }_v,\mathbf Z ^v|\gamma _v,\tilde{\mu }_v,\tilde{\kappa }_v,\phi )] \nonumber \\ \end{aligned}$$

(30)

We expand this lower bound as follows

$$\begin{aligned} L(\varvec{\gamma }_{v},\tilde{\varvec{\mu }}_{v},\tilde{\varvec{\kappa }}_{v},\varvec{\phi })&= E_{q}[\log P(\mathbf X ^v|\mathbf Z ^v,\varvec{\kappa }_{v},\varvec{\mu }_v))] + E_{q}[\log P(\mathbf Z ^v|\mathbf U ^v)] \nonumber \\&\quad + E_{q}[\log P(\mathbf U _v|\alpha _v)] + E_{q}[\log P(\varvec{\mu }_v |\zeta _{v},\rho _{v})] - E_{q}[\log q(\mathbf U _v|\varvec{\gamma }_{v})]\nonumber \\&\quad - E_{q}\left[ \log q(\mathbf Z ^v|\varvec{\phi })\right] \nonumber \\&\quad - E_{q}\left[ \log q(\varvec{\mu }_v |\tilde{\varvec{\mu }}_{v},\tilde{\varvec{\kappa }}_{v})\right] = \sum _{n=1}^{N_v}(E_{q}[\log P(x_n^v|z_n^v,\varvec{\kappa }_{v},\varvec{\mu }_v))]\nonumber \\&\quad + E_{q}\left[ \log P(z_n^v|\mathbf U ^v)\right] ) \nonumber \\&\quad + \sum _{t=1}^{\infty }E_{q}[\log P(u_{v,t}|\alpha _v)] + \sum _{t=1}^{\infty }E_{q}[\log P(\mu _{v,t} |\zeta _{v},\rho _{v})]\nonumber \\&\quad - \sum _{t=1}^{\infty }E_{q}[\log q(u_{v,t}|\gamma _{v,t_1},\gamma _{v,t_2})] \nonumber \\&\quad - \sum _{n=1}^{N_v}E_{q}[\log q(z_n^v|\phi _n)] - \sum _{t=1}^{\infty }E_{q}[\log q(\mu _{v,t} |\tilde{\mu }_{v,t},\tilde{\kappa }_{v,t})] \end{aligned}$$

(31)

We note that $q(z_{n}^v>T)=0$ and $E_q[\log (1-u_{v,T})]=0$ when truncating topics. Therefore,

$$\begin{aligned} L =&\sum _{n=1}^{N_v}\sum _{t=1}^{T} \phi _{n,t}E_q\left[ \log v\hbox {MF}(x_n|\mu _{n,t},\kappa _{v,t})\right] + \sum _{n=1}^{N_v} E_{q}\left[ \log (\prod _{t=1}^{T}(1-u_{v,t})^{1[z_{n}^v>t]}u_{v,t}^{1[z_n^v=t]})\right] \nonumber \\&+ \sum _{t=1}^{T}E_{q}[\log \hbox {beta}(u_{v,t}|1,\alpha _v)] + \sum _{t=1}^{T}E_{q}[\log v\hbox {MF}(\mu _{v,t} |\zeta _{v},\rho _{v})] \nonumber \\&-\sum _{t=1}^{T}E_{q}[\log \hbox {beta}(u_{v,t}|\gamma _{v,t_1},\gamma _{v,t_2})] - \sum _{n=1}^{N_v}\sum _{t=1}^{T} \phi _{n,t}\log \phi _{n,t}\nonumber \\&- \sum _{t=1}^{T}E_{q}[\log v\hbox {MF}(\mu _{v,t} |\tilde{\mu }_{v,t},\tilde{\kappa }_{v,t})] \nonumber \\&= \sum _{n=1}^{N_v}\sum _{t=1}^{T} \phi _{n,t}(\log C_d(\kappa _{v,t})+ \kappa _{v,t}A_d(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}x_{n}^v)\nonumber \\&+\sum _{n=1}^{N_v}\sum _{t=1}^{T}(\sum _{j=t+1}^{T} \phi _{n,j}(\varPsi (\gamma _{v,t_{2}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})) \nonumber \\&+ \phi _{n,t}(\varPsi (\gamma _{v,t_{1}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}}))) + \sum _{t=1}^{T}(\alpha _v-1)(\varPsi (\gamma _{v,t_{2}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})) \nonumber \\&+ \sum _{t=1}^{T}(\log C_d(\rho _{v})+\rho _{v}A_d(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}\zeta _{v}) + \sum _{t=1}^{T}((\gamma _{v,t_{1}}-1)(\varPsi (\gamma _{v,t_{1}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})) \nonumber \\&+ (\gamma _{v,t_{2}}-1)(\varPsi (\gamma _{v,t_{2}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})))- \sum _{n=1}^{N_v}\sum _{t=1}^{T} \phi _{n,t}\log \phi _{n,t} \nonumber \\&- \sum _{t=1}^{T}(\log C_d(\tilde{\kappa }_{v,t})+ \tilde{\kappa }_{v,t}A_d(\tilde{\kappa }_{v,t})) \end{aligned}$$

(32)

Now, we optimize this objective function to infer variational parameters and estimate hyperparameters.

Variational parameter $\phi _{n,t}$

We know that $\sum _{t=1}^T \phi _{n,t}=1$, and by isolating the terms that contain $\phi _{n}$ in objective function (as in Eq. 32) and adding the Lagrange multipliers $\lambda$, we obtain

$$\begin{aligned} L_{\phi _n}&= \sum _{t=1}^{T} \phi _{n,t}\left( \log C_d(\kappa _{v,t})+ \kappa _{v,t}A_d(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}x_{n}^v\right) \nonumber \\&\quad +\,\sum _{t=1}^{T}\left( \sum _{j=t+1}^{T} \phi _{n,j}\left( \varPsi (\gamma _{v,t_{2}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})\right) \right. \nonumber \\&\left. \quad +\,\phi _{n,t}\left( \varPsi (\gamma _{v,t_{1}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})\right) \right) -\sum _{t=1}^{T} \phi _{n,t}\log \phi _{n,t} + \lambda \left( \sum _{t=1}^T \phi _{n,t}-1\right) \end{aligned}$$

(33)

We compute the derivative with respect to $\phi _{n,t}$ as follows

$$\begin{aligned} \frac{\delta L_{\phi _{n,t}}}{\delta \phi _{n,t}} =&\left( \log C_d(\kappa _{v,t})+ \kappa _{v,t}A_d(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}x_{n}^v\right) +\sum _{t=1}^{T-1}(\varPsi (\gamma _{v,t_{2}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})) \nonumber \\&+ (\varPsi (\gamma _{v,t_{1}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}}))-1-\log \phi _{n,t} + \lambda \end{aligned}$$

(34)

When $\{\kappa _{v,t}\}_{t=1}^{T}$ are equally set and the Eq. 34 is set to zero, we have

$$\begin{aligned} \phi _{n,t} \propto \exp (S_{n,t}) \end{aligned}$$

(35)

where

$$\begin{aligned} S_{n,t}&= \kappa _{v,t} A_{d}(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}x_{n} + (\varPsi (\gamma _{v,t_{1}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})) \nonumber \\&\quad + \sum _{t=1}^{T-1}(\varPsi (\gamma _{v,t_{2}})-\varPsi (\gamma _{v,t_{1}}+\gamma _{v,t_{2}})) \end{aligned}$$

(36)

Variational parameter $\tilde{\mu }_{v,t}$

We maximize Eq. 32 with respect to $\tilde{\mu }_{v,t}$ subject to $||\tilde{\mu }_{v,t}||=1$. By adding the appropriate Lagrange multiplier, we have

$$\begin{aligned}&L_{\tilde{\mu }_{v,t}} = \sum _{n=1}^{N_v} \phi _{n,t}\kappa _{v,t}A_d(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}x_{n}^v + \rho _{v}A_d(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}\zeta _{v} - \lambda \left( \tilde{\mu }_{v,t}^{T}\tilde{\mu }_{v,t}-1\right) \end{aligned}$$

(37)

Taking derivatives with respect to $\tilde{\mu }_{v,t}$, we obtain

$$\begin{aligned} \frac{\delta L}{\delta \tilde{\mu }_{v,t}} = \sum _{n=1}^{N_v} \phi _{n,t}\kappa _{v,t}A_d(\tilde{\kappa }_{v,t})x_{n}^v + \rho _{v}A_d(\tilde{\kappa }_{v,t})\zeta _{v} - 2\lambda \tilde{\mu }_{v,t} \end{aligned}$$

(38)

Setting it to zero, we get

$$\begin{aligned} \tilde{\mu }_{v,t} = \frac{\sum _{n=1}^{N}\kappa _{v,t} \phi _{n,t}x_{n}+\rho _{v}A_d(\tilde{\kappa }_{v,t})\zeta _{v}}{\Vert \sum _{n=1}^{N}\kappa _{v,t} \phi _{n,t}x_{n}+\rho _{v}A_d(\tilde{\kappa }_{v,t})\zeta _{v}\Vert } \end{aligned}$$

(39)

Variational parameter $\tilde{\kappa }_{v,t}$

We maximize Eq. 32 with respect to $\tilde{\kappa }_{v,t}$.

$$\begin{aligned}&L_{\tilde{\kappa }_{v,t}} = \sum _{n=1}^{N_v} \phi _{n,t} \kappa _{v,t}A_d(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}x_{n}^v+ \rho _{v}A_d(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}\zeta _{v} - (\log C_d(\tilde{\kappa }_{v,t})+ \tilde{\kappa }_{v,t}A_d(\tilde{\kappa }_{v,t})) \end{aligned}$$

(40)

We take the derivative with respect to $\tilde{\kappa }_{v,t}$

$$\begin{aligned} \frac{\delta L}{\delta \tilde{\kappa }_{v,t}} =&\sum _{n=1}^{N_v} \phi _{n,t} \kappa _{v,t}A_d^{'}(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}x_{n}^v+ \rho _{v}A_d^{'}(\tilde{\kappa }_{v,t})\tilde{\mu }_{v,t}^{T}\zeta _{v} \nonumber \\&- \left( \frac{C_d^{'}(\tilde{\kappa }_{v,t})}{C_d(\tilde{\kappa }_{v,t})} +A_d(\tilde{\kappa }_{v,t})+\tilde{\kappa }_{v,t}A_d^{'}(\tilde{\kappa }_{v,t})\right) \end{aligned}$$

(41)

where $A_d^{'}(\tilde{\kappa }_{v,t})= \frac{\delta A_d(\tilde{\kappa }_{v,t})}{\delta \tilde{\kappa }_{v,t}}$ and $C_d^{'}(\tilde{\kappa }_{v,t})= \frac{\delta C_d(\tilde{\kappa }_{v,t})}{\delta \tilde{\kappa }_{v,t}}$. Setting this derivative to zero, we have:

$$\begin{aligned} \tilde{\kappa }_{v,t} = \sum _{n=1}^{N_v} \phi _{n,t} \kappa _{v,t}\tilde{\mu }_{v,t}^{T}x_{n}^v+ \rho _{v}\tilde{\mu }_{v,t}^{T}\zeta _{v} \end{aligned}$$

(42)