Hybrid generative discriminative approaches based on Multinomial Scaled Dirichlet mixture models

Abstract

Developing both generative and discriminative techniques for classification has achieved significant progress in the last few years. Considering the capabilities and limitations of both, hybrid generative discriminative approaches have received increasing attention. Our goal is to combine the advantages and desirable properties of generative models, i.e. finite mixture, and the Support Vector Machines (SVMs) as powerful discriminative techniques for modeling count data that appears in many domains in machine learning and computer vision applications. In particular, we select accurate kernels generated from mixtures of Multinomial Scaled Dirichlet distribution and its exponential approximation (EMSD) for support vector machines. We demonstrate the effectiveness and the merits of the proposed framework through challenging real-world applications namely; object recognition and visual scenes classification. Large scale datasets have been considered in the empirical study such as Microsoft MOCR, Fruits-360 and MIT places.

Appendices

Appendix A: Proof of (5)

The composition of the Multinomial and Scaled Dirichlet is obtained by the following integration:

$$\begin{array}{@{}rcl@{}} \mathcal{M}\mathcal{S}\mathcal{D}(\mathbf{X}|\boldsymbol{\alpha},\boldsymbol{\beta} )&=& {\int}_{\rho} \mathcal{M}(\mathbf{X}|\boldsymbol{\rho}) \mathcal{S}\mathcal{D}(\boldsymbol{\rho}|\boldsymbol{\alpha}, \boldsymbol{\beta}) d\rho \\ &=& {\int}_{\rho} \frac{n!}{\prod\limits_{d = 1}^{D} x_{d}!} \prod\limits_{d = 1}^{D} \rho_{d}^{x_{d}} \frac{{\Gamma} (A)}{\prod\limits_{d = 1}^{D} {\Gamma}(\alpha_{d})} \frac{\prod\limits_{d = 1}^{D} \beta_{d}^{\alpha_{d}} p_{d}^{\alpha_{d}-1}}{\left( \sum\limits_{d = 1}^{D} \beta_{d} \rho_{d} \right)^{A}} d\rho \end{array} $$

$$\begin{array}{@{}rcl@{}} &=& \frac{n!}{\prod\limits_{d = 1}^{D} x_{d}!}\frac{{\Gamma} (A)}{\prod\limits_{d = 1}^{D} {\Gamma}(\alpha_{d})} \prod\limits_{d = 1}^{D} \beta_{d}^{\alpha_{d}} {\int}_{\rho} \frac{\prod \limits_{d = 1}^{D} \rho_{d}^{x_{d}+\alpha_{d}-1}}{\left( \sum\limits_{d = 1}^{D} \beta_{d} \rho_{d} \right)^{A}} d\rho\\ \end{array} $$

(29)

Using the fact that the integration of the PDF = 1, we have: ${\int }_{\rho } \mathcal {M}\mathcal {S}\mathcal {D}(\rho |\theta ) d\rho = 1$, straightforward manipulation yield:

$$\begin{array}{@{}rcl@{}} && {\int}_{\rho} \frac{{\Gamma} \left( \sum\limits_{d = 1}^{D} \alpha_{d}\right)}{\prod\limits_{d = 1}^{D} {\Gamma}(\alpha_{d})} \frac{\prod\limits_{d = 1}^{D} \beta_{d}^{\alpha_{d}} \rho_{d}^{\alpha_{d}-1}}{\left( \sum\limits_{d = 1}^{D} \beta_{d} \rho_{d} \right)^{A}} d\rho = 1 \\ && \frac{{\Gamma} \left( \sum\limits_{d = 1}^{D} \alpha_{d}\right) \prod\limits_{d = 1}^{D} \beta_{d}^{\alpha_{d}}}{\prod\limits_{d = 1}^{D} {\Gamma}(\alpha_{d})} {\int}_{\rho} \frac{\prod\limits_{d = 1}^{D} \rho_{d}^{\alpha_{d}-1}}{\left( \sum\limits_{d = 1}^{D} \beta_{d} \rho_{d} \right)^{A}} d\rho = 1 \end{array} $$

(30)

and we solve the integration using the following empirically found approximation: $\left ({\sum }_{d = 1}^{D} \beta _{d} \ \rho _{d}\right )^{{\sum }_{d = 1}^{D} x_{d}} \simeq {\prod }_{d = 1}^{D} \beta _{d}^{x_{d}}$, as:

$$\begin{array}{@{}rcl@{}} {\int}_{\rho} \frac{\prod\limits_{d = 1}^{D} \rho_{d}^{\alpha_{d}-1}}{\left( \sum\limits_{d = 1}^{D} \beta_{d} \rho_{d} \right)^{A}} d\rho = \frac{\prod\limits_{d = 1}^{D} {\Gamma}(\alpha_{d})}{{\Gamma} \left( \sum\limits_{d = 1}^{D} \alpha_{d}\right) \prod\limits_{d = 1}^{D} \beta_{d}^{\alpha_{d}}} \end{array} $$

(31)

Using this to solve the integration in (29), we obtain (5).

Appendix B: Newton Raphson approach

The complete data log likelihood corresponding to a K-component mixture is given by:

$$ \mathcal{L}(\mathcal{X},\mathcal{Z}|{\Theta})=\sum\limits_{k = 1}^{K} \sum\limits_{i = 1}^{N} z_{ik} \left( \log \pi_{k} + \log p(\mathbf{X}_{i}|\theta_{k}) \right) $$

(32)

By computing the second and mixed derivatives of $\mathcal {L}(\mathcal {X},\mathcal {Z}|{\Theta })$ with respect to $\alpha _{kd},\ d = 1,\dots ,D$, we obtain:

$$\begin{array}{@{}rcl@{}} &&\frac{\partial^{2} \mathcal{L}(\mathcal{X},\mathcal{Z}|{\Theta})}{\partial\alpha_{kd1}\partial\alpha_{kd2}} = \\ &&\left\{\begin{array}{ll} \sum\limits_{i = 1}^{N} z_{ik} \left( {\Psi}^{\prime}(A)-{\Psi}^{\prime}(n_{i}+A)\right. \\ \left. +{\Psi}^{\prime}(x_{id}+\alpha_{kd})-{\Psi}^{\prime}(\alpha_{kd}) \right) &\text{if}\quad d_{1}=d_{2}=d, \\ \sum\limits_{i = 1}^{N} z_{ik} \left( {\Psi}^{\prime}(A)-{\Psi}^{\prime}(n_{i}+A) \right) & \text{otherwise,} \end{array}\right. \end{array} $$

(33)

where ${\Psi }^{\prime }$ is the trigamma function. By computing the second and mixed derivatives of $\mathcal {L}(\mathcal {X},\mathcal {Z}|{\Theta })$ with respect to $\beta _{kd},\ d = 1,\dots ,D$, we obtain:

$$ \frac{\partial^{2} \mathcal{L}(\mathcal{X},\mathcal{Z}|{\Theta})}{\partial\beta_{kd1}\partial\beta_{kd2}} = \left\{\begin{array}{ll} \sum\limits_{i = 1}^{N} z_{ik} \left( \frac{x_{id}}{\beta_{kd}^{2}} \right) & \text{if}\ d_{1}=d_{2}=d, \\ \\ 0 & \text{otherwise,} \end{array}\right. $$

(34)

The second and mixed derivatives of $\mathcal {L}(\mathcal {X},\mathcal {Z}|{\Theta })$ with respect to α_kd and β_kd, $d = 1,\dots ,D$, is 0.

Appendix C: Proof of (19)

The KL-divergence between two exponential distributions is given by [40]:

$$\begin{array}{@{}rcl@{}} KL(p(X|{\Theta}),p^{\prime}(X|{\Theta}^{\prime}))&=&{\Phi}(\theta)-{\Phi}(\theta^{\prime})\\&&+[G(\theta)-G(\theta^{\prime})]^{tr} E_{\theta}[T(X)]\\ \end{array} $$

(35)

where E_𝜃 is the expectation with respect to p(X|𝜃). Moreover, we have the following [16]:

$$ E_{\theta}[T(X)]=-{\Phi}^{\prime}(\theta) $$

(36)

Thus, according to (14), we have:

$$\begin{array}{@{}rcl@{}} E_{\theta} \left[\sum\limits_{d = 1}^{D} I(x_{d} \geq 1)\right]&=&-\frac{\partial {\Phi}(\theta)}{\partial \lambda_{d}} \\&=&{\Psi}\left( \sum\limits_{d = 1}^{D} \lambda_{d}+n\right) -{\Psi}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right) \\ E_{\theta} \left[\sum\limits_{d = 1}^{D} I(x_{d} \geq 1) x_{d}\right]&=&-\frac{\partial {\Phi}(\theta)}{\partial \nu_{d}} = 0 \end{array} $$

(37)

where $n={\sum }_{d = 1}^{D} x_{d}$, and Ψ(.) is the digamma function. By substituting the previous two equations into Eq.(35), we obtain:

$$\begin{array}{@{}rcl@{}} KL &&(p(X|{\Theta}),p^{\prime}(X|{\Theta}^{\prime})) \\ &=&\log \left( {\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right)\right)-\log \left( {\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda^{\prime}_{d}\right)\right)\\ &&-\log\left( {\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d} +n\right)\right)+\log \left( {\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda^{\prime}_{d} +n\right)\right) \\ &&+{\sum}_{d = 1}^{D} \left( {\Psi}\left( \sum\limits_{d = 1}^{D} \lambda_{d}+n\right) -{\Psi}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right) \right) (\lambda_{d}-\lambda^{\prime}_{d}) \\ &=& \log \left[\frac{{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda_{d}\right).{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda^{\prime}_{d} +n\right)}{{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda^{\prime}_{d}\right).{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda_{d} +n\right)}\right] \\ &&+{\sum}_{d = 1}^{D} \left( {\Psi}\left( \sum\limits_{d = 1}^{D} \lambda_{d}+n\right) -{\Psi}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right) \right) (\lambda_{d}-\lambda^{\prime}_{d})\\ \end{array} $$

(38)

Appendix D: Proof of (23)

In the case of the EMSD distribution, we can show that:

$$\begin{array}{@{}rcl@{}} {\int}_{0}^{+\infty} & p&(\mathbf{X}|{\Theta})^{\sigma} p^{\prime}(\mathbf{X}|{\Theta}^{\prime})^{1-\sigma} dX= \\ && \left[\frac{{\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right)}{{\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}+n\right)}\right]^{\sigma} \left[\frac{{\Gamma}\left( \sum\limits_{d = 1}^{D} {\lambda^{\prime}_{d}}\right)}{{\Gamma}\left( \sum\limits_{d = 1}^{D} {\lambda^{\prime}_{d}}+\sum\limits_{d = 1}^{d} x_{d}\right)}\right]^{1-\sigma} \\ &\times& {\int}_{0}^{+\infty} \left[\frac{n!}{\prod\limits_{d = 1}^{D} x_{d}} \prod\limits_{d = 1}^{D} \frac{ \lambda_{d}}{\nu_{d}^{x_{d}}}\right]^{\sigma} dX \\ &\times& {\int}_{0}^{+\infty} \left[\frac{n!}{\prod\limits_{d = 1}^{D} x_{d}} \prod\limits_{d = 1}^{D} \frac{\lambda^{\prime}_{d}}{{\nu^{\prime}_{d}}^{x_{d}}}\right]^{1-\sigma} dX \\ &=&\left[\frac{{\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right) }{{\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}+\sum\limits_{d = 1}^{D} x_{d}\right)}\right]^{\sigma} \left[\frac{{\Gamma}\left( \sum\limits_{d = 1}^{D} {\lambda^{\prime}_{d}}\right)}{{\Gamma}(s^{\prime}+n)}\right]^{1-\sigma} \\ &\times& {\int}_{0}^{+\infty} \frac{n!}{\prod\limits_{d = 1}^{D} x_{d}} \prod\limits_{d = 1}^{D} \lambda_{d} \nu_{d}^{-\sigma x_{d}} dX \\ &\times& {\int}_{0}^{+\infty} \frac{n!}{\prod\limits_{d = 1}^{D} x_{d}} \prod\limits_{d = 1}^{D} \lambda^{\prime}_{d}{\nu^{\prime}_{d}}^{-x_{d}+\sigma x_{d}} dX \end{array} $$

(39)

We have the PDF of an EMSD distribution that integrates to one which gives:

$$ {\int}_{0}^{+\infty} \frac{n!}{{\prod}_{d = 1}^{D} x_{d}} \prod\limits_{d = 1}^{D} \frac{\lambda_{d}}{\nu_{d}^{x_{d}}}dX=\frac{{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda_{d} + {\sum}_{d = 1}^{D} x_{d}\right)}{{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda_{d}\right) } $$

(40)

By substituting (40) into (39), we obtain:

$$\begin{array}{@{}rcl@{}} {\int}_{0}^{+\infty} & p&(\mathbf{X}|{\Theta})^{\sigma} p^{\prime}(\mathbf{X}|{\Theta}^{\prime})^{1-\sigma} dX= \\ && \left[\frac{{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda_{d}\right) }{{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda_{d}+n\right)}\right]^{\sigma} \left[\frac{{\Gamma}\left( {\sum}_{d = 1}^{D} {\lambda^{\prime}_{d}}\right)}{{\Gamma}\left( {\sum}_{d = 1}^{D} {\lambda^{\prime}_{d}}+{\sum}_{d = 1}^{d} x_{d}\right)}\right]^{1-\sigma} \\ &\times& \frac{{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda_{d}+{\sum}_{d = 1}^{D} -\sigma x_{d}\right)}{{\Gamma}\left( {\sum}_{d = 1}^{D} \lambda_{d}\right)}\\ &\times& \frac{{\Gamma}\left( {\sum}_{d = 1}^{D} {\lambda^{\prime}_{d}}+{\sum}_{d = 1}^{D} -x_{d}+\sigma x_{d}\right)}{{\Gamma}\left( {\sum}_{d = 1}^{D} {\lambda^{\prime}_{d}}\right)} \end{array} $$

(41)

Appendix E: Proof of (27)

$$\begin{array}{@{}rcl@{}} H[p(\mathbf{X}|{\Theta})]&=&- {\int}_{0}^{+ \infty} p(\mathbf{X}|{\Theta}) \log p(\mathbf{X}|{\Theta}) dX \\ &=&- {\int}_{0}^{+ \infty} p(\mathbf{X}|{\Theta}) \left[\log {\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right) \right.\\&&\left.-\log {\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}+n\right)\right. \\ &&+\sum\limits_{d = 1}^{D} \log(\lambda_{d}) E_{\theta}[I(x_{d} \geq 1)] \\ &&\left.-\sum\limits_{d = 1}^{D} \log(\nu_{d}) E_{\theta} [I(x_{d} \geq 1) x_{d}]\right] \end{array} $$

(42)

By substituting (37) into the previous equation, we obtain the following:

$$\begin{array}{@{}rcl@{}} H[p(\mathbf{X}|{\Theta})]&=&-\log {\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right) +\log {\Gamma}\left( \sum\limits_{d = 1}^{D} \lambda_{d}+n\right) \\ &-&\sum\limits_{d = 1}^{D} \log(\lambda_{d}) \left( {\Psi}\left( \sum\limits_{d = 1}^{D} \lambda_{d}+n\right) \right.\\&&\left.-{\Psi}\left( \sum\limits_{d = 1}^{D} \lambda_{d}\right) \right) \end{array} $$

(43)