Marginal and simultaneous predictive classification using stratified graphical models

Abstract

An inductive probabilistic classification rule must generally obey the principles of Bayesian predictive inference, such that all observed and unobserved stochastic quantities are jointly modeled and the parameter uncertainty is fully acknowledged through the posterior predictive distribution. Several such rules have been recently considered and their asymptotic behavior has been characterized under the assumption that the observed features or variables used for building a classifier are conditionally independent given a simultaneous labeling of both the training samples and those from an unknown origin. Here we extend the theoretical results to predictive classifiers acknowledging feature dependencies either through graphical models or sparser alternatives defined as stratified graphical models. We show through experimentation with both synthetic and real data that the predictive classifiers encoding dependencies have the potential to substantially improve classification accuracy compared with both standard discriminative classifiers and the predictive classifiers based on solely conditionally independent features. In most of our experiments stratified graphical models show an advantage over ordinary graphical models.

Appendices

Appendix A: Proof of Theorem 1

To prove Theorem 1 it suffices to consider a single class $k$ and a single maximal clique in $G_L^k$. If the scores for the marginal and simultaneous classifiers are asymptotically equivalent for an arbitrary maximal clique and class it automatically follows that the scores for the whole system are asymptotically equivalent. We start by considering the simultaneous classifier. The training data ${\mathbf {X}}^R$ and test data ${\mathbf {X}}^T$ are now assumed to cover only one maximal clique of an SG in one class. Looking at $\log S_{\text {sim}}({\mathbf {X}}^T \mid {\mathbf {X}}^R)$ using (2) we get

$$\begin{aligned} \log S_{\text {sim}}({\mathbf {X}}^T \mid {\mathbf {X}}^R)&= \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \log \frac{\varGamma (\sum _{i = 1}^{k_{j}} \beta _{jil})}{\varGamma (n(\pi _{j}^{l}) + \sum _{i = 1}^{k_{j}} \beta _{jil})} \\&+ \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} \log \frac{\varGamma (n(x_{j}^{i} \mid \pi _{j}^{l}) + \beta _{jil})}{\varGamma (\beta _{jil})}. \end{aligned}$$

Using Stirling’s approximation, $\log \varGamma (x)=(x - 0.5) \log (x) - x$, this equals

$$\begin{aligned}&\sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \left( \left( \sum _{i = 1}^{k_{j}} \beta _{jil} - 0.5 \right) \log \left( \sum _{i = 1}^{k_{j}} \beta _{jil} \right) - \sum _{i = 1}^{k_{j}} \beta _{jil} \right) \\&\qquad - \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \left( \left( n(\pi _{j}^{l}) + \sum _{i = 1}^{k_{j}} \beta _{jil} - 0.5 \right) \log \left( n(\pi _{j}^{l}) + \sum _{i = 1}^{k_{j}} \beta _{jil} \right) - n(\pi _{j}^{l}) - \sum _{i = 1}^{k_{j}} \beta _{jil} \right) \\&\qquad + \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} \left( \left( n(x_{j}^{i} \mid \pi _{j}^{l}) + \beta _{jil} - 0.5 \right) \log \left( n(x_{j}^{i} \mid \pi _{j}^{l}) + \beta _{jil} \right) - n(x_{j}^{i} \mid \pi _{j}^{l}) - \beta _{jil} \right) \\&\qquad - \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} \left( (\beta _{jil} - 0.5) \log (\beta _{jil}) - \beta _{jil} \right) \\&\quad = - \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \left( \left( \sum _{i = 1}^{k_{j}} \beta _{jil} - 0.5 \right) \log \left( 1 + \frac{n(\pi _{j}^{l})}{\sum _{i = 1}^{k_{j}} \beta _{jil}} \right) + n(\pi _{j}^{l}) \log \left( n(\pi _{j}^{l}) + \sum _{i = 1}^{k_{j}} \beta _{jil} \right) \right) \\&\qquad + \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} \left( (\beta _{jil} - 0.5) \log \left( 1 + \frac{n(x_{j}^{i} \mid \pi _{j}^{l})}{\beta _{jil}} \right) \!+\! n(x_{j}^{i} \mid \pi _{j}^{l}) \log \left( n(x_{j}^{i} \mid \pi _{j}^{l}) + \beta _{jil} \right) \right) . \end{aligned}$$

When looking at the marginal classifier we need to summarize over each single observation ${\mathbf {X}}_h^T$. We use $h(\pi _{j}^{l})$ to denote if the outcome of the parents of variable $X_j$ belongs to group $l$ and $h(x_{j}^{i} \mid \pi _{j}^{l})$ to denote if the outcome of $X_j$ is $i$ given that the observed outcome of the parents belongs to $l$. Observing that $h(\pi _{j}^{l})$ and $h(x_{j}^{i} \mid \pi _{j}^{l})$ are either 0 or 1 we get the result

$$\begin{aligned}&\log S_{\text {mar}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) = \sum _{h=1}^n \log P({\mathbf {X}}_h^T \mid {\mathbf {X}}^R) \\&\quad = \sum _{h=1}^n \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \log \frac{\varGamma (\sum _{i = 1}^{k_{j}} \beta _{jil})}{\varGamma (h(\pi _{j}^{l}) + \sum _{i = 1}^{k_{j}} \beta _{jil})} + \sum _{h=1}^n \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} \log \frac{\varGamma (h(x_{j}^{i} \mid \pi _{j}^{l}) + \beta _{jil})}{\varGamma (\beta _{jil})} \\&\quad = - \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} n(\pi _{j}^{l}) \log \left( \sum _{i = 1}^{k_{j}} \beta _{jil} \right) + \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} n(x_{j}^{i} \mid \pi _{j}^{l}) \log (\beta _{jil}). \end{aligned}$$

Considering the difference $\log S_{\text {sim}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) - \log S_{\text {mar}}({\mathbf {X}}^T \mid {\mathbf {X}}^R)$ results in

$$\begin{aligned}&\log S_{\text {sim}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) - \log S_{\text {mar}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) \\&= - \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \left( \sum _{i = 1}^{k_{j}} \beta _{jil} - 0.5 \right) \log \left( 1 + \frac{n(\pi _{j}^{l})}{\sum _{i = 1}^{k_{j}} \beta _{jil}} \right) \\&\quad \,\, + \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} (\beta _{jil} - 0.5) \log \left( 1 + \frac{n(x_{j}^{i} \mid \pi _{j}^{l})}{\beta _{jil}}\right) \\&\quad \,\, + \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \left( n(\pi _{j}^{l}) \log \left( \sum _{i = 1}^{k_{j}} \beta _{jil} \right) - n(\pi _{j}^{l}) \log \left( n(\pi _{j}^{l}) + \sum _{i = 1}^{k_{j}} \beta _{jil} \right) \right) \\&\quad \,\, + \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} \left( n(x_{j}^{i} \mid \pi _{j}^{l}) \log \left( n(x_{j}^{i} \mid \pi _{j}^{l}) + \beta _{jil} \right) - n(x_{j}^{i} \mid \pi _{j}^{l}) \log (\beta _{jil}) \right) . \end{aligned}$$

We now make the assumption that all the limits of relative frequencies of feature values are strictly positive under an infinitely exchangeable sampling process of the training data, i.e. all hyperparameters $\beta _{jil} \rightarrow \infty$ when the size of the training data $m \rightarrow \infty$. Using the standard limit $\lim _{y \rightarrow \infty } (1+x/y)^y = e^x$ results in

$$\begin{aligned}&\lim _{m \rightarrow \infty } \log S_{\text {sim}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) - \log S_{\text {mar}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) \\&\quad = - \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} n(\pi _{j}^{l}) + \sum _{j = 1}^{d} \sum _{l = 1}^{q_{j}} \sum _{i = 1}^{k_{j}} n(x_{j}^{i} \mid \pi _{j}^{l}) = 0. \end{aligned}$$

Appendix B: Proof of Theorem 2

This proof follows largely the same structure as the proof of Theorem 1 and covers the simultaneous score. It is assumed that the underlying graph of the SGM coincides with the GM, this is a fair assumption since when the size of the training data goes to infinity this property will hold for the SGM and GM maximizing the marginal likelihood. Again we consider only a single class $k$ and a single maximal clique in $G_L^k$, using the same reasoning as in the proof above. Additionally, it will suffice to consider the score for the last variable $X_d$ in the ordering, the variable corresponding to the node associated with all of the stratified edges, and a specific parent configuration $l$ of the parents $\Pi _d$ of $X_d$. The equation for calculating the score for variables $X_1, \ldots , X_{d-1}$ will be identical using either the GM or the SGM. If the asymptotic equivalence holds for an arbitrary parent configuration it automatically holds for all parent configurations. Under this setting we start by looking at the score for the SGM

$$\begin{aligned} \log S_{\text {SGM}}({\mathbf {X}}^T \mid {\mathbf {X}}^R)= \log \frac{\varGamma (\sum _{i = 1}^{k_{j}} \beta _{jil})}{\varGamma (n(\pi _{j}^{l}) + \sum _{i = 1}^{k_{j}} \beta _{jil})} + \sum _{i = 1}^{k_{j}} \log \frac{\varGamma (n(x_{j}^{i} \mid \pi _{j}^{l}) + \beta _{jil})}{\varGamma (\beta _{jil})}, \end{aligned}$$

which using Stirling’s approximation and the same techniques as in the previous proof equals

$$\begin{aligned}&- \left( \sum _{i=1}^{k_j}\beta _{jil}-0.5\right) \log \left( 1 + \frac{n(\pi _j^l)}{\sum _{i=1}^{k_j}\beta _{jil}}\right) - n(\pi _j^l) \log \left( n(\pi _j^l) + \sum _{i=1}^{k_j}\beta _{jil}\right) \\&\quad + \sum _{i=1}^{k_j} n(x_j^i \mid \pi _j^l) \log (n(x_j^i \mid \pi _j^l) + \beta _{jil}) + \sum _{i=1}^{k_j} (\beta _{jil} - 0.5) \log \left( 1+\frac{n(x_j^i \mid \pi _j^l)}{\beta _{jil}}\right) . \end{aligned}$$

When studying the GM score we need to separately consider each outcome in the parent configuration $l$. Let $h$ denote such an outcome in $l$ with the total number of outcomes in $l$ totaling $q_l$. We then get the following score for the GM,

$$\begin{aligned} \log S_{\text {GM}}({\mathbf {X}}^T \mid {\mathbf {X}}^R)&= \sum _{h=1}^{q_l} \log \frac{\varGamma (\sum _{i = 1}^{k_{j}} \beta _{jih})}{\varGamma (n(\pi _{j}^{h}) + \sum _{i = 1}^{k_{j}} \beta _{jih})} \\&+ \sum _{h=1}^{q_l} \sum _{i = 1}^{k_{j}} \log \frac{\varGamma (n(x_{j}^{i} \mid \pi _{j}^{h}) + \beta _{jih})}{\varGamma (\beta _{jih})}. \end{aligned}$$

Which, using identical calculations as before, equals

$$\begin{aligned} -&\sum _{h=1}^{q_l} \left( \!\sum _{i=1}^{k_j}\beta _{jih}-0.5\!\right) \log \left( \!1 + \frac{n(\pi _j^h)}{\sum _{i=1}^{k_j}\beta _{jih}}\!\right) \!-\!\sum _{h=1}^{q_l} n(\pi _j^h) \log \left( n(\pi _j^h) + \sum _{i=1}^{k_j}\beta _{jih}\right) \\&\quad + \sum _{h=1}^{q_l}\sum _{i=1}^{k_j} n(x_j^i \mid \pi _j^h) \log (n(x_j^i \mid \pi _j^h) + \beta _{jih}) \\&\quad + \sum _{h=1}^{q_l}\sum _{i=1}^{k_j} (\beta _{jih} - 0.5) \log \left( 1 + \frac{n(x_j^i \mid \pi _j^h)}{\beta _{jih}}\right) . \end{aligned}$$

Considering the difference $\log S_{\text {SGM}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) - \log S_{\text {GM}}({\mathbf {X}}^T \mid {\mathbf {X}}^R)$ we get

$$\begin{aligned}&- \left( \sum _{i=1}^{k_j}\beta _{jil}-0.5\right) \log \left( 1 {+} \frac{n(\pi _j^l)}{\sum _{i=1}^{k_j}\beta _{jil}}\right) {+}\sum _{i{=}1}^{k_j} (\beta _{jil}{-}0.5) \log \left( 1+\frac{n(x_j^i \mid \pi _j^l)}{\beta _{jil}}\right) \\&\quad - n(\pi _j^l) \log \left( n(\pi _j^l) + \sum _{i=1}^{k_j}\beta _{jil}\right) +\sum _{h=1}^{q_l} n(\pi _j^h) \log \left( n(\pi _j^h) + \sum _{i=1}^{k_j}\beta _{jih}\right) \\&\quad + \sum _{i{=}1}^{k_j} n(x_j^i \mid \pi _j^l) \log (n(x_j^i \mid \pi _j^l) \!{+}\! \beta _{jil}) \!-\!\sum _{h{=}1}^{q_l}\sum _{i=1}^{k_j} n(x_j^i \mid \pi _j^h) \log (n(x_j^i \mid \pi _j^h) \!+\! \beta _{jih}) \\&\quad + \sum _{h=1}^{q_l} \left( \sum _{i=1}^{k_j}\beta _{jih}-0.5\right) \log \left( 1 + \frac{n(\pi _j^h)}{\sum _{i=1}^{k_j}\beta _{jih}}\right) \\&\quad -\sum _{h=1}^{q_l}\sum _{i=1}^{k_j} (\beta _{jih} - 0.5) \log \left( 1 + \frac{n(x_j^i \mid \pi _j^h)}{\beta _{jih}}\right) . \end{aligned}$$

Under the assumption that $\beta _{jil} \rightarrow \infty$ as $m \rightarrow \infty$, the terms in rows one and four will sum to 0 as $m \rightarrow \infty$. The remaining terms can be written

$$\begin{aligned} \log \frac{\prod _{h=1}^{q_l} (n(\pi _j^h) + \sum _{i=1}^{k_j}\beta _{jih})^{n(\pi _j^h)}}{(n(\pi _j^l) + \sum _{i=1}^{k_j}\beta _{jil})^{n(\pi _j^l)}} - \sum _{i=1}^{k_j} \log \frac{\prod _{h=1}^{q_l} (n(x_j^i \mid \pi _j^h) + \beta _{jih})^{n(x_j^i \mid \pi _j^h)}}{(n(x_j^i \mid \pi _j^l) + \beta _{jil})^{n(x_j^i \mid \pi _j^l)}}. \end{aligned}$$

Noting that $n(\pi _j^l) = \sum _{h=1}^{q_l} n(\pi _j^h)$ and $n(x_j^i \mid \pi _j^l) = \sum _{h=1}^{q_l} n(x_j^i \mid \pi _j^h)$ we get

$$\begin{aligned} \sum _{h=1}^{q_l} n(\pi _j^h) \log \frac{ n(\pi _j^h) + \sum _{i=1}^{k_j}\beta _{jih}}{n(\pi _j^l) + \sum _{i=1}^{k_j}\beta _{jil}} - \sum _{i=1}^{k_j} \sum _{h=1}^{q_l} n(x_j^i \mid \pi _j^h) \log \frac{n(x_j^i \mid \pi _j^h) + \beta _{jih}}{n(x_j^i \mid \pi _j^l) + \beta _{jil}}. \end{aligned}$$

By investigating the definition of the $\beta$ parameters in (3), in combination with the fact that the probabilities of observing the value $i$ for variable $X_j$ given that the outcome of the parents is $h$ are identical for any outcome $h$ comprising the group $l$, we get the limits

$$\begin{aligned} \lim _{m \rightarrow \infty } \frac{ n(\pi _j^h) + \sum _{i=1}^{k_j}\beta _{jih}}{n(\pi _j^l) + \sum _{i=1}^{k_j}\beta _{jil}} = \lim _{m \rightarrow \infty } \frac{n(x_j^i \mid \pi _j^h) + \beta _{jih}}{n(x_j^i \mid \pi _j^l) + \beta _{jil}} = \zeta _{jh}. \end{aligned}$$

And subsequently as $m \rightarrow \infty$ the difference $\log S_{\text {SGM}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) - \log S_{\text {GM}}({\mathbf {X}}^T \mid {\mathbf {X}}^R) \rightarrow$

$$\begin{aligned}&\sum _{h=1}^{q_l} \left( n(\pi _j^h) \log \zeta _{jh} - \sum _{i=1}^{k_j} n(x_j^i \mid \pi _j^h) \log \zeta _{jh} \right) \\&\quad = \sum _{h=1}^{q_l} \left( n(\pi _j^h) \log \zeta _{jh} - n(\pi _j^h) \log \zeta _{jh} \right) = 0. \end{aligned}$$

Appendix C: List of political parties in the Finnish parliament

Table 4 contains a list of political parties elected to the Finnish parliament in the parliament elections of 2011.

Table 4 List of political parties that are members of the Finnish parliament