Additive and multiplicative mixed normal distributions and finding cluster centers

Abstract

Mixed normal distributions are considered in additive and multiplicative forms. While the weighted arithmetic mean of the probability density functions typically demonstrates several peaks corresponding to the parent sub-distributions, their weighted geometric mean is always expressed in one unimodal multivariate normal distribution. Estimation of the cluster center parameters from such a synthesized distribution is considered. The problem is solved by a non-linear least squares optimization yielding the cluster centers and sizes. The relationship to factor analysis by unweighted least squares and generalized least squares is noted, and numerical results are discussed. The described approach uses only the sample variance–covariance matrix and not the observations, so it can be applied for difficult clustering tasks on huge data sets from data bases and for data mining problems such as finding the approximation for the cluster centers and sizes. The suggested techniques can enrich both theoretical consideration and practical applications for clustering problems.

Appendix: Geometric mean of multinormal distributions

To find the explicit form for the geometric mean of sub-distributions, use (1) in (3) yielding:

$$ \begin{aligned} G(x) & = \prod\limits_{q = 1}^{K} {\left[ {\left( {(2\pi )^{n} |S_{q} |} \right)^{{ - \gamma_{q} /2}} \exp \left( { - \frac{1}{2}(x - m_{q} )^{\prime } (\gamma_{q} S_{q}^{ - 1} )(x - m_{q} )} \right)} \right]} \\ & = \exp \left( { - \frac{1}{2}Q(x)} \right)\prod\limits_{q = 1}^{K} {\left( {(2\pi )^{n} |S_{q} |} \right)^{{ - \gamma_{q} /2}} } \\ \end{aligned} $$

(38)

where the total of the quadratic forms can be represented as follows:

$$ \begin{aligned} Q(x) & = \sum\limits_{q = 1}^{K} {(x - m_{q} )^{\prime } (\gamma_{q} S_{q}^{ - 1} )(x - m_{q} )} \\ & = x^{\prime } \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)x - 2x^{\prime}\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} + } \sum\limits_{q = 1}^{K} {m_{q}^{\prime } (\gamma_{q} S_{q}^{ - 1} )m_{q} } \\ & = x^{\prime } \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)\left\{ {x - 2\left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)^{ - 1} \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } } \right)} \right\} + \sum\limits_{q = 1}^{K} {m_{q}^{\prime } \gamma_{q} S_{q}^{ - 1} m_{q} } \\ \end{aligned} $$

(39)

Completing the first term to the whole square we get:

$$ \begin{aligned} Q(x) & = \left\{ {x - \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)^{ - 1} \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } } \right)} \right\}^{\prime } \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right) \\ & \quad \times \left\{ {x - \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)^{ - 1} \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } } \right)} \right\} + \sum\limits_{q = 1}^{K} {m_{q}^{\prime } \gamma_{q} S_{q}^{ - 1} m_{q} } \\ & \quad - \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } } \right)^{\prime } \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)^{ - 1} \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } } \right) \\ \end{aligned} $$

(40)

Let us denote the weighted mean of the inverted covariance matrices in (40) as:

$$ S_{tot}^{ - 1} \equiv \sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } . $$

(41)

So the total covariance matrix is defined via the covariance matrices of sub-distributions:

$$ S_{tot} = \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)^{ - 1} . $$

(42)

The weighted aggregate of the Fisher discriminator vectors in (40) (in another formulation, a combination of the coefficients of regressions of binary indices of belonging to each class by predictors) we denote as:

$$ F_{tot} \equiv \sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } . $$

(43)

Then the result in (40) can be presented as

$$ Q(x) = \left( {x - x^{ * } } \right)^{\prime } S_{tot}^{ - 1} \left( {x - x^{ * } } \right) + C, $$

(44)

where the vector $x^{ * }$ is defined as:

$$ x^{ * } = \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)^{ - 1} \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } } \right), $$

(45)

and the constant C is explained below. The vector (45) can also be presented as:

$$ x^{ * } = S_{tot} F_{tot} = \sum\limits_{q = 1}^{K} {(\gamma_{q} S_{tot} S_{q}^{ - 1} )m_{q} } , $$

(46)

so it is a weighted vector of means, because the total matrix of weights in (46) equals the identity matrix:

$$ \sum\limits_{q = 1}^{K} {\gamma_{q} S_{tot} S_{q}^{ - 1} = } S_{tot} \sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} = } S_{tot} S_{tot}^{ - 1} = I. $$

(47)

When $x = x^{ * }$, the quadratic form (44) equals its maximum value:

$$ \begin{aligned} C & = \sum\limits_{q = 1}^{K} {m_{q}^{\prime } \gamma_{q} S_{q}^{ - 1} m_{q} } - \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } } \right)^{\prime } \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)^{ - 1} \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } } \right) \\ & = \sum\limits_{q = 1}^{K} {m_{q}^{\prime } \gamma_{q} S_{q}^{ - 1} m_{q} } - F^{\prime}_{tot} S_{tot} F_{tot} = \sum\limits_{q = 1}^{K} {m_{q}^{\prime } \gamma_{q} S_{q}^{ - 1} m_{q} } - F_{tot}^{\prime } x^{ * } \\ & = \sum\limits_{q = 1}^{K} {m_{q}^{\prime } \gamma_{q} S_{q}^{ - 1} (m_{q} } - x^{ * } ) = \sum\limits_{q = 1}^{K} {(m_{q} - x^{ * } )^{\prime } (\gamma_{q} S_{q}^{ - 1} )(m_{q} } - x^{ * } ) + A \\ \end{aligned} $$

(48)

where the additional constant A is actually zero, because

$$ \begin{aligned} A & = \sum\limits_{q = 1}^{K} {(x^{ * } )^{\prime } (\gamma_{q} S_{q}^{ - 1} )(m_{q} } - x^{ * } ) = (x^{ * } )^{\prime } \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} m_{q} } - \left( {\sum\limits_{q = 1}^{K} {\gamma_{q} S_{q}^{ - 1} } } \right)x^{ * } } \right) \\ & = (x^{ * } )^{\prime } \left( {F_{tot} - S_{tot}^{ - 1} S_{tot} F_{tot} } \right) = 0 \\ \end{aligned} $$

(49)

Thus, (44) is reduced explicitly to:

$$ Q(x) = (x - x^{ * } )^{\prime } S_{tot}^{ - 1} (x - x^{ * } ) + \sum\limits_{q = 1}^{K} {(m_{q} - x^{ * } )^{\prime } (\gamma_{q} S_{q}^{ - 1} )(m_{q} - x^{ * } )} . $$

(50)

Using (50) in (38) yields:

$$ \begin{aligned} G(x) & = \exp \left( { - \frac{1}{2}(x - x^{ * } )^{\prime } S_{tot}^{ - 1} (x - x^{ * } )} \right) \\ & \quad \times \prod\limits_{q = 1}^{K} {\left[ {\left( {(2\pi )^{n} |S_{q} |} \right)^{ - 1/2} \exp \left( { - \frac{1}{2}(x^{ * } - m_{q} )^{\prime } (S_{q}^{ - 1} )(x^{ * } - m_{q} )} \right)} \right]}^{{\gamma_{q} }} \\ \end{aligned} $$

(51)

Then the expression (51) can be reduced to the following:

$$ \begin{aligned} G(x) & = \exp \left( { - \frac{1}{2}(x - x^{ * } )^{\prime } S_{tot}^{ - 1} (x - x^{ * } )} \right)\prod\limits_{q = 1}^{K} {\left[ {f_{q} \left( {x^{ * } ,m_{q} ,S_{q} } \right)} \right]}^{{\gamma_{q} }} \\ & = G(x^{ * } )\exp \left( { - \frac{1}{2}(x - x^{ * } )^{\prime } S_{tot}^{ - 1} (x - x^{ * } )} \right) \\ \end{aligned} $$

(52)

which is the geometric mean (3) in the point $x^{ * }$ (45), and the dependence on x is given in one exponent.