A New Clustering Separation Measure Based on Negentropy

Abstract

Conventional distances used in clustering basically grow with the square of the distance in means, and are therefore insensitive when the clusters are tightly separated. This paper proposes a new separation measure between two Gaussians by blending the concept of non-Gaussianity and information theoretic distances with the goal of improving the separation of tightly coupled clusters. The measure called Quadratic Renyi’s Negentropy Separation (QRNS) relies on the estimation of negentropy between the Gaussian fitted clusters and the Gaussian mixture constructed by individual clusters. An analytical form to estimate separation is derived using the quadratic Rényi entropy instead of the common Shannon entropy. QRNS is not a distance, and its properties are analyzed and discussed. Finally, QRNS is used to develop a simple clustering algorithm, which is preliminary validated in several classical problems with separable clusters. Although the proposed clustering algorithm is not among the state of the art in clustering, it shows very good results when using the new measure.

Appendices

Appendix 1: Case Study

In order to obtain a separation distance using QRNS, we cannot use Gaussians to measure the non-Gaussianity. Instead we have to use a distribution that maximizes the quadratic Renyi’s entropy (QRE). From the literature (Bashkirov 2004; Vignat et al. 2003), the general distribution that maximizes $\alpha$-Rényi entropy is given by

$$\begin{aligned} P_\alpha (x) = \left\{ {{\begin{array}{ll} {A_\mathrm{C} \left( {\frac{(x-\mu )^{t} C^{-1} (x-\mu )}{k}} \right) ^{\frac{1}{\alpha -1}}} &{} \quad {x\in \varOmega } \\ 0 &{} \quad \mathrm{otherwise}\\ \end{array} }} \right. \end{aligned}$$

(14)

for $\alpha >1$, where $\varOmega$ defines hard limits for which $P_\alpha (x) >0.\,\, A_C$ and $k$ are scale constants depending on the covariance and dimension of the variable, $\mu$ and $C$ are the first moment and second central moment (mean and covariance) of the distribution. For $\alpha <1$, the distribution becomes

$$\begin{aligned} P_\alpha (x) = A_\mathrm{C} \left( {\frac{(x-\mu )^{t} C^{-1} (x-\mu )}{k}} \right) ^{\frac{1}{\alpha -1}}, \end{aligned}$$

(15)

where the limits for which $P_\alpha (x) >0$ can be all $x$. For the case of QRE, the distribution (here called Quadratic distribution) can be written as

$$\begin{aligned}&P_2 (x) \nonumber \\&= \left\{ {{\begin{array}{ll} {A_C \left( {1-\frac{(x-\mu )^{t} C^{{-}1} (x{-}\mu )}{k}} \right) ^{\frac{1}{\alpha {-}1}}}&{}\quad {(x{-}\mu )^{t} C^{{-}1} (x{-}\mu )<kx} \\ 0 &{}\quad \mathrm{otherwise} .\\ \end{array} }} \right. \nonumber \\ \end{aligned}$$

(16)

So in order to measure separation between two variables using Rényi entropy, the “non-Quadraticity” of the mixture needs to be measured.

In order to use this distribution as the basis for the derivation of QRNS, a close form solution for the entropy of a mixture of two distributions is required. Figure 14 shows a plot of a mixture of two 2D quadratic distributions.

Fig. 14

Plot of a 2D quadratic distribution mixture $(P_1 = P_2 =1$, $\mu _1 = [0\quad 0]^{t}, \mu _2 = [0\quad 3]^{t}, C_1 = I, \hbox { and } C_2 = I)$

This raises a technical problem because the pdf has finite support and integrating $p_2 (x)$ (specifically the mixture) is very complicated. Here the analytical solution for the 1D case is presented as a support for the claim of using quadratic distributions. In this case, the limits are defined by high and low bounds of the integration and can be computed analytically without excessive complications.

1.1 Separation for 1D Distribution

Here an analytical form for QRNS in the 1D case of two quadratic distributions is derived. The 1D quadratic distribution can be written as

$$\begin{aligned} P_x (x) {=} \left\{ {{\begin{array}{ll} {\frac{3}{4c\sqrt{5}} \left( {1{-}\frac{(x{-}\mu )^{2}}{5c^{2}}} \right) }&{} \ {\mu {-}c\sqrt{5}{<}x{<}\mu {+}c\sqrt{5}} \\ 0 &{}\quad \mathrm{otherwise}, \\ \end{array} }} \right. \end{aligned}$$

(17)

where $\mu$ is the mean of the variable and $c$ is its variance. As we can see, the limits of the function in 1D, are simply a high and a low bound, which makes the integral possible to solve in a simple manner.

The QRNS for a quadratic mixture has the following expression:

$$\begin{aligned} J_\mathrm{QRN} = H_{2,q}(x)-H_2 (x), \end{aligned}$$

where $H_2 (x)$ is the QRE of the mixture and $H_{2,q}(x)$ is the QRE for a quadratic distribution with the same covariance as the mixture. The mean and variance for a mixture of quadratic distributions with a priori probabilities given by $P_1$ and $P_2$, and means and covariance matrix given by $\mu _1$ , $\mu _2$ , $\sum _1$, and $\sum _2$ is given by

$$\begin{aligned}&\mu _q = P_1 \mu _1 + P_2 \mu _2 \\ \nonumber&C_q^2 = P_1 C_1^2 + P_2 C_2^2 + P_1 P_2 (\mu _1 -\mu _2 )^{2} \end{aligned}$$

(18)

So, QRNS can be calculated as

$$\begin{aligned} \mathrm{QRNS}&=-\log {\int \limits _{S_q }}{\left( {\frac{\hbox {3}}{\hbox {4}\,c_q \,\sqrt{5}}\left( {1-\frac{(x-\mu _q )^{2}}{5\,c_q^2 }} \right) } \right) ^{\hbox {2}}\hbox {d}x} \nonumber \\&\quad +\log \int \limits _S \left( \frac{\hbox {3}}{\hbox {4}\,c_1 \,\sqrt{5}}\left( {1-\frac{(x-\mu _1 )^{2}}{5\,c_1^2 }} \right) \right. \nonumber \\&\quad \left. + \, \frac{\hbox {3}}{\hbox {4}\,c_2 \,\sqrt{5}}\left( {1-\frac{(x-\mu _2 )^{2}}{5\,c_2^2 }} \right) \right) ^{\hbox {2}}\hbox {d}x. \end{aligned}$$

(19)

As the distributions have finite support, we have to find the correct limits $S$ and $S_q$ in order to solve the integrals. For the first term of the equation (19), the limits are well defined as $\mu _q -C_q \sqrt{5}<x<\mu _q +C_q \sqrt{5}$ (as defined in Eq. 17) since the integral just integrates over one distribution. The second term (the mixture term) requires some attention. There are basically tree situations for a mixture with different parameters depending on the means and covariances. Those situations are illustrated in Fig. 15.

Fig. 15

Possible situations for integrating a mixture of quadratic distributions

All others cases involving the changes in variance can be stated like one of those tree situations, so they will not be analyzed. Each of those cases has to be treated separately in order to lead to the correct value of separation. This situation suggests that the second term of QRNS depends on the separation of the means (and the values of variances). We can state the cases as follows:

Case 1 Distributions are separated enough to be integrated separately. And limits of integration can be used as the limits of each distribution.

Case 2 Part of the first distribution is coinciding with part of the second. In that case, we have tree integrals; Integrate $P_1 p_{1,X} (x)$ from the beginning of the first distribution until the first intersection, integrate $P_1 p_{1,X} (x)+P_2 p_{2,X} (x)$ from the first intersection to the second intersection, and integrate $P_2 p_{2,X} (x)$ from the second intersection until the end of the second distribution.

Case 3 The second distribution is completely inside the first one. In that case we have to integrate $P_2 p_{2,X} (x)$ from the beginning of the second distribution until the first intersection; from the second intersection until the end of the second distribution; and integrate $P_1 p_{1,X} (x)+P_2 p_{2,X} (x)$ within the limits of the first distribution.

So, QRNS can be calculated as

$$\begin{aligned} \mathrm{QRNS} = \left\{ \begin{array}{cl} H_q +\log \left( {\int \nolimits _{\mu _1 -c_1 \sqrt{5}}^{\mu _1 +c_2 \sqrt{5}}{pp_{1,X} (x)^{2}\hbox {d}x} +\int \nolimits _{\mu _2 -c_2\sqrt{5}}^{\mu _2 +c_2 \sqrt{5}} {pp_{2,X} (x)^{2}\hbox {d}x} }\right) &{} \quad \mu _1 +C_1\sqrt{5}\le \mu _2 -C_2 \sqrt{5}\\ H_q +\log \left( {\int \nolimits _{\mu _1 -c_1 \sqrt{5}}^{\mu _2 -c_2\sqrt{5}} {pp_{1,X} (x)^{2}\hbox {d}x} +\int \nolimits _{\mu _2 -c_2\sqrt{5}}^{\mu _1 +c_1 \sqrt{5}} {mp (x)^{2}\hbox {d}x} +\int \nolimits _{\mu _1 -c_1 \sqrt{5}}^{\mu _2 +c_2 \sqrt{5}}{pp_{2,X} (x)^{2}\hbox {d}x} } \right) &{} \quad \left\{ \begin{array}{l} {\mu _1 -C_1 \sqrt{5}\le \mu _2 -C_2 \sqrt{5}} \\ {\hbox {and}} \\ {\mu _2 -C_2 \sqrt{5}\le \mu _1 -C_1 \sqrt{5}} \\ \end{array} \right. \\ H_q +\log \left( {\int \nolimits _{\mu _2 -c_2 \sqrt{5}}^{\mu _1 -c_1 \sqrt{5}} {pp_{2,X} (x)^{2}\hbox {d}x} +\int \nolimits _{\mu _1 -c_1 \sqrt{5}}^{\mu _1 +c_1 \sqrt{5}} {mp (x)^{2}\hbox {d}x} + \int \nolimits _{\mu _1 -c_1 \sqrt{5}}^{\mu _2 +c_2 \sqrt{5}} {pp_{2,X} (x)^{2}\hbox {d}x} } \right) &{} \quad \hbox {otherwise}, \\ \end{array} \right. \end{aligned}$$

where

$$\begin{aligned} H_q&= -\log \int _{\mu _q -c_q \sqrt{5}}^{\mu _q +c_q \sqrt{5}} {\frac{3}{4c_q \sqrt{5}}\left( {1-\frac{(x-\mu _q )^{2}}{5c_q^2 }} \right) \hbox {d}x} \\ pp_{1,X} (x)&= \frac{3P_1 }{4c_1 \sqrt{5}}\left( {1-\frac{(x-\mu _1 )^{2}}{5c_1^2 }} \right) \\ pp_{2,X} (x)&= \frac{3P_2 }{4c_2 \sqrt{5}}\left( {1-\frac{(x-\mu _2 )^{2}}{5c_2^2 }} \right) \\ mp(x)&= pp_{1,X} (x)+pp_{2,X} (x). \end{aligned}$$

The integrals are trivial to solve, but the results are quite big expressions, so they are not presented here. Figure 16 shows a plot of QRNS for $\mu _1 =0, P_1 =P_1 =0.5, \mu _2$ varying from 0 to 35 and several values of variances.

Fig. 16

Plot of QRNS as function of the difference between the means

We can see that the values of QRNS stays near zero until the difference between the means are larger than the apparent variance of the pdfs. QRNS is a new measure in statistics that can be called separation between two pdfs.

Appendix 2: Properties of QRNS

1.1 Scale Invariance of QRNS

QRNS is scale invariant as the others statistical measures, which is an important property in clustering where thresholds are often needed. Let us consider the following transformation:

$$\begin{aligned} y = Ax+b , \end{aligned}$$

(20)

where $A$ is an invertible matrix and $b$ is a bias vector. If this transformation is applied to a random variable $X$ with covariance matrix $\Sigma$ and mean vector $\mu$, a new variable $Y$ with covariance matrix ${A\sum {A^{t}}}$ and mean vector $A\mu +b$ is obtained. If we take these new parameters in QRNS and $A$ is invertible, one can (after some algebra) conclude that QRNS is unchanged.

1.2 Correspondence of QRNS and Fitting Error

If we zoom in one of the subfigures in Fig. 4, we can see that the behavior for small changes in the mean (stable region) presents also a non-monotonic growing. This is not a

desirable behavior because QRNS cannot be a distance. Also, the negative values are a consequence of using the Gaussian distribution in the Renyi negentropy measure. Appendix  explains that in detail and gives an alternative to one dimension problem as an example.

To explain this undesirable behavior, we will use QRNS not as it derived in Eq. 13, but rather we will use the expression for QRNS derived (see Appendix ) in one dimension with the distribution that maximized Renyi’s quadratic entropy (the quadratic distribution) instead of Gaussians. This procedure consists basically in deriving the expression of QRNS using quadratic distributions instead of Gaussians. The reason to do that in this analysis is because we can look only at the effect of the non-monotonicity without having negative values (see Appendix  for explanation about the negative values). The new expression for QRNS (derived from Renyi’s max entropy) is not practical, since it involves complicated integration limits and it is feasible only in 1D (see Appendices  and  for details). As we stated previously, we simple rewrite the QRNS, but instead of Gaussians, we use quadratic distributions. The idea is the same as with Gaussians, we measure the difference between a mixture of distributions and one single distribution (in this case, quadratic distributions) with the same second-order moment of the mixture.

Let us perform the same test as in Fig. 4, but with the QRNS derived from Renyi’s max entropy distribution. Figure 18 shows (continuous line) the behavior with the mean separation. Again we can see a “stable” region before the QRNS starts to increase. This “stable” region shows a non-monotonic behavior (a “knee”) as the QRNS derived in Eq. 13. To explain this, let us measure the fitting error of one quadratic distribution to the mixture and measure that error for each separation. Figure 17 illustrates the fitting error for several mean separations.

Fig. 17

Fitting error for several separations in the mean of the mixture. Horizontal axis is mean separation and vertical axis is the probability density

As we expect, the fitting error (Fig. 18, dashed) is zero when the separation is zero (after all, we have only one quadratic distribution). What is surprising is that the error is not monotonic. In fact it grows and decays almost exactly as the QRNS. So, if we interpret the error as the “goodness of fit a quadratic distribution in a mixture of two” we can say how “non-Quadratic” is the mixture. Indeed, that is exactly what negentropy is all about in the Renyi’s definition. Obviously we do not expect exactly the same values for both as well their first derivatives (that is why they have different extreme points) but the overall shape is quite the same.

Fig. 18

Plot of QRNS and fitting error of a quadratic curve into the mixture of two quadratic distributions (distributions that maximizes Renyi’s quadratic entropy)

In some sense, the same thing occurs with QRNS when using Gaussians. The negative values of the Gaussian version QRNS presented in this paper also have a “knee” but, as argued before, to comparison purposes, that is not an issue and since we cannot use Shannon’s entropy to investigate, as detailed as in the quadratic case, we leave that a further investigation.

1.3 Relationship Between QRNS and Probability of Error in Classification

Another important relationship that must be stated is the comparison between QRNS and error probability. That is important because, since we are interested in measure separation between two Gaussians, one could think in treating them as distributions for two random variables and measure the probability of classifying one as another. This is in accordance with our goal because if we have two very distinct variables, the error probability of classifying one as another is low, whereas if we have two variables that are practically the same, the error would be maximum.

Again, the tests were performed in 1-D for the sake of simplicity. We measured the error probability with two Gaussian clusters with same covariance, as a function of the mean separation (because we cannot have an analytical expression in the general case, we used a numerical integration technique). We plotted the error probability and QRNS for several values of variances (from 0.4 to 4) as shown in Fig. 19.

Fig. 19

Plot of probability of error and QRNS versus mean separation for several values of variances

In the figure, the decreasing curves are the classification error probability. As expected, they decrease with the separation of the means. Each curve corresponds to a different variance value. The increasing curves are QRNS for the same variances. When analyzing the plots, we found a very interesting relation between the mean separation where the QRNS crosses the horizontal axis and the probability of error. The vertical dotted lines go from the point QRNS crosses zero to the probability error value in the plot corresponding to the same covariance value curve. We observed that the zero crossing occurs approximately at $2\sigma$. The horizontal thick line cuts the plot at approximately 0.14 that is the practically the value when QRNS crosses zero. So, if we pick a separation in witch QRNS crosses zero (the second zero, actually) we are always choosing the separation that gives us 13 % of error in classification no matter the variance of the data. Moreover, if we fix the value of QRNS, we always have a corresponding value of probability of error, regardless the variance of the clusters