Cluster validity index based on Jeffrey divergence

Abstract

Cluster validity indexes are very important tools designed for two purposes: comparing the performance of clustering algorithms and determining the number of clusters that best fits the data. These indexes are in general constructed by combining a measure of compactness and a measure of separation. A classical measure of compactness is the variance. As for separation, the distance between cluster centers is used. However, such a distance does not always reflect the quality of the partition between clusters and sometimes gives misleading results. In this paper, we propose a new cluster validity index for which Jeffrey divergence is used to measure separation between clusters. Experimental results are conducted using different types of data and comparison with widely used cluster validity indexes demonstrates the outperformance of the proposed index.

Appendices

Appendix A: parameters estimation

In all the expressions below, x is a vector of random variables whose mean vector and covariance matrix are given by: $E(x) = \mu$ and $E((x-\mu )(x-\mu )^{T}) = \Sigma$ where $E$ means the expectation. Using matrix properties:

• $\frac{\partial {\rm A}^{T}\cdot x}{\partial x}=\frac{\partial x^{T}\cdot {\rm A}}{\partial x}= \mathrm{A}$

• $\frac{\partial x^{T}\cdot \mathrm{A}\cdot x}{\partial x}=\mathrm{A}^{T}+\mathrm{A}$

• $\frac{\partial }{\partial \mathrm{A}} {\text {log}} |\mathrm{A}|= \left( \mathrm{A}^{-1}\right) ^{T}$

• $\frac{\partial }{\partial \mathrm{A}} {\text {tr}} \left[ \mathrm{AB}\right] ={\text {tr}} \left[ \mathrm{BA}\right] =\mathrm{B}^{T}$

The log likelihood of the multivariate Gaussian distribution is given by:

$$\begin{aligned} L(x|\mu ,\Sigma )&= \frac{-nD}{2}{\text {log}}(2\pi ) - \frac{n}{2}{\text {log}}(|\Sigma |) \nonumber \\&\quad -\frac{1}{2} \sum _{i=1}^{n}(x_{i}-\mu )^{T} \Sigma ^{-1}(x_{i}-\mu ) \end{aligned}$$

(27)

The estimates of the mean and covariance matrix are determined by computing the derivatives of $L(x|\mu ,\Sigma )$ with relative to $\mu$ and $\Sigma$ and set it equal to zero.

$$\begin{aligned} \frac{\partial L(x|\mu ,\Sigma )}{\partial \mu }&= \frac{\partial }{\partial \mu } \left( \sum _{n}^{i=1}(x_{i}-\mu )^{T}\Sigma ^{-1}(x_{i}-\mu )\right) \\&= \frac{\partial }{\partial \mu } \left( \sum _{i=1}^{n} \left( x_{i}^{T}\Sigma ^{-1}x_{i} - \mu ^{T}\Sigma ^{-1}x_{i} - x_{i}^{T}\Sigma ^{-1}\mu \right. \right. \\&\quad +\left. \left. \mu ^{T} \Sigma ^{-1}\mu \right) \right) \\&= \sum _{i=1}^{n}\left( \Sigma ^{-1}x_{i}+ \left( \Sigma ^{-1} \right) ^{T}x_{i}\right) \\&\quad -N\left( \Sigma ^{-1}+\left( \Sigma ^{-1} \right) ^{T}\right) \mu \\&= 0\\&\Rightarrow \hat{\mu }=\frac{1}{n}\sum _{i=1}^{n}x_{i}\\ \end{aligned}$$

$$\begin{aligned} \frac{\partial L(x|\mu ,\Sigma )}{\partial \Sigma ^{-1}}&= \frac{\partial }{\partial \Sigma ^{-1} } \left( -\frac{N}{2}{\text {log}}(|\Sigma |)\right. \\&\left. \quad -\frac{1}{2} \sum ^{n}_{i=1} (x_{i}-\mu )^{T}\Sigma ^{-1}(x_{i}-\mu )\right) \\&\propto \frac{\partial }{\partial \Sigma ^{-1} } \left( -\frac{N}{2}{\text {log}}(|\Sigma |) -\frac{1}{2} \sum ^{n}_{i=1}{\text {tr}} \left[ \Sigma ^{-1} (x_{i}-\mu )\cdot \right. \right. \\&\left. \left. \quad (x_{i}-\mu )^{T} \right] \right) \\&= \frac{\partial }{\partial \Sigma ^{-1} } \left( \frac{N}{2}{\text {log}}(|\Sigma ^{-1}|) -\frac{1}{2} {\text {tr}} \left[ \sum ^{n}_{i=1} \Sigma ^{-1} (x_{i}-\mu )\cdot \right. \right. \\&\left. \left. \quad (x_{i}-\mu )^{T} \right] \right) \\&= 0 \\&\Rightarrow \hat{\Sigma }=\sum _{i=1}^{n}(x_{i}- \hat{\mu })(x_{i}- \hat{\mu })^{T} \end{aligned}$$

Appendix B: Jeffrey divergence for multivariate Gaussian distribution

We use the following formula:

• $E(x^{T}\cdot \mathrm{A} \cdot x)= {\text {tr}}(\mathrm{A}\cdot \Sigma )+ \mu ^{T}\cdot \mathrm{A}\cdot \mu$

• $\left\langle \cdot \right\rangle$ is the expectation symbol.

$$\begin{aligned}&p(x|\mu _{1},\Sigma _{1})=\frac{1}{(2\pi )^{d/2}|\Sigma _{1}|^{1/2}}{\text {exp}}\left( -\frac{1}{2}(x-\mu _{1})^{T}\Sigma _{1}^{-1}(x-\mu _{1})\right) \\&q(x|\mu _{2},\Sigma _{2})=\frac{1}{(2\pi )^{d/2}|\Sigma _{2}|^{1/2}}{\text {exp}}\left( -\frac{1}{2}(x-\mu _{2})^{T}\Sigma _{2}^{-1}(x-\mu _{2})\right) \\&{\rm JD}(p,q)= {\rm KL}(p/q)+{\rm KL}(q/p) \end{aligned}$$

Where KL is the Kullback-Leiber divergence.

$$\begin{aligned} {\rm KL}(p/q)&= \int {\text {log}}\left( p(x)-q(x)\right) p(x)\\&= \int \left( \frac{1}{2}{\text {log}}\left( \frac{|\Sigma _{2}|}{|\Sigma _{1}|}\right) - \frac{1}{2}(x-\mu _{1})^{T}\Sigma _{1}^{-1}(x-\mu _{1})\right. \\&\quad \left. +\frac{1}{2}(x-\mu _{2})^{T}\Sigma _{2}^{-1}(x-\mu _{2})\right) p(x) \\&= \frac{1}{2} {\text {log}}\left( \frac{|\Sigma _{2}|}{|\Sigma _{1}|}\right) - \frac{1}{2} \left\langle (x-\mu _{1})^{T}\Sigma _{1}^{-1}(x-\mu _{1}) \right\rangle \\&\quad + \frac{1}{2}\left\langle (x-\mu _{2})^{T}\Sigma _{2}^{-1}(x-\mu _{2}) \right\rangle \\&= \frac{1}{2} \left( {\text {log}}\left( \frac{|\Sigma _{2}|}{|\Sigma _{1}|}\right) - {\text {tr}} \left( \Sigma _{1}^{-1}\Sigma _{1}\right) + {\text {tr}} \left( \Sigma _{2}^{-1}\Sigma _{1}\right) \right) \\&\quad + \frac{1}{2} \left( (\mu _{1}-\mu _{2})^{T}\Sigma _{2}^{-1}(\mu _{1}-\mu _{2})\right) \\&= \frac{1}{2} \left( {\text {log}}\left( \frac{|\Sigma _{2}|}{|\Sigma _{1}|}\right) -d + + {\text {tr}} \left( \Sigma _{2}^{-1}\Sigma _{1}\right) \right) \\&\quad + \frac{1}{2} \left( (\mu _{1}-\mu _{2})^{T}\Sigma _{2}^{-1}(\mu _{1}-\mu _{2}) \right) \end{aligned}$$

Thus:

$$\begin{aligned} {\rm JD}(p,q)&= \frac{1}{2}\left( {\text {tr}}(\Sigma _{1}^{-1}\Sigma _{2})+{\text {tr}}(\Sigma _{2}^{-1}\Sigma _{1}) \right) \\&\quad + \frac{1}{2} \left( (\mu _{1}-\mu _{2})^{T}(\Sigma _{1}^{-1}+\Sigma _{2}^{-1})(\mu _{1}-\mu _{2})\right) -d \end{aligned}$$

Appendix C: Similarity measures

The four similarity measures we have used are described in this section. Let $P1$ and $P2$ two partitions. We define $a$ as the number of object pairs that belong to the same clusters in both $P1$ and $P2$. Let $b$ be the number of object pairs that belongs to different clusters in both pairs. Let $c$ be the number of object pairs that belong to the same clusters in $P1$ but in different clusters in $P2$. Finally, let $d$ the number of pairs that belong to different clusters in $P1$ but belong to the same cluster in $P2$. The four similarity measures are defined as:

$$\begin{aligned} {\text {Rand}}&= \frac{a+b}{a+b+c+d} \\ {\text {Fowlkes-Mallows}}&= \frac{a}{\sqrt{(a+d)(a+c)}} \\ {\text {Jaccard}}&= \frac{a}{a+c+d}\\\ {\text {Adjsuted-Rand}}&= \frac{a-\frac{(a+d)(a+c)}{a+b+c+d}}{\frac{a+b+c+d}{2}-\frac{(a+d)(a+c)}{a+b+c+d}} \end{aligned}$$