Consensus of Clusterings Based on High-Order Dissimilarities

Abstract

Over the years, many clustering algorithms have been developed, handling different issues such as cluster shape, density or noise. Most clustering algorithms require a similarity measure between patterns, either implicitly or explicitly. Although most of them use pairwise distances between patterns, e.g., the Euclidean distance, better results can be achieved using other measures. The dissimilarity increments is a new high-order dissimilarity measure, that uses the information from triplets of nearest neighbor patterns. The distribution of such measure (DID) was recently derived under the hypothesis of local Gaussian generative models, leading to new clustering algorithms. DID-based algorithm builds upon an initial data partition, different initializations producing different data partitions. To overcome this issue, we present an unifying approach based on a combination strategy of all these different initializations. Even though this allows obtaining a robust partition of the data, one must select a clustering algorithm to extract the final partition. We also present a validation criterion based on DID to select the best final partition, consisting in the estimation of graph probabilities for each cluster based on the DID.

Appendix

Assume that X = {x ₁, …, x _N } is a l-dimensional set of patterns, and $\mathbf{x}_{i} \sim \mathcal{N}(\boldsymbol{\mu },\varSigma )$. Also, with no loss of generality, assume that $\boldsymbol{\mu }= 0$ and Σ is diagonal; this only involves translation and rotation of the data, which does not affect Euclidean distances. If x denotes a sample from this Gaussian, we transform x into x ^∗ through a process known as “whitening” or “sphering”, such that its i-th entry is given by $x_{i}^{{\ast}}\equiv x_{i}/\varSigma _{ii}$; x _i ^∗ thus follows the standard normal distribution, $\mathcal{N}(0,1)$. Now, it is known that the difference between samples, such as $x_{i}^{{\ast}}- y_{i}^{{\ast}}$, from two univariate standard normal distributions follows a normal distribution with covariance 2. Therefore,

$$\displaystyle{ \frac{x_{i}^{{\ast}}- y_{i}^{{\ast}}} {\sqrt{2}} \sim \mathcal{N}(0,1). }$$

(10.30)

It can be shown that the squared Euclidean distance,

$$\displaystyle{ (d^{{\ast}}(\mathbf{x}^{{\ast}}/\sqrt{2},\mathbf{y}^{{\ast}}/\sqrt{2}))^{2} =\sum _{ i=1}^{l}\left (\frac{x_{i}^{{\ast}}- y_{ i}^{{\ast}}} {\sqrt{2}} \right )^{2} \sim \chi ^{2}(l), }$$

(10.31)

i.e., follows a chi-square distribution with l degrees of freedom [28]. Thus, the probability density function for $(d^{{\ast}})^{2} \equiv (d^{{\ast}}(\cdot,\cdot ))^{2}$ is given by:

$$\displaystyle{ p(x) = \frac{2^{-l/2}} {\varGamma (l/2)} \;x^{l/2-1}\exp \left \{-\frac{x} {2}\right \},\;x \in [0,+\infty [, }$$

(10.32)

where Γ(⋅ ) denotes the gamma function.

After the sphering, the transformed data has circular symmetry in $\mathbb{R}^{l}$. We define angular coordinates in a (l − 1)-sphere, with θ _i  ∈ [0, π[, i = 1, …, l − 2 and θ _l−1 ∈ [0, 2π[. Define $\mathbf{x} -\mathbf{y} \equiv \left (b_{1},b_{2},\ldots,b_{l}\right )$, where b _i can be expressed in terms of polar coordinates as

$$\displaystyle\begin{array}{rcl} b_{1}& =& \sqrt{2\varSigma _{11}}\,d^{{\ast}}\cos \theta _{ 1} {}\\ b_{i}& =& \sqrt{2\varSigma _{ii}}\,d^{{\ast}}\left [\prod _{ k=1}^{i-1}\sin \theta _{ k}\right ]\cos \theta _{i},i = 2,\ldots,l - 1 {}\\ b_{l}& =& \sqrt{2\varSigma _{ll}}\,d^{{\ast}}\left [\prod _{ k=1}^{l-1}\sin \theta _{ k}\right ]. {}\\ \end{array}$$

The squared Euclidean distance in the original space, d ², is

$$\displaystyle\begin{array}{rcl} d^{2}& =& \|\mathbf{x} -\mathbf{y}\|^{2} \\ & =& 2\left [\varSigma _{11}\cos ^{2}\theta _{ 1} +\sum _{ i=2}^{l-1}\varSigma _{ ii}\left (\prod _{k=1}^{i-1}\sin ^{2}\theta _{ k}\right )\cos ^{2}\theta _{ i} +\varSigma _{ll}\left (\prod _{k=1}^{l-1}\sin ^{2}\theta _{ k}\right )\right ]\left (d^{{\ast}}\right )^{2} \\ & \equiv & 2A(\varTheta )\left (d^{{\ast}}\right )^{2}, {}\end{array}$$

(10.33)

where A(Θ), with Θ = (θ ₁, θ ₂, …, θ _l−1), is called the expansion factor. Naturally, this expansion factor depends on the angle vector Θ, and it is hard to properly deal with this dependence. Thus, we will use the approximation that the expansion factor is constant and equal to the expected value of the true expansion factor over all angles Θ. This expected value is given by

$$\displaystyle{ \mathbb{E}[A(\varTheta )] =\int _{S^{l-1}}\left [\prod _{i=1}^{l-2}p_{\theta _{ i}}(\theta _{i})\right ]p_{\theta _{l-1}}(\theta _{l-1})A(\varTheta )\;\mathrm{d}_{S^{l-1}}V }$$

(10.34)

where the volume element is $\mathrm{d}_{S^{l-1}}V = \left (\prod _{i=1}^{l-2}\sin ^{l-(i+1)}\theta _{i}\right )\,\mathrm{d}\theta _{1}\ldots \,\mathrm{d}\theta _{l-1}$. Since we sphered the data, we can assume for simplicity that $\theta _{i} \sim Unif([0,\pi [)$ for i = 1, …, l − 2 and that $\theta _{l-1} \sim Unif([0,2\pi [)$; then $p_{\theta _{i}}(\theta _{i}) = 1/\pi$ and $p_{\theta _{l-1}}(\theta _{l-1}) = 1/2\pi$. Thus, after some computations (see [2] for details), the expected value of the true expansion factor over all angles Θ is given by

$$\displaystyle{ \mathbb{E}[A(\varTheta )] = \frac{\pi ^{-l/2+1}} {2\varGamma (1 + l/2)}\mathrm{tr}(\varSigma ). }$$

(10.35)

Using Eq. (10.35), the transformation Eq. (10.33) from the normalized space to the original space is given by

$$\displaystyle{ \left (d\right )^{2} = \frac{\pi ^{-l/2+1}} {\varGamma (1 + l/2)}\mathrm{tr}(\varSigma )\,\left (d^{{\ast}}\right )^{2}. }$$

(10.36)

Assume that Y = aX, a constant, with p _X (x) the probability density function of X, so $p_{Y }(y) = p_{X}(y/a)\mathrm{d}x/\mathrm{d}y = p_{X}(y/a) \cdot 1/a$. From Eq. (10.32), we obtain the probability density function for the squared Euclidean distance in the original space, $\left (d\right )^{2}$. Again, assuming that Y ² = X and p _X (x) the probability density function of X, we have $p_{Y }(y) = p_{X}(y^{2})\mathrm{d}x/\mathrm{d}y = p_{X}(y^{2}) \cdot 2y$. Therefore, we obtain the probability density function of the Euclidean distance, $d \equiv d(\mathbf{x},\mathbf{y})$, as

$$\displaystyle{ p(y) = 2G_{l}(\mathrm{tr}(\varSigma ))y^{l-1}\exp \left \{-C_{ l}(\mathrm{tr}(\varSigma ))y^{2}\right \},\,y \in [0,+\infty [, }$$

(10.37)

where we define

$$\displaystyle{ G_{l}(\mathrm{tr}(\varSigma )) \equiv l^{l/2}\varGamma (l/2)^{l/2-1}2^{-l}\mathrm{tr}(\varSigma )^{-l/2}\pi ^{l/2(l/2-1)} }$$

(10.38)

and

$$\displaystyle{ C_{l}(\mathrm{tr}(\varSigma )) \equiv l\varGamma (l/2)(4\mathrm{tr}(\varSigma ))^{-1}\pi ^{l/2-1}. }$$

(10.39)

Now, the dissimilarity increment is defined as the absolute value of the difference of two Euclidean distances. Define d ≡ d(x, y) and d ^′ ≡ d(y, z), which follow the distribution in Eq. (10.37). The probability density function of d _inc = d − d ^′ is given by the convolution

$$\displaystyle\begin{array}{rcl} p_{d_{ \mbox{ inc}}}(w;\mathrm{tr}(\varSigma ))& =& \int _{-\infty }^{\infty }(2G_{ l}(\mathrm{tr}(\varSigma )))^{2}\,(t(t + w))^{l-1}\exp \left \{-C_{ l}(\mathrm{tr}(\varSigma ))\left (t^{2} + (t + w)^{2}\right )\right \} \\ & & \mathbf{1}_{\{t\geq 0\}}\mathbf{1}_{\{t+w\geq 0\}}\mathrm{d}t. {}\end{array}$$

(10.40)

Therefore, after some calculations (see [2] for details), the probability density function for the dissimilarity increments is given by

$$\displaystyle\begin{array}{rcl} p_{d_{ \mbox{ inc}}}(w;\mathrm{tr}(\varSigma ))& =& \frac{G_{l}(\mathrm{tr}(\varSigma ))^{2}} {2^{l-5/2}C_{l}(\mathrm{tr}(\varSigma ))^{l-1/2}}\exp \left \{-\frac{C_{l}(\mathrm{tr}(\varSigma ))} {2} \;w^{2}\right \} \\ & & \Bigg[\sum _{k=0}^{l-1}\sum _{ i=0}^{2l-2-k}(-1)^{i}w^{k+i}\binom{l - 1}{k}\binom{2l - 2 - k}{i}2^{k/2-i/2}C_{ l}(\mathrm{tr}(\varSigma ))^{k/2+i/2}\Bigg. \\ & & \left.\varGamma \left (\frac{2l - 1 - k - i} {2}, \frac{C_{l}(\mathrm{tr}(\varSigma ))} {2} \;w^{2}\right )\right ], {}\end{array}$$

(10.41)

where Γ(a, x) is the incomplete gamma function [44].