Generalization of clustering agreements and distances for overlapping clusters and network communities

Abstract

A measure of distance between two clusterings has important applications, including clustering validation and ensemble clustering. Generally, such distance measure provides navigation through the space of possible clusterings. Mostly used in cluster validation, a normalized clustering distance, a.k.a. agreement measure, compares a given clustering result against the ground-truth clustering. The two widely-used clustering agreement measures are adjusted rand index and normalized mutual information. In this paper, we present a generalized clustering distance from which these two measures can be derived. We then use this generalization to construct new measures specific for comparing (dis)agreement of clusterings in networks, a.k.a. communities. Further, we discuss the difficulty of extending the current, contingency based, formulations to overlapping cases, and present an alternative algebraic formulation for these (dis)agreement measures. Unlike the original measures, the new co-membership based formulation is easily extendable for different cases, including overlapping clusters and clusters of inter-related data. These two extensions are, in particular, important in the context of finding communities in complex networks.

Appendix: Proofs

1.1 Proof of Proposition 1

From the definition of Variation of information we have:

$$\begin{aligned} VI(U,V)= & {} H(U) + H(V) - 2I(U,V) = 2H(U,V)-H(U)-H(V)\\= & {} {\mathbf {H(V|U)+H(U|V)}} \end{aligned}$$

On the other hand, we have:

$$\begin{aligned} RI(U,V)&\propto \frac{1}{n^2-n}\left( \sum _{i=1}^k \left[ \sum _{j=1}^r n_{ij}^2 - \left( \sum _{j=1}^r n_{ij}\right) ^2\right] + \sum _{j=1}^r \left[ \sum _{i=1}^k n_{ij}^2 - \left( \sum _{i=1}^k n_{ij}\right) ^2\right] \right) \\&\overset{*}{\propto } \sum _{i=1}^k \left[ E_j(n_{ij}^2) - E_j( n_{ij})^2\right] + \sum _{j=1}^r \left[ E_i(n_{ij}^2) - E_i(n_{ij})^2\right] \\&\overset{*}{\propto } \sum _{i=1}^k Var_j(n_{ij}) + \sum _{j=1}^r Var_i(n_{ij}) \quad \overset{**}{\propto } \quad \mathbf { Var(V|U)+ Var(U|V) } \quad \quad \end{aligned}$$

$\square$

$(*)$ $E_j$/$Var_j$ shows the average/variance of values in the $j^{th}$ column of the contingency table.

$(**)$ The RI is in fact proportional to the average variance of rows/columns values in the contingency table, which we denote by conditional variance. For other forms of conditional variance for categorical data see Light and Margolin (1971).

1.2 Proof of Corollary 1

We first show that $0 \le \mathcal {D}_{\varphi }^\eta (U||V)$ which also results in the lower bound 0 for $\mathcal {D}_{\varphi }^\eta (U,V)$ since, $\mathcal {D}_{\varphi }^\eta (U,V) = \mathcal {D}_{\varphi }^\eta (U||V) + \mathcal {D}_{\varphi }^\eta (V||U)$. From the superadditivity of $\varphi$ we have:

$$\begin{aligned} \sum _{u\in U} \varphi (\eta _{uv}) \le \varphi \left( \sum _{u\in U} \eta _{uv}\right)&\implies \sum _{v\in V} \left[ \varphi \left( \sum _{u\in U} \eta _{uv}\right) - \sum _{u\in U} \varphi (\eta _{uv}) \right] \ge 0\\&\implies \mathbf {\mathcal {D}_{\varphi }^\eta (U||V) \ge 0} \end{aligned}$$

Similarly for the upper bound, from positivity and super-additivity we get respectively:

$$\begin{aligned} \mathcal {D}_{\varphi }^\eta (U||V) = \sum _{v\in V} \varphi \left( \sum _{u\in U} \eta _{uv}\right) - \sum _{v\in V} \sum _{u\in U} \varphi (\eta _{uv}) \le \sum _{v\in V} \varphi \left( \sum _{u\in U} \eta _{uv}\right) \le \varphi \left( \sum _{v\in V} \sum _{u\in U} \eta _{uv}\right) \ \end{aligned}$$

1.3 Proof of Identity 1

The proof is elementary, if we write the definition for $\varphi ={x}\log x$, we get:

$$\begin{aligned} \mathcal {N}\mathcal {D}_{x\log x}^{|\cap |}(U,V)&= \frac{\sum _{v\in V} \sum _{u\in U} |u\cap v| \left[ \log \left( \sum _{u\in U} |u\cap v|\right) - \log \left( |u\cap v|\right) \right] }{\left( \sum _{v\in V} \sum _{u\in U} |u\cap v| \right) \log \left( \sum _{v\in V} \sum _{u\in U} |u\cap v| \right) }\\&\quad + \frac{\sum _{u\in U} \sum _{v\in V} |u\cap v| \left[ \log \left( \sum _{v\in V} |u\cap v|\right) - \log (|u\cap v|) \right] }{\left( \sum _{u\in U} \sum _{v\in V}|u\cap v|\right) \log \left( \sum _{u\in U} \sum _{v\in V}|u\cap v|\right) }\\&\overset{*}{=} \frac{\sum _{j}^r \sum _{i}^k n_{ij} \left[ \log \left( \sum _{i}^k n_{ij}\right) + \log \left( \sum _{j}^r n_{ij}\right) -2 \log \left( n_{ij}\right) \right] }{\left( \sum _{i}^k\sum _{j}^{r} n_{ij}\right) \log \left( \sum _{i} ^k\sum _{j}^r n_{ij}\right) }\\&\overset{**}{=} \frac{1}{\log n}\sum _{j}^r \sum _{i}^k \frac{ n_{ij} }{n} \log \left( \frac{ n_{i.} n_{.j}}{n_{ij}^2}\right) = \frac{VI(U,V)}{\log n}\quad \quad \end{aligned}$$

$\square$

$(*)$ slight change of notation, i.e. from $\sum _{u\in U}$ to $\sum _{i}^k$, $\sum _{v\in V}$ to $\sum _{j}^r$ and $|u\cap v|$ to $n_{ij}$.

$(**)$ assuming disjoint covering partitionings: $\sum _{i}^k\sum _{j}^r n_{ij} = n$, $\sum _{i}^k n_{ij} = n_{.j}$ and $\sum _{j}^r n_{ij} = n_{i.}$.

1.4 Proof of Identity 2

Similar to the previous proof from the definition we derive:

$$\begin{aligned} \mathcal {N}\mathcal {D}_{\left( {\begin{array}{c}x\\ 2\end{array}}\right) }^{|\cap |}(U,V)&\overset{*}{=} \frac{ \sum _{j}^r \left[ \left( \sum _{i}^k n_{ij}\right) ^2 -\sum _{i}^k n_{ij}^2 \right] + \sum _{i}^k \left[ \left( \sum _{j}^r n_{ij}\right) ^2 - \sum _{j}^r n_{ij}^2 \right] }{\left( \sum _{i}^k\sum _{j}^r n_{ij}\right) ^2-\sum _{i}^k\sum _{j}^r n_{ij}} \\&\overset{**}{=} \frac{1}{n^2-n} \left[ \sum _{j}^r (n_{.j})^2 + \sum _{i}^k (n_{i.})^2 - 2 \sum _{j}^r \sum _{i}^k n_{ij}^2 \right] = 1-RI(U,V) \end{aligned}$$

$(*), (**)$ same as previous proof. $\square$

1.5 Proof of Identity 3 and 4

$$\begin{aligned}&\mathcal {A}\mathcal {D}_{\varphi }^\eta = \frac{\sum _{v\in V} \varphi (\eta _{.v}) + \sum _{u\in U} \varphi (\eta _{u.}) -2 \sum _{v\in V} \sum _{u\in U} \varphi (\eta _{uv})}{ \sum _{v\in V} \varphi (\eta _{.v}) + \sum _{u\in U} \varphi (\eta _{u.}) - 2 \sum _{u\in U}\sum _{v\in V} \varphi \left( \frac{\eta _{.v}\eta _{u.}}{\sum _{u\in U}\sum _{v\in V} \eta _{uv} }\right) }\\ \Rightarrow \;&1 - \mathcal {A}\mathcal {D}_{\varphi }^{\eta }(U,V) = \frac{ \sum _{v\in V} \sum _{u\in U} \varphi (\eta _{uv})- \sum _{u\in U}\sum _{v\in V} \varphi \left( \frac{\eta _{.v}\eta _{u.}}{\sum _{u\in U}\sum _{v\in V} \eta _{uv} }\right) }{ \frac{1}{2}\left[ \sum _{v\in V} \varphi (\eta _{.v}) + \sum _{u\in U} \varphi (\eta _{u.})\right] - \sum _{u\in U}\sum _{v\in V} \varphi \left( \frac{\eta _{.v}\eta _{u.}}{\sum _{u\in U}\sum _{v\in V} \eta _{uv} }\right) } \end{aligned}$$

This formula resembles the adjuctment for chance in Eq. 4, where the measure being adjusted is $\sum _{v\in V}\sum _{u\in U} \varphi (\eta _{uv})$, the upper bound used for it is $\frac{1}{2}[\sum _{v\in V} \varphi (\eta _{.v}) + \sum _{u\in U} \varphi (\eta _{u.})]$, and the expectation is defined as:

$$\begin{aligned} E\left[ \sum _{v\in V}\sum _{u\in U} \varphi (\eta _{uv})\right] = \sum _{u\in U}\sum _{v\in V} \varphi \left( \frac{\eta _{.v}\eta _{u.}}{\sum _{u\in U}\sum _{v\in V} \eta _{uv} }\right) \end{aligned}$$

Now if we have $\varphi (xy) = \varphi (x)\varphi (y)$, which is true for $\varphi (x)=x^2$, we get:

$$\begin{aligned} E\left[ \sum _{v\in V}\sum _{u\in U} \varphi (\eta _{uv})\right] = \sum _{u\in U}\sum _{v\in V} \frac{\varphi (\eta _{.v})\varphi (\eta _{u.})}{\varphi \left( \sum _{u\in U}\sum _{v\in V} \eta _{uv} \right) } = \frac{\sum _{v\in V} \varphi (\eta _{.v})\sum _{u\in U}\varphi (\eta _{u.})}{\varphi \left( \sum _{u\in U}\sum _{v\in V} \eta _{uv} \right) } \end{aligned}$$

Using this expecation, if we substitute $\varphi =x^2$ we get the $ARI'$ of Eq. 6, and using the $\varphi =\left( {\begin{array}{c}x\\ 2\end{array}}\right)$ and the later reformulation of E, we get the original ARI of Eq. 5, as:

$$\begin{aligned} 1 - \mathcal {A}\mathcal {D}_{\left( {\begin{array}{c}x\\ 2\end{array}}\right) }^{|\cap |}(U,V)&=\frac{ \sum \limits _{v\in V} \sum \limits _{u\in U} \left( {\begin{array}{c}|u\cap v|\\ 2\end{array}}\right) - E\left( \sum \limits _{v\in V} \sum \limits _{u\in U} \left( {\begin{array}{c}|u\cap v|\\ 2\end{array}}\right) \right) }{ \frac{1}{2}\left[ \sum \limits _{v\in V}\left( {\begin{array}{c}\sum \limits _{u\in U} |u\cap v|\\ 2\end{array}}\right) + \sum \limits _{u\in U}\left( {\begin{array}{c}\sum \limits _{v\in V} |u\cap v|\\ 2\end{array}}\right) \right] - E\left( \sum \limits _{v\in V} \sum \limits _{u\in U} \left( {\begin{array}{c}|u\cap v|\\ 2\end{array}}\right) \right) }\\&\text {where} \quad E\left( \sum \limits _{v\in V} \sum \limits _{u\in U} \left( {\begin{array}{c}|u\cap v|\\ 2\end{array}}\right) \right) = \frac{ \sum \limits _{v\in V}\left( {\begin{array}{c}\sum \limits _{u\in U} |u\cap v|\\ 2\end{array}}\right) \sum \limits _{u\in U}\left( {\begin{array}{c}\sum \limits _{v\in V} |u\cap v|\\ 2\end{array}}\right) }{ \left( {\begin{array}{c} n\\ 2\end{array}}\right) }\\ \Rightarrow 1 - \mathcal {A}\mathcal {D}_{\left( {\begin{array}{c}x\\ 2\end{array}}\right) }^{|\cap |}(U,V)&\overset{*,**}{=} \frac{ \sum _j^r \sum _i^k \left( {\begin{array}{c}n_{ij}\\ 2\end{array}}\right) - \sum _j^r \left( {\begin{array}{c}n_{.j}\\ 2\end{array}}\right) \sum _i^k\left( {\begin{array}{c}n_{i.}\\ 2\end{array}}\right) /\left( {\begin{array}{c}n\\ 2\end{array}}\right) }{ \frac{1}{2}\left[ \sum _j^r\left( {\begin{array}{c}n_{.j}\\ 2\end{array}}\right) + \sum _i^k\left( {\begin{array}{c}n_{i.}\\ 2\end{array}}\right) \right] -\sum _j^r \left( {\begin{array}{c}n_{.j}\\ 2\end{array}}\right) \sum _i^k\left( {\begin{array}{c}n_{i.}\\ 2\end{array}}\right) /\left( {\begin{array}{c}n\\ 2\end{array}}\right) } \;= ARI(U,V)\\&(*), (**) \text { same as proof of identity 1.} \end{aligned}$$

$\square$

On the other hand for the NMI, we have:

$$\begin{aligned}&1 - \mathcal {A}\mathcal {D}_{x\log x}^{|\cap |}(U,V) = \frac{ \sum \limits _{v\in V} \sum \limits _{u\in U} {n_{uv}}\log {n_{uv}} - E\left( \sum \limits _{v\in V} \sum \limits _{u\in U} {n_{uv}}\log {n_{uv}}\right) }{ \frac{1}{2}\left[ \sum \limits _{v\in V} {n_{.v}\log {n_{.v}}} + \sum \limits _{u\in U} n_{u.}\log {n_{u.}} \right] - E\left( \sum \limits _{v\in V} \sum \limits _{u\in U}{n_{uv}} \log {n_{uv}}\right) }\\&\quad where \; E\left( \sum \limits _{v\in V} \sum \limits _{u\in U} {n_{uv}}\log {n_{uv}}\right) = \sum _{u\in U}\sum _{v\in V} \left( \frac{\eta _{.v}\eta _{u.}}{\sum _{u\in U}\sum _{v\in V} \eta _{uv} }\right) \log \left( \frac{\eta _{.v}\eta _{u.}}{\sum _{u\in U}\sum _{v\in V} \eta _{uv} }\right) \\&\quad \Rightarrow \;1 - \mathcal {A}\mathcal {D}_{x\log {x}}^{|\cap |}(U,V) \overset{*,**}{=} \frac{ \sum _j^r \sum _i^k n_{ij}\log {n_{ij}} - \sum _i^k\sum _j^r \frac{n_{.j}n_{i.}}{n}\log {\frac{n_{.j}n_{i.}}{n}} }{ \frac{1}{2}\left[ \sum _j^r n_{.j}\log {n_{.j}} + \sum _i^k n_{i.}\log {n_{i.}}\right] - \sum _i^k\sum _j^r \frac{n_{.j}n_{i.}}{n}\log {\frac{n_{.j}n_{i.}}{n}}} \\&\quad = \frac{ n \sum _j^r \sum _i^k \frac{n_{ij}}{n}\log \frac{n_{ij}}{n} + n \log n - \sum _i^k\sum _j^r \frac{n_{.j}n_{i.}}{n} [ \log {\frac{n_{.j}}{n}}+\log {\frac{n_{i.}}{n}}+\log {n}] }{ \frac{n}{2}\left[ \sum _j^r \frac{n_{.j}}{n} \log {\frac{n_{.j}}{n}} + \sum _i^k \frac{n_{i.}}{n}\log \frac{n_{i.}}{n} + 2\log n \right] - \sum _i^k\sum _j^r \frac{n_{.j}n_{i.}}{n}[ \log {\frac{n_{.j}}{n}}+\log {\frac{n_{i.}}{n}}+\log {n}] } \\&\quad = \frac{ -H(U,V) + \log n - \sum _i^k\frac{n_{i.}}{n}\sum _j^r \frac{n_{.j}}{n} \log {\frac{n_{.j}}{n}}+\sum _i^k\frac{n_{.j}}{n} -\sum _j^r \frac{n_{i.}}{n}\log {\frac{n_{i.}}{n}}-\sum _i^k\sum _j^r \frac{n_{.j}n_{i.}}{n^2}\log {n} }{ \frac{1}{2}\left[ -H(U) - H(V) \right] + \log n + \sum _i^k\frac{n_{i.}}{n}H(V) +\sum _i^k\frac{n_{.j}}{n}H(U)-\log {n} }\\&\quad = \frac{ - H(U,V) + H(V) +H(U) }{ -\frac{1}{2}\left[ H(U) + H(V) \right] + H(V) +H(U) } = \frac{I(U,V)}{\frac{1}{2}\left[ H(U) + H(V) \right] } = NMI_{sum}(U,V)\\&\quad (*), (**) \text { same as proof of identity 1.} \end{aligned}$$

$\square$

1.6 Proof of Identity 5 and 6

First we prove that in general cases we have:

$$\begin{aligned} \Vert UU^T - VV^T\Vert ^2_F = \Vert U^TU\Vert ^2_F + \Vert V^TV\Vert ^2_F - 2\Vert U^TV\Vert ^2_F \end{aligned}$$

where $\Vert .\Vert ^2_F$ is squared Frob norm. This holds since we have:

$$\begin{aligned} \Vert UU^T - VV^T\Vert ^2_F&= \sum _{ij}(UU^T - VV^T)_{ij}^2\\&= \sum _{ij}(UU^T)_{ij}^2 + \sum _{ij}(VV^T)_{ij}^2 - 2\sum _{ij}(UU^T)_{ij}(VV^T)_{ij}\\&= \Vert UU^T\Vert ^2_F + \Vert VV^T\Vert ^2_F - 2 |UU^T \circ VV^T| \end{aligned}$$

Where the $\circ$ is element-wise matrix product, a.k.a. hadamard product, and |.| is sum of all elements in the matrix.¹⁶ The proof is complete with showing:

$$\begin{aligned} |UU^T \circ VV^T|&= tr ((UU^T)^T VV^T) = tr (V^TUU^TV)\\ {}&= tr ( (U^TV)^T U^TV) = ||U^TV||^2_F \\ ||UU^T||^2_F&=tr((UU^T)^T UU^T) =tr (U^TU U^TU) \\ {}&=tr ( (U^TU)^T U^TU) = ||U^TU||^2_F \end{aligned}$$

Now, we can prove the identities for the cases of disjoint hard clusters, using the notation $n_{ij} = (U^TV)_{ij}$, we have $\Vert U^TV\Vert ^2_F = \sum _{ij} n^2_{ij}$ and:

$$\begin{aligned} \Vert U^TU\Vert ^2_F&= \sum _{ij} <U_{.i},U_{.j}>^2 = \sum _{ij}\left( \sum _k u_{ki}u_{kj}\right) ^2 \overset{*}{=} \sum _{i} \left( \sum _k u^2_{ki}\right) ^2\\&\overset{**}{=} \sum _{i} \left( \sum _k u_{ki}\right) ^2 \overset{***}{=} \sum _{i} n_{i.}^2 \end{aligned}$$

$(*)$ with assumption that clusters are disjoint, $u_{ki}u_{kj}$ is only non-zero iff $i=j$

$(**)$ with the assumption that memberships are hard, $u_{ki}$ is either 0 or 1, therefore $u_{ki}= u^2_{ki}$

$(***)$ marginals of N give cluster sizes in U and V, i.e. $n_{i.} = \sum _{j} n_{ij} = \sum _{k} u_{ki}=|V_i|$

Therefore for disjoint hard clusters we get:

$$\begin{aligned} \Vert UU^T - VV^T\Vert ^2_F = \sum _{i} n_{i.}^2 +\sum _{j} n_{.j}^2 - 2 \sum _{ij} n^2_{ij} \end{aligned}$$

The RI normalization assumes that all pairs are in disagreement, i.e. $|{\mathbf {1}}_{n\times n}| = n^2$, since $max(UU^T)=1$ and, $max(VV^T)=1$. The ARI normalization compares $\varDelta$ to the difference where the two random variable of $UU^T_{ij}$ and $VV^T_{ij}$ are independent, in which case we would have:

$$\begin{aligned} E( UU^T_{ij}VV^T_{ij}) = E((UU^T)_{ij}) E((VV^T)_{ij}) \end{aligned}$$

which is calculated by:

$$\begin{aligned} \frac{\sum _{ij} ((UU^T)_{ij} (VV^T)_{ij} )}{n^2} = \frac{\sum _{ij} (UU^T)_{ij}}{n^2} \frac{\sum _{ij} (VV^T)_{ij}}{n^2} \end{aligned}$$

Since $\varDelta = ||UU^T - VV^T||^2_F = ||UU^T||^2_F + ||VV^T||^2_F - 2Sum (UU^T \circ VV^T )$, we have $ARI = 0$ or normalized distance 1, i.e. agreement no better than chance, when this independence condition holds, i.e.:

$$\begin{aligned} Sum (UU^T \circ VV^T ) = \frac{|UU^T||VV^T|}{n^2} \end{aligned}$$