Homophily outlier detection in non-IID categorical data

Abstract

Most of existing outlier detection methods assume that the outlier factors (i.e., outlierness scoring measures) of data entities (e.g., feature values and data objects) are Independent and Identically Distributed (IID). This assumption does not hold in real-world applications where the outlierness of different entities is dependent on each other and/or taken from different probability distributions (non-IID). This may lead to the failure of detecting important outliers that are too subtle to be identified without considering the non-IID nature. The issue is even intensified in more challenging contexts, e.g., high-dimensional data with many noisy features. This work introduces a novel outlier detection framework and its two instances to identify outliers in categorical data by capturing non-IID outlier factors. Our approach first defines and incorporates distribution-sensitive outlier factors and their interdependence into a value-value graph-based representation. It then models an outlierness propagation process in the value graph to learn the outlierness of feature values. The learned value outlierness allows for either direct outlier detection or outlying feature selection. The graph representation and mining approach is employed here to well capture the rich non-IID characteristics. Our empirical results on 15 real-world data sets with different levels of data complexities show that (i) the proposed outlier detection methods significantly outperform five state-of-the-art methods at the 95%/99% confidence level, achieving 10–28% AUC improvement on the 10 most complex data sets; and (ii) the proposed feature selection methods significantly outperform three competing methods in enabling subsequent outlier detection of two different existing detectors.

Appendices

A Proofs of the theorems

1.1 A.1 Proof of Theorem 1

Proof

If the graph $\mathsf {G}$ is irreducible and aperiodic, then based on the Perron–Frobenius Theorem (Meyer 2000), the URWs on $\mathsf {G}$ based on the adjacency matrix $\mathbf {A}$ will converge to a unique probability vector. $\square$

Lemma 1

(Equivalence between BRWs and URWs) BRWs based on the adjacency matrix $\mathbf {A}$ and the bias $\delta$ is equivalent to URWs on a graph $\mathsf {G}^{b}$ with an adjacency matrix $\mathbf {B}$, in which

$$\begin{aligned} \mathbf {B}(u,v)=\delta (u)\mathbf {A}(u,v)\delta (v), \; \forall u,v \in \mathcal {V}. \end{aligned}$$

(35)

This lemma holds iff the transition matrix $\mathbf {T}$ of $\mathsf {G}^{b}$ satisfies: $\mathbf {T} \equiv \mathbf {W}^{b}$. Since $\mathbf {B}(u,v)=\delta (u)\mathbf {A}(u,v)\delta (v)$, we have

$$\begin{aligned} \mathbf {T}(u,v)&=\frac{\mathbf {B}(u,v)}{\sum _{v \in \mathcal {V}}\mathbf {B}(u,v)}= \frac{\delta (u)\mathbf {A}(u,v)\delta (v)}{\sum _{v\in V}\delta (u)\mathbf {A}(u,v)\delta (v)} \\&=\frac{\mathbf {A}(u,v)\delta (v)}{\sum _{v\in V}\mathbf {A}(u,v)\delta (v)}=\mathbf {W}^{b}(u,v), \end{aligned}$$

which completes the proof of the lemma.

Since $\delta$ is always positive, the inclusion of $\delta$ into $\mathbf {A}$ does not change the graph’s irreducibility and aperiodicity. Based on Lemma 1, $\mathbf {B}$ and $\mathbf {A}$ have the same irreducibility and aperiodicity. Therefore, if $\mathsf {G}$ is irreducible and aperiodic, so is $\mathsf {G}^{b}$. We therefore have $\varvec{\pi }^{*}=\mathbf {W}^{b}\varvec{\pi }^{*}$. $\square$

1.2 A.2 Proof of Theorem 2

Proof

To prove Eq. (20), we need to show that when $\varvec{\pi }^{\prime }(u) =\frac{d^{\prime }(u)}{ vol (\mathsf {G}^{\prime })}$, $\forall u \in \mathcal {V}$, we have $\varvec{\pi }^{\prime }=\mathbf {W}^{b\prime }\varvec{\pi }^{\prime }$, i.e., $\varvec{\pi }^{\prime }$ becomes steady w.r.t. the time step.

First, the probability of visiting v is $\varvec{\pi }^{\prime ,t+1}(v) = \sum _{u\in \mathcal {V}}\varvec{\pi }^{\prime ,t}(u)\mathbf {W}^{b\prime }(u,v)$. We then have

$$\begin{aligned} \varvec{\pi }^{\prime ,t+1}(v) = \sum _{u\in \mathcal {V}}\varvec{\pi }^{\prime ,t}(u)\frac{\mathbf {B}^{\prime }(u,v)}{\sum _{w \in \mathcal {V}}\mathbf {B}^{\prime }_{u,w}}. \end{aligned}$$

When $\varvec{\pi }^{\prime ,t}(u) =\frac{d^{\prime }(u)}{ vol (\mathsf {G}^{\prime })}$, we have

$$\begin{aligned} \varvec{\pi }^{\prime ,t+1}(v) =\sum _{u\in \mathcal {V}}\frac{d^{\prime }(u)}{ vol (\mathsf {G}^{\prime })}\frac{\mathbf {B}^{\prime }(u,v)}{\sum _{w \in \mathcal {V}}\mathbf {B}^{\prime }_{u,w}}= \sum _{u\in \mathcal {V}}\frac{d^{\prime }(u)}{ vol (\mathsf {G}^{\prime })}\frac{\mathbf {B}^{\prime }(u,v)}{d^{\prime }(u)}=\sum _{u\in \mathcal {V}}\frac{\mathbf {B}^{\prime }(u,v)}{ vol (\mathsf {G}^{\prime })}. \end{aligned}$$

Since $\mathbf {B}^{\prime }(u,v)= \mathbf {B}^{\prime }(v,u)$, we further have

$$\begin{aligned} \varvec{\pi }^{\prime ,t+1}(v) =\sum _{u\in \mathcal {V}}\frac{\mathbf {B}^{\prime }(u,v)}{ vol (\mathsf {G}^{\prime })}=\sum _{u\in \mathcal {V}}\frac{\mathbf {B}^{\prime }(v,u)}{ vol (\mathsf {G}^{\prime })}= \frac{d^{\prime }(v)}{ vol (\mathsf {G}^{\prime })}. \end{aligned}$$

Therefore, we also have $\varvec{\pi }^{\prime ,t+1}(u)=\frac{d^{\prime }(u)}{ vol (\mathsf {G}^{\prime })}=\varvec{\pi }^{\prime ,t}(u)$, i.e., $\varvec{\pi }^{\prime }$ becomes steady. $\square$

1.3 A.3 Proof of Theorem 3

Proof

We show $diff =\left( \delta (u^{\prime }) - \phi _{ freq }(u^{\prime })\right) - \left( \delta (v^{\prime }) - \phi _{ freq }(v^{\prime })\right) > 0$ to complete the proof.

First, we have

$$\begin{aligned} \delta (u^{\prime }) - \phi _{ freq }(u^{\prime })&= \left( 1 - freq (m) + \frac{ freq (m) - freq (u^{\prime })}{ freq (m)}\right) - \left( 1 - freq (u^{\prime })\right) \\&= \frac{ supp (m)N- supp (u^{\prime })N + supp (m) supp (u^{\prime })- supp (m)^{2}}{ supp (m)N}. \end{aligned}$$

Let $C = N - supp (m)$ and $H = supp (m) + supp (u^{\prime })$. Then after some algebra, we have

$$\begin{aligned} \delta (u^{\prime }) - \phi _{ freq }(u^{\prime }) = \frac{NH- supp (m)H}{ supp (m)N}. \end{aligned}$$

Similarly, we can obtain

$$\begin{aligned} \delta (v^{\prime }) - \phi _{ freq }(v^{\prime }) = \frac{ supp (m)N- supp (m)^{2}- supp (v^{\prime })N+ supp (v^{\prime }) supp }{ supp (m)N}. \end{aligned}$$

Let $supp (v^{\prime }) = supp (m) - I$. Then after some algebra, we obtain

$$\begin{aligned} \delta (v^{\prime }) - \phi _{ freq }(v^{\prime }) = \frac{NI- supp (m)I}{ supp (m)N}. \end{aligned}$$

Therefore,

$$\begin{aligned} diff&= \frac{NH- supp (m)H}{ supp (m)N} - \frac{NI- supp (m)I}{ supp (m)N} \\&= \frac{(N- supp (m))(H- I)}{ supp (m)N}. \end{aligned}$$

We always have $N> supp (m)$. Moreover, as $u^{\prime }$ and $v^{\prime }$ are outlying and normal values respectively, we have $supp (u^{\prime }) < supp (v^{\prime })$, and thus $H > I$. Therefore, $\left( \delta (u^{\prime }) - \phi _{ freq }(u^{\prime })\right) - \left( \delta (v^{\prime }) - \phi _{ freq }(v^{\prime })\right) >0$. $\square$

B Convergence and sensitivity test results of CBRW

The empirical convergence analysis for CBRW is provided in the first subsection, followed by the sensitivity test of CBRW w.r.t. the parameter $\alpha$.

Table 8 Two key properties of a value graph

Fig. 4

Convergence test results

1.1 B.1 Convergence test

The convergence rate of random walks is governed by two key graph properties - the graph diameter and the Cheeger constant (Diaconis and Stroock 1991; Fill 1991). The runtime for computing the Cheeger constant is prohibitive for large graphs, so we replace this constant with clustering coefficients. The graph’s diameter and clustering coefficients of the value graph for each data set are presented in Table 8. It is clear that all the value graphs has small graph diameter and large clustering coefficient. This is because a value in one feature often co-occurs with most, if not all, of the values in other features. Moreover, there exist linkages between values as long as the values co-occur together, resulting in a highly connected dense value graph. Fast convergence rates are expected for random walks on such graphs (Diaconis and Stroock 1991; Fill 1991).

The convergence test results in Fig. 4 show that CBRW converges quickly on all 15 data sets, i.e., within 70 iterations. CBRW converges after about 10 iterations on 13 data sets, but takes about 70 iterations to converge on Probe and U2R. This is because these two data sets contain a large proportion of feature values having frequencies of less than three. This is particularly true for Probe. As a result, although their overall clustering coefficient is high, its Cheeger constant can be quite small, which leads to slower convergence.

1.2 B.2 Sensitivity test w.r.t. the damping factor $\alpha$

CBRW only has one parameter, the damping factor $\alpha$. The use of $\alpha$ is to avoid the random walking getting stuck in isolated nodes by offering a small restart probability $(1-\alpha )$, which guarantees the algorithmic convergence while does not affect the effectiveness. $\alpha =1.0$ is not recommended as this may break the convergence condition. Also, $\alpha$ should be sufficiently large, e.g., $\alpha \ge 0.85$, and the underlying graph structure is ignored otherwise. Below we examine the sensitivity of CBRW w.r.t. $\alpha$ in a wide range of values [0.85, 0.99] by performing direct outlier detection (i.e., $\text {CBRW}_{\mathrm{od}}$ is used). Fig. 5 reports the AUC results w.r.t. $\alpha$ on all 15 data sets.

Fig. 5

Sensitivity test results w.r.t. the parameter $\alpha$

Table 9 Transition matrix resulted in CBRW for the value graph derived from Table 1

Table 10 Value outlierness yielded by CBRW

The results show that CBRW performs very stably over a large range of tuning options on most of the data sets, and a large $\alpha$ is more preferable than a small one. This is because (i) $\alpha$ is introduced to guarantee the convergence of the CBRW algorithm and it is data-insensitive in terms of effectiveness, which is different from some data-sensitive parameters in other detectors, such as the minimum support in FPOF and the subsampling size in iForest; and (ii) the graph structure and edges weights are carefully designed to highlight the outlying values, and we need to make use of this graph nature by setting a large $\alpha$. A large $\alpha$ is needed to achieve the best performance on some data sets, e.g., U2R, APAS, w7a and AD. These data sets may contain some highly noisy values. A large $\alpha$ is required to increase the gap between the outlierness of outlying values and the highly noisy values. On the other hand, a medium $\alpha$ is needed to obtain the best performance on other data sets, like CT. This may be because some outlying values in these data sets cannot attract sufficiently large outlierness in the original graph structure, but rather rely on some outlierness propagated through restart probabilities. Therefore, we recommend using a relatively large $\alpha$ (e.g., $\alpha =0.95$) to leverage both cases.

C Key outputs of CBRW and SDRW

We provide several key outputs of CBRW to enable an in-depth understanding of its algorithmic procedures. All these outputs are built upon the toy dataset in Table 1. In Table 9 we present the transition matrix used by the BRWs in CBRW, i.e., $\mathbf {W}^b$ in Eq. (8), in which each entry $\mathbf {W}^b(u, v)$ is determined by the inter-feature outlierness influence $\eta$ and the intra-feature outlier factor $\delta$. After having $\mathbf {W}^b$, the power iteration method is used to perform random walks and obtains a stationary probability vector $\varvec{\pi }^{*}$. In CBRW, these stationary probabilities are used as the outlierness of the values in the value-value graph according to Eq. (10). Table 10 shows the outlierness of each value of the toy dataset, with each outlierness corresponding to one entry of $\varvec{\pi }^{*}$. It is then followed by the calculation of the outlierness of data objects in Eq. (29) based on the value outlierness. Finally, CBRW produces the outlierness of all data objects as in Table 11. The first data object that is the only genuine outlier is assigned with larger outlierness than all data objects, including the noisy data object #10.

Table 11 Final object outlierness yielded by CBRW

We also provide similar outputs for SDRW. In Table 12 we present the adjacency matrix used by SDRW, i.e., $\mathbf {C}$ in Eq. (13). Note that the value graph in SDRW is undirected, so we have $\mathbf {C}(u,v)=\mathbf {C}(v,u)$. We then incorporate the subgraph density-based outlier factor into the matrix and calculate the value outlierness using the closed-form solution in Eq. (21). The resulting value outlierness is shown in Table 13. SDRW finally uses the same object outlierness calculation as in CBRW, i.e., Eq. (29) to obtain the object-level outlier scores. As shown in Table 14, SDRW can also easily identify the outliers in the toy dataset.

Table 12 Adjacency matrix resulted in SDRW for the value graph derived from table 1

Table 13 Value outlierness yielded by SDRW

Table 14 Final object outlierness yielded by SDRW

It should be noted that the above results are built upon a simple synthetic toy dataset to demonstrate the procedure of our methods; they do not imply any bias and discrimination issues in the applications of our methods to real-world datasets.