Outlyingness: Which variables contribute most?

Abstract

Outlier detection is an inevitable step to most statistical data analyses. However, the mere detection of an outlying case does not always answer all scientific questions associated with that data point. Outlier detection techniques, classical and robust alike, will typically flag the entire case as outlying, or attribute a specific case weight to the entire case. In practice, particularly in high dimensional data, the outlier will most likely not be outlying along all of its variables, but just along a subset of them. If so, the scientific question why the case has been flagged as an outlier becomes of interest. In this article, a fast and efficient method is proposed to detect variables that contribute most to an outlier’s outlyingness. Thereby, it helps the analyst understand in which way an outlier lies out. The approach pursued in this work is to estimate the univariate direction of maximal outlyingness. It is shown that the problem of estimating that direction can be rewritten as the normed solution of a classical least squares regression problem. Identifying the subset of variables contributing most to outlyingness, can thus be achieved by estimating the associated least squares problem in a sparse manner. From a practical perspective, sparse partial least squares (SPLS) regression, preferably by the fast sparse NIPALS (SNIPLS) algorithm, is suggested to tackle that problem. The performed method is demonstrated to perform well both on simulated data and real life examples.

A Appendix: Proofs

A.1 Proof of Proposition 1

Proof

Note that our weighted covariance matrix $\hat{\varvec{\varSigma }}_w$, like all covariance matrices, is a positive-semidefinite matrix. Since we also assume it is not singular and $\hat{\varvec{\varSigma }}_w^{-1}$ exists, we know that $\hat{\varvec{\varSigma }}_w$ is positive-definite. We now apply the Cauchy- Bunyakovskiy-Schwarz inequality to $\varvec{x}= \hat{\varvec{\varSigma }}_w^{-1/2}\varvec{x}_1$ and $\varvec{y}= \hat{\varvec{\varSigma }}_w^{1/2}\varvec{y}_1$, for arbitrary $\varvec{x}_1,\varvec{y}_1 \in \mathbb {R}^p$. This results in the following inequality

$$\begin{aligned} (\varvec{x}_1^T\varvec{y}_1)^2 \le \varvec{x}_1^T\hat{\varvec{\varSigma }}_w^{-1}\varvec{x}_1 \varvec{y}_1^T \hat{\varvec{\varSigma }}_w\varvec{y}_1 \end{aligned}$$

We have equality if $\varvec{y}= c\varvec{x}$ with $c\in \mathbb {R}$, which means $\hat{\varvec{\varSigma }}_w^{1/2}\varvec{y}_1 = c \hat{\varvec{\varSigma }}_w^{-1/2}\varvec{x}_1$ or $\varvec{y}_1 = c \hat{\varvec{\varSigma }}_w^{-1}\varvec{x}_1$. So summarized, for any $\varvec{x},\varvec{y}\in \mathbb {R}^p$ we have the inequality

$$\begin{aligned} (\varvec{x}^T\varvec{y})^2 \le \varvec{x}^T\hat{\varvec{\varSigma }}_w^{-1}\varvec{x}\varvec{y}^T \hat{\varvec{\varSigma }}_w\varvec{y}, \end{aligned}$$

where there is equality if and only if $\varvec{y}= c \hat{\varvec{\varSigma }}_w^{-1}\varvec{x}$.

We now look at

$$\begin{aligned} \frac{(\varvec{x}^T\varvec{a}- \hat{\varvec{\mu }}_w^T\varvec{a})^2}{\varvec{a}^T\hat{\varvec{\varSigma }}_w\varvec{a}} \end{aligned}$$

and apply this inequality:

$$\begin{aligned} \frac{((\varvec{x}- \hat{\varvec{\mu }}_w)^T\varvec{a})^2}{\varvec{a}^T\hat{\varvec{\varSigma }}_w\varvec{a}}\le & {} \frac{(\varvec{x}- \hat{\varvec{\mu }}_w)^T \hat{\varvec{\varSigma }}_w^{-1}(\varvec{x}- \hat{\varvec{\mu }}_w) \varvec{a}^T \hat{\varvec{\varSigma }}_w \varvec{a}}{\varvec{a}^T\hat{\varvec{\varSigma }}_w\varvec{a}}\\= & {} (\varvec{x}- \hat{\varvec{\mu }}_w)^T \hat{\varvec{\varSigma }}_w^{-1}(\varvec{x}- \hat{\varvec{\mu }}_w). \end{aligned}$$

We have equality in the above inequality if

$\varvec{a}= c \hat{\varvec{\varSigma }}_w^{-1}(\varvec{x}- \hat{\varvec{\mu }}_w)$. So

$$\begin{aligned} \varvec{a}= \frac{\hat{\varvec{\varSigma }}_w^{-1}(\varvec{x}- \hat{\varvec{\mu }}_w)}{\Vert \hat{\varvec{\varSigma }}_w^{-1}(\varvec{x}- \hat{\varvec{\mu }}_w)\Vert } \end{aligned}$$

is the direction $\varvec{a}$ that maximizes

$$\begin{aligned} \frac{|\varvec{x}^T\varvec{a}- \hat{\varvec{\mu }}_w^T\varvec{a}|}{\sqrt{\varvec{a}^T\hat{\varvec{\varSigma }}_w\varvec{a}}} \end{aligned}$$

and for this $\varvec{a}$ we have

$$\begin{aligned}&\left( \frac{|\varvec{x}^T\varvec{a}- \hat{\varvec{\mu }}_w^T\varvec{a}|}{\sqrt{\varvec{a}^T\hat{\varvec{\varSigma }}_w\varvec{a}}}\right) ^2\\&\quad = (\varvec{x}- \hat{\varvec{\mu }}_w)^T \hat{\varvec{\varSigma }}_w^{-1}(\varvec{x}- \hat{\varvec{\mu }}_w) = r(\varvec{x};\varvec{X})^2. \end{aligned}$$

$\square$

A.2 Proof of Theorem 1

Proof

We know that, by the theory of ordinary least squares regression,

$$\begin{aligned} \varvec{\theta }_{\varepsilon } = (\varvec{X}_{w,\varepsilon }^T\varvec{X}_{w,\varepsilon })^{-1}\varvec{X}_{w,\varepsilon }^T \varvec{y}^{n+1}_{w,\varepsilon } \end{aligned}$$

and by the definition of our weighted covariance matrix, $\hat{\varvec{\varSigma }}_{w,\varepsilon } = \frac{1}{n_{w,\varepsilon }-1} \varvec{X}_{w,\varepsilon }^T\varvec{X}_{w,\varepsilon }$, we can write

$$\begin{aligned} \varvec{\theta }_{\varepsilon } = ((n_{w,\varepsilon }-1)\hat{\varvec{\varSigma }}_{w,\varepsilon })^{-1}\varvec{X}_{w,\varepsilon }^T\varvec{y}^{n+1}_{w,\varepsilon }. \end{aligned}$$

We know that $((n_{w,\varepsilon }-1)\hat{\varvec{\varSigma }}_{w,\varepsilon })^{-1} = \frac{1}{n_{w,\varepsilon }-1} \hat{\varvec{\varSigma }}_{w,\varepsilon }^{-1}$ and it is easy to see that $\varvec{X}^T_{w,\varepsilon }\varvec{y}^{n+1}_{w,\varepsilon } = \sqrt{\varepsilon }(\varvec{x}- \hat{\varvec{\mu }}_{w,\varepsilon })$, if we look at the definitions of $\varvec{X}_{w,\varepsilon }$ and $\varvec{y}^{n+1}_{w,\varepsilon }$. Thus we have that

$$\begin{aligned} \varvec{\theta }_{\varepsilon } = \frac{\sqrt{\varepsilon }}{n_{w,\varepsilon }-1} \hat{\varvec{\varSigma }}_{w,\varepsilon }^{-1}(\varvec{x}- \hat{\varvec{\mu }}_{w,\varepsilon }). \end{aligned}$$

Since $\varepsilon$ is strictly larger than zero, we have that

$$\begin{aligned} \frac{\varvec{\theta }_{\varepsilon }}{\Vert \varvec{\theta }_{\varepsilon }\Vert } = \frac{\hat{\varvec{\varSigma }}_{w,\varepsilon }^{-1}(\varvec{x}- \hat{\varvec{\mu }}_{w,\varepsilon })}{\Vert \hat{\varvec{\varSigma }}_{w,\varepsilon }^{-1}(\varvec{x}- \hat{\varvec{\mu }}_{w,\varepsilon })\Vert }. \end{aligned}$$

Then we get that

$$\begin{aligned} \lim _{\varepsilon \rightarrow 0}\frac{\varvec{\theta }_{\varepsilon }}{\Vert \varvec{\theta }_{\varepsilon }\Vert }=\frac{\hat{\varvec{\varSigma }}_w^{-1}(\varvec{x}- \hat{\varvec{\mu }}_w)}{\Vert \hat{\varvec{\varSigma }}_w^{-1}(\varvec{x}- \hat{\varvec{\mu }}_w)\Vert } = \varvec{a}(\varvec{x}) \end{aligned}$$

since $\lim _{\varepsilon \rightarrow 0}n_{w,\varepsilon }=n_w$, $\lim _{\varepsilon \rightarrow 0}\hat{\varvec{\mu }}_{w,\varepsilon }=\hat{\varvec{\mu }}_w$ and

$\lim _{\varepsilon \rightarrow 0}\hat{\varvec{\varSigma }}_{w,\varepsilon }^{-1}=\hat{\varvec{\varSigma }}_w^{-1}$. $\square$