Missing Values and Directional Outlier Detection in Model-Based Clustering

Abstract

Model-based clustering tackles the task of uncovering heterogeneity in a data set to extract valuable insights. Given the common presence of outliers in practice, robust methods for model-based clustering have been proposed. However, the use of many methods in this area becomes severely limited in applications where partially observed records are common since their existing frameworks often assume complete data only. Here, a mixture of multiple scaled contaminated normal (MSCN) distributions is extended using the expectation-conditional maximization (ECM) algorithm to accommodate data sets with values missing at random. The newly proposed extension preserves the mixture’s capability in yielding robust parameter estimates and performing automatic outlier detection separately for each principal component. In this fitting framework, the MSCN marginal density is approximated using the inversion formula for the characteristic function. Extensive simulation studies involving incomplete data sets with outliers are conducted to evaluate parameter estimates and to compare clustering performance and outlier detection of our model to other mixtures.

Appendices

Appendix A:   Characteristic Functions and the Inversion Formula

In statistics, characteristic functions provide a powerful tool for deriving probability density functions by means of Fourier transformations. One major advantage of this approach is that there always exists a unique characteristic function for every probability distribution.

Definition A.1

Let $\varvec{X}= \left( X_1, \dots , X_p \right) ^\top \in \mathbb {R}^p$ be a p-variate random vector, $\varvec{t}= \left( t_1, \dots , t_p \right) ^\top \in \mathbb {R}^p$, and i be an imaginary unit. The function

$$\begin{aligned} \phi _{\varvec{X}} (\varvec{t}) = E \left( \exp ( i\varvec{t}^\top \varvec{X}) \right) \end{aligned}$$

(39)

is called the characteristic function of $\varvec{X}$.

From a characteristic function, the associated probability density function can be obtained using the inversion formula.

Theorem A.1

(Inversion Formula) Let $\varvec{X}= \left( X_1, \dots , X_p \right) ^\top \in \mathbb {R}^p$ be a p-variate random vector, $\phi _{\varvec{X}} (\varvec{t})$ be the characteristic function of $\varvec{X}$ with $\varvec{t}= \left( t_1, \dots , t_p \right) ^\top \in \mathbb {R}^p$, and i be an imaginary unit. The probability density function of $\varvec{X}$ can be obtained by

$$\begin{aligned} f_{\varvec{X}} (\varvec{x})&= (2 \pi )^{-p} {\int _{-\infty }^\infty } \dots {\int _{-\infty }^\infty } \exp \left( -i \varvec{t}^\top \varvec{x}\right) \phi _{\varvec{X}} (\varvec{t}) \, d\varvec{t}\nonumber \\&= (2 \pi )^{-p} {\int _{-\infty }^\infty } \dots {\int _{-\infty }^\infty } \exp \left( -i \sum _{j = 1}^p t_j x_j \right) \phi _{\varvec{X}} (t_1, \dots , t_p) \, dt_1 \dots dt_p. \end{aligned}$$

(40)

To obtain the marginals of the MSCN distribution, the propositions describing the characteristic functions of the MN and MCN distributions are needed. The marginals of the MCN and MSCN distribution are outlined in the methodology under Sect.  3.

Proposition A.1

The characteristic function of a p-variate random vector $\varvec{X}\!=\! \left( X_1, \dots , X_p \right) ^\top$ $\in \mathbb {R}^p$ that follows a multivariate normal distribution with mean vector $\varvec{\mu }$ and covariance matrix $\varvec{\Sigma }$ is

$$\begin{aligned} \phi _{\varvec{X}} (\varvec{t}) = \exp \left( i \varvec{t}^\top \varvec{\mu }- \frac{1}{2} \varvec{t}^\top \varvec{\Sigma }\varvec{t}\right) , \end{aligned}$$

(41)

where $\varvec{t}= \left( t_1, \dots , t_p \right) ^\top \in \mathbb {R}^p$ and i is an imaginary unit.

Appendix B:   Proofs

Proposition 3.1

Proof

For data generation purposes, the MCN random variable $\varvec{X}$ can be represented as

$$\begin{aligned} \varvec{X}= \left( V + \frac{1 - V}{\eta } \right) ^{-1/2} \varvec{Y}, \end{aligned}$$

where V follows a Bernoulli distribution such that $V = 1$ with probability $\alpha \in (0.5, 1)$ and $V = 0$ with probability $1 - \alpha$; and $\varvec{Y}$ follows an MN distribution with mean vector $\varvec{\mu }$ and covariance matrix $\varvec{\Sigma }$. By Definition A.1 and the law of total expectation, we can establish the following

$$\begin{aligned} \phi _X (t) = E ( \exp ( i\varvec{t}^\top \varvec{X}) )&= \sum _{v = 0}^1 E ( \exp ( i\varvec{t}^\top \varvec{X}) | V = v ) P (V = v) \\&= \alpha E ( \exp ( i\varvec{t}^\top \varvec{Y}) ) + (1 - \alpha ) E ( \exp ( i \eta ^{1/2} \varvec{t}^\top \varvec{Y}) ) \\&= \alpha \phi _{\varvec{Y}} (\varvec{t}) + (1 - \alpha ) \phi _{\varvec{Y}} (\eta ^{1/2} \varvec{t}) \\&= \alpha \exp \left( i \varvec{t}^\top \varvec{\mu }- \frac{1}{2} \varvec{t}^\top \varvec{\Sigma }\varvec{t}\right) + (1 - \alpha ) \exp \left( i\varvec{t}^\top \varvec{\mu }- \frac{1}{2} \eta \varvec{t}^\top \varvec{\Sigma }\varvec{t}\right) . \end{aligned}$$

$\square$

Proposition 3.2

Proof

From Definition A.1 and the fact that $\tilde{\varvec{Y}}$ contains p independent univariate contaminated normal random variables, the characteristic function of the marginal variable $\varvec{X}_1$ is given by

$$\begin{aligned} \phi _{\varvec{X}_1} (\varvec{t}) = E \left( \exp ( i\varvec{t}^\top \varvec{X}_1 ) \right) = \prod _{j = 1}^q \exp ( i t_j \mu _j ) \prod _{h = 1}^p \phi _{\tilde{Y}_h} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{jh} \right) , \end{aligned}$$

where from Proposition 3.1, for $h = 1, \dots p$, we know that

$$\begin{aligned}{} & {} \phi _{\tilde{Y}_h} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{jh} \right) = \alpha _h \exp \left[ - \frac{1}{2} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^ {1/2}]_{jh} \right) ^2 \right] \\{} & {} + (1 - \alpha _h) \exp \left[ - \frac{1}{2} \eta _h \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{jh} \right) ^2 \right] . \end{aligned}$$

$\square$

Proposition 3.6

Proof

From Definition A.1 and the fact that we are dealing with linear combinations of independent random variables, we have the characteristic function of $\varvec{X}_1 { \; \mid \; }V_r = v_r, r \in \mathcal {A}$ to be

$$\begin{aligned} \phi _{\varvec{X}_1 { \; \mid \; }V_r, r \in \mathcal {A}} (\varvec{t}) = \prod _{j = 1}^q \exp ( i t_j \mu _j ) \prod _{r \in \mathcal {A}} \phi _{\tilde{Y}_r} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{jr} \right) \prod _{s \in \mathcal {B}} \phi _{\tilde{Y}_s} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{js} \right) . \end{aligned}$$

Herein, for $r \in \mathcal {A}$, $\tilde{Y}_r$ follows a univariate normal distribution with mean 0 and variance 1 if $v_r = 1$ or variance $\eta _r$ if $v_r = 0$. Thus,

$$\begin{aligned}{} & {} \phi _{\tilde{Y}_r} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{jr} \right) \\{} & {} = \left\{ \exp \left[ - \frac{1}{2} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^ {1/2}]_{jr} \right) ^2 \right] \right\} ^{v_r} \left\{ \exp \left[ - \frac{1}{2} \eta _r \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{jr} \right) ^2 \right] \right\} ^ {1 - v_r}. \end{aligned}$$

On the other hand, for $s \in \mathcal {B}$, $\tilde{Y}_s$ follows a univariate contaminated normal distribution with mean 0, variance 1, proportion of good observation $\alpha _s$, and degree of contamination $\eta _s$. As the result,

$$\begin{aligned}{} & {} \phi _{\tilde{Y}_s} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{js} \right) \\{} & {} = \left\{ \alpha _s \exp \left[ - \frac{1}{2} \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{js} \right) ^2 \right] + (1 - \alpha _s) \exp \left[ - \frac{1}{2} \eta _s \left( \sum _{j = 1}^q t_j [\varvec{\Gamma }\varvec{\Lambda }^{1/2}]_{js} \right) ^2 \right] \right\} . \end{aligned}$$

$\square$