Mixtures of multivariate contaminated normal regression models

Abstract

Mixtures of regression models (MRMs) are widely used to investigate the relationship between variables coming from several unknown latent homogeneous groups. Usually, the conditional distribution of the response in each mixture component is assumed to be (multivariate) normal (MN-MRM). To robustify the approach with respect to possible elliptical heavy-tailed departures from normality, due to the presence of mild outliers, the multivariate contaminated normal MRM is here introduced. In addition to the parameters of the MN-MRM, each mixture component has a parameter controlling the proportion of outliers and one specifying the degree of contamination with respect to the response variable(s). Crucially, these parameters do not have to be specified a priori, adding flexibility to our approach. Furthermore, once the model is estimated and the observations are assigned to the groups, a finer intra-group classification in typical points and (mild) outliers, can be directly obtained. Identifiability conditions are provided, an expectation-conditional maximization algorithm is outlined for parameter estimation, and various implementation and operational issues are discussed. Properties of the estimators of the regression coefficients are evaluated through Monte Carlo experiments and compared with other procedures. The performance of this novel family of models is also illustrated on artificial and real data, with particular emphasis to the application in allometric studies.

Appendix A: Updates in the first CM-step

The estimates of $\pi _j, {\varvec{B}}_j, \varvec{\varSigma }_j$, and $\alpha _j, j=1,\ldots ,k$, at the $\left( r+1\right)$th first CM-step of the ECM algorithm, require the maximization of

$$\begin{aligned} Q\left( \varvec{\vartheta }_1|\varvec{\vartheta }^{\left( r\right) }\right) =Q_1\left( \varvec{\pi }|\varvec{\vartheta }^{\left( r\right) }\right) +Q_2\left( \varvec{\alpha }|\varvec{\vartheta }^{\left( r\right) }\right) +Q_3\left( {\varvec{B}},\varvec{\varSigma }|\varvec{\vartheta }^{\left( r\right) }\right) , \end{aligned}$$

(14)

where

$$\begin{aligned} Q_1\left( \varvec{\pi }|\varvec{\vartheta }^{\left( r\right) }\right)&=\sum _{i=1}^{n}\sum _{j=1}^{k}z_{ij}^{\left( r\right) }\ln \pi _j,\\ Q_2\left( \varvec{\alpha }|\varvec{\vartheta }^{\left( r\right) }\right)&=\sum _{i=1}^{n}\sum _{j=1}^{k}z_{ij}^{\left( r\right) }\left[ u_{ij}^{\left( r\right) }\ln \alpha _j+\left( 1-u_{ij}^{\left( r\right) }\right) \ln \left( 1-\alpha _j\right) \right] ,\\ Q_3\left( {\varvec{B}},\varvec{\varSigma }|\varvec{\vartheta }^{\left( r\right) }\right)&=-\frac{1}{2}\sum _{i=1}^n\sum _{j=1}^k\Biggl \{z_{ij}^{\left( r\right) }\ln \left| \varvec{\varSigma }_j\right| \\&\quad +z_{ij}^{\left( r\right) }\left( u_{ij}^{\left( r\right) }+\frac{1-u_{ij}^{\left( r\right) }}{\eta _j^{\left( r\right) }}\right) \delta \left( {\varvec{x}}_i,\varvec{\mu }\left( {\varvec{x}}_i;{\varvec{B}}_j\right) ;\varvec{\varSigma }_j\right) \Biggr \}. \end{aligned}$$

Terms which are independent by the parameters of interest have been removed from $Q_3$. As the three terms on the right-hand side of (14) have zero cross-derivatives, they can be maximized separately.

1.1 A.1 Update of $\varvec{\pi }$

The maximum of $Q_1\left( \varvec{\pi }|\varvec{\vartheta }^{\left( r\right) }\right)$ with respect to $\varvec{\pi }$, subject to the constraints on those parameters, is obtained by maximizing the augmented function

$$\begin{aligned} \sum _{i=1}^n\sum _{j=1}^kz_{ij}^{\left( r\right) }\ln \pi _j-\lambda \left( \sum _{j=1}^k\pi _j-1\right) , \end{aligned}$$

(15)

where $\lambda$ is a Lagrangian multiplier. Setting the derivative of equation (15) with respect to $\pi _j$ equal to zero and solving for $\pi _j$ yields

$$\begin{aligned} \pi _j^{\left( r+1\right) }=\displaystyle \displaystyle \sum _{i=1}^nz_{ij}^{\left( r\right) }\Big /n. \end{aligned}$$

1.2 A.2 Update of $\varvec{\alpha }_{{\varvec{Y}}}$

The updates for $\varvec{\alpha }$ can be obtained through the first partial derivatives

$$\begin{aligned} \frac{\partial Q_2\left( \varvec{\alpha }|\varvec{\vartheta }^{\left( r\right) }\right) }{\partial \alpha _j} = \frac{\displaystyle \sum _{i=1}^n z_{ij}^{\left( r\right) }u_{ij}^{\left( r\right) }-\alpha _j\sum _{i=1}^n z_{ij}^{\left( r\right) }}{\alpha _j\left( 1-\alpha _j\right) }, \qquad j=1,\ldots ,k. \end{aligned}$$

(16)

Equating (16) to zero yields

$$\begin{aligned} \alpha _j^{\left( r+1\right) }=\displaystyle \sum _{i=1}^n z_{ij}^{\left( r\right) }u_{ij}^{\left( r\right) }\Bigg /\displaystyle \sum _{i=1}^n z_{ij}^{\left( r\right) }, \qquad j=1,\ldots ,k. \end{aligned}$$

1.3 A.3 Update of ${\varvec{B}}$ and $\varvec{\varSigma }_{{\varvec{Y}}}$

Using properties of trace and transpose, the updates for ${\varvec{B}}$ can be obtained through the first partial derivatives

$$\begin{aligned}&\frac{\partial Q_3\left( {\varvec{B}},\varvec{\varSigma }_{{\varvec{Y}}}|\varvec{\vartheta }^{\left( r\right) }\right) }{ \partial {\varvec{B}}_j'} \nonumber \\&\quad = \frac{\partial \left\{ -\displaystyle \frac{1}{2}\displaystyle \sum _{i=1}^nz_{ij}^{\left( r\right) }\left( u_{ij}^{\left( r\right) }+\frac{1-u_{ij}^{\left( r\right) }}{\eta _j^{\left( r\right) }}\right) \left[ {\varvec{y}}_i-\varvec{\mu }\left( {\varvec{x}}_i;{\varvec{B}}_j\right) \right] '\varvec{\varSigma }_j^{-1}\left[ {\varvec{y}}_i-\varvec{\mu }\left( {\varvec{x}}_i;{\varvec{B}}_j\right) \right] \right\} }{\partial {\varvec{B}}_j'}\nonumber \\&\quad = \frac{\partial \left[ -\displaystyle \frac{1}{2}\displaystyle \sum _{i=1}^nz_{ij}^{\left( r\right) }\left( u_{ij}^{\left( r\right) }+\frac{1-u_{ij}^{\left( r\right) }}{\eta _j^{\left( r\right) }}\right) \left( -{\varvec{y}}_i'\varvec{\varSigma }_j^{-1}{\varvec{B}}_j'{\varvec{x}}_i^*-{\varvec{x}}_i^{*'}{\varvec{B}}_j\varvec{\varSigma }_j^{-1}{\varvec{y}}_i+{\varvec{x}}_i^{*'}{\varvec{B}}_j\varvec{\varSigma }_j^{-1}{\varvec{B}}_j'{\varvec{x}}_i^*\right) \right] }{\partial {\varvec{B}}_j'} \nonumber \\&\quad = \frac{\partial \left\{ \displaystyle \frac{1}{2}\displaystyle \sum _{i=1}^nz_{ij}^{\left( r\right) }\left( u_{ij}^{\left( r\right) }+\frac{1-u_{ij}^{\left( r\right) }}{\eta _j^{\left( r\right) }}\right) \left[ \text{ tr }\left( {\varvec{B}}_j'{\varvec{x}}_i^*{\varvec{y}}_i'\varvec{\varSigma }_j^{-1}\right) +\text{ tr }\left( \left( \varvec{\varSigma }_j^{-1}{\varvec{y}}_i{\varvec{x}}_i^{*'}\right) '{\varvec{B}}_j'\right) -\text{ tr }\left( {\varvec{B}}_j'{\varvec{x}}_i^*{\varvec{x}}_i^{*'}{\varvec{B}}_j\varvec{\varSigma }_j^{-1}\right) \right] \right\} }{\partial {\varvec{B}}_j'}\nonumber \\&\quad =\displaystyle \frac{1}{2}\displaystyle \sum _{i=1}^nz_{ij}^{\left( r\right) }\left( u_{ij}^{\left( r\right) }+\frac{1-u_{ij}^{\left( r\right) }}{\eta _j^{\left( r\right) }}\right) \left( 2\varvec{\varSigma }_j^{-1}{\varvec{y}}_i{\varvec{x}}_i^{*'}-2\varvec{\varSigma }_j^{-1}{\varvec{B}}_j'{\varvec{x}}_i^*{\varvec{x}}_i^{*'}\right) , \qquad j=1,\ldots ,k. \end{aligned}$$

(17)

Equating (17) to the null matrix yields

$$\begin{aligned} {\varvec{B}}_j^{\left( r+1\right) }= & {} \left[ \sum _{i=1}^n z_{ij}^{\left( r\right) }\left( u_{ij}^{\left( r\right) }+\frac{1-u_{ij}^{\left( r\right) }}{\eta _j^{\left( r\right) }}\right) {\varvec{x}}_i^*{\varvec{x}}_i^{*'}\right] ^{-1}\\&\left[ \sum _{i=1}^n z_{ij}^{\left( r\right) }\left( u_{ij}^{\left( r\right) }+\frac{1-u_{ij}^{\left( r\right) }}{\eta _j^{\left( r\right) }}\right) {\varvec{x}}_i^*{\varvec{y}}_i\right] , \qquad j=1,\ldots ,k. \end{aligned}$$

Finally, the updates for $\varvec{\varSigma }$ can be obtained through the first partial derivatives

$$\begin{aligned} \frac{\partial Q_3\left( {\varvec{B}},\varvec{\varSigma }|\varvec{\vartheta }^{\left( r\right) }\right) }{\partial \varvec{\varSigma }_j^{-1}}= & {} \displaystyle \frac{1}{2}\displaystyle \sum _{i=1}^n z_{ij}^{\left( r\right) } \left\{ \varvec{\varSigma }_j+ \left( u_{ij}^{\left( r\right) }+\displaystyle \frac{1-u_{ij}^{\left( r\right) }}{ \eta _j^{\left( r\right) }}\right) \right. \nonumber \\&\left. \left[ {\varvec{y}}_i-\varvec{\mu }\left( {\varvec{x}}_i;{\varvec{B}}_j^{\left( r+1\right) }\right) \right] \right. \nonumber \\&\left. \left[ {\varvec{y}}_i-\varvec{\mu }\left( {\varvec{x}}_i;{\varvec{B}}_j^{\left( r+1\right) }\right) \right] '\right\} , \qquad j=1,\ldots ,k. \end{aligned}$$

(18)

Equating (18) to the null matrix yields

$$\begin{aligned} \varvec{\varSigma }_j^{\left( r+1\right) }= & {} \frac{1}{n_j^{\left( r\right) }}\sum _{i=1}^nz_{ij}^{\left( r\right) } \left( u_{ij}^{\left( r\right) }+\frac{1-u_{ij}^{\left( r\right) }}{\eta _j^{\left( r \right) }}\right) \\&\left[ {\varvec{y}}_i-\displaystyle \varvec{\mu }\left( {\varvec{x}}_i;{\varvec{B}}_j^{\left( r+1\right) }\right) \right] \left[ {\varvec{y}}_i-\displaystyle \varvec{\mu }\left( {\varvec{x}}_i;{\varvec{B}}_j^{\left( r+1\right) }\right) \right] ', \qquad j=1,\ldots ,k. \end{aligned}$$