Multivariate cluster weighted models using skewed distributions

Abstract

Much work has been done in the area of the cluster weighted model (CWM), which extends the finite mixture of regression model to include modelling of the covariates. Although many types of distributions have been considered for both the response(s) and covariates, to our knowledge skewed distributions have not yet been considered in this paradigm. Herein, a family of 24 novel CWMs is considered which allows both the responses and covariates to be modelled using one of four skewed distributions (the generalized hyberbolic and three of its skewed special cases, i.e., the skew-t, the variance-gamma and the normal-inverse Gaussian distributions) or the normal distribution. Parameter estimation is performed using the expectation-maximization algorithm and both simulated and real data are used for illustration.

Appendices

Appendix

Technical details on the ST distribution

In the fashion of Kim and Browne (2019), it is possible to show that the pdf in (12) can be obtained from the pdf in (8), by forcing $\lambda$ and $\omega$ to be a convenient function of $\nu$, by letting $\varvec{\Sigma }$ and $\varvec{\alpha }$ to become large in a controlled way, and by letting $\omega$ to become small in a controlled way. Specifically, let

$$\begin{aligned} {\varvec{\theta }}=\left( {\varvec{\mu }},\gamma ^{-1}\varvec{\alpha },\gamma ^{-1}\varvec{\Sigma },-\frac{\nu }{2},\nu \gamma \right) , \end{aligned}$$

where $\gamma >0$ is a scaling factor. By substituting these parameter values into (8) we obtain

$$\begin{aligned} f_{\text {GH}}(\varvec{v}; {\varvec{\theta }})=&\, \frac{\exp \left[ (\varvec{v}-{\varvec{\mu }})'\varvec{\Sigma }^{-1}\varvec{\alpha }\right] }{(2\pi )^{\frac{d}{2}}| \gamma ^{-1} \varvec{\Sigma }|^{\frac{1}{2}}K_{-\frac{\nu }{2}}(\nu \gamma )} \left[ \frac{\gamma \delta (\varvec{v};{\varvec{\mu }},\varvec{\Sigma })+\nu \gamma }{\gamma ^{-1}\rho (\varvec{\alpha },\varvec{\Sigma })+\nu \gamma }\right] ^{-\frac{\nu +d}{4}}\\&\times K_{-\frac{\nu +d}{2}}\left( \sqrt{\left[ \gamma ^{-1}\rho (\varvec{\alpha },\varvec{\Sigma })+\nu \gamma \right] \left[ \gamma \delta (\varvec{v};{\varvec{\mu }},\varvec{\Sigma })+\nu \gamma \right] }\right) , \end{aligned}$$

which after some manipulation becomes

$$\begin{aligned} f_{\text {GH}}(\varvec{v}; {\varvec{\theta }})=&\, \frac{\gamma ^{-\frac{\nu }{2}}}{K_{-\frac{\nu }{2}}(\nu \gamma )} \frac{\exp \left[ (\varvec{v}-{\varvec{\mu }})'\varvec{\Sigma }^{-1}\varvec{\alpha }\right] }{(2\pi )^{\frac{d}{2}} |\varvec{\Sigma }|^{\frac{1}{2}}} \left[ \frac{\delta (\varvec{v};{\varvec{\mu }},\varvec{\Sigma })+\nu }{\rho (\varvec{\alpha },\varvec{\Sigma })+\nu \gamma ^2}\right] ^{-\frac{\nu +d}{4}}\\&\times K_{-\frac{\nu +d}{2}}\left( \sqrt{\left[ \rho (\varvec{\alpha },\varvec{\Sigma })+\nu \gamma ^2\right] \left[ \delta (\varvec{v};{\varvec{\mu }},\varvec{\Sigma })+\nu \right] }\right) . \end{aligned}$$

Now, letting $\gamma \rightarrow 0$ and by using the following asymptotic relation

$$\begin{aligned} K_{\lambda }\left( x\right) \sim \Gamma \left( -\lambda \right) 2^{-\lambda -1}x^{\lambda },\quad \text {for } x\rightarrow 0 \text { and } \lambda < 0, \end{aligned}$$

we obtain

$$\begin{aligned}&\frac{2\left( \frac{\nu }{2}\right) ^{\frac{\nu }{2}}\exp \left[ (\varvec{v}-{\varvec{\mu }})'\varvec{\Sigma }^{-1}\varvec{\alpha }\right] }{(2\pi )^{\frac{d}{2}}| \varvec{\Sigma }|^{\frac{1}{2}}\Gamma (\frac{\nu }{2})} \left( \frac{\delta (\varvec{v};{\varvec{\mu }},\varvec{\Sigma })+\nu }{\rho (\varvec{\alpha },\varvec{\Sigma })}\right) ^{-\frac{\nu +d}{4}} \\&\qquad \qquad \qquad \qquad \times K_{-\frac{\nu +d}{2}}\left( \sqrt{\rho (\varvec{\alpha },\varvec{\Sigma })\left[ \delta (\varvec{v};{\varvec{\mu }},\varvec{\Sigma })+\nu \right] }\right) , \end{aligned}$$

which is the density reported in (12).

Parameter estimation

Let $\left( \varvec{x}_{1}',\varvec{y}_{1}'\right) ',\ldots ,\left( \varvec{x}_{N}',\varvec{y}_{N}'\right) '$ be a random sample of N independent observations from (15). In the context of the EM algorithm, the random sample is considered incomplete. Specifically, we have two sources of incompleteness. The first source arises from the fact that, for each observation, we do not know its component membership; to govern this source, we use an indicator vector $\varvec{z}_i=\left( z_{i1},\ldots ,z_{iG}\right)$, where $z_{ig}=1$ if observation i is in group g, and $z_{ig}=0$ otherwise. The second source arises if $f\left( \varvec{y}|\varvec{x};{\varvec{\theta }}_{\varvec{Y}|g}\right)$ or $f\left( \varvec{x};{\varvec{\theta }}_{\varvec{X}|g}\right)$ are skewed; to govern this source, we need the latent variables $W_{\varvec{Y}|g}$ and $W_{\varvec{X}|g}$ introduced in (17).

Based on this source of incompleteness, we can write the complete-data log-likelihood in the following way

$$\begin{aligned} l({\varvec{\vartheta }})=l_1(\varvec{\pi })+l_2({\varvec{\theta }}_{\varvec{X}})+l_3({\varvec{\theta }}_{\varvec{Y}}), \end{aligned}$$

(22)

where $\varvec{\pi }=(\pi _1,\ldots ,\pi _G)'$, and

$$\begin{aligned} l_1(\varvec{\pi })=\sum _{g=1}^G\sum _{i=1}^Nz_{ig}\log \left( \pi _g\right) . \end{aligned}$$

If $\varvec{X}$ in component g, $g=1,\ldots ,G$, follows one of the four skewed distributions,

$$\begin{aligned} l_2({\varvec{\theta }}_{\varvec{X}}) =&\sum _{g=1}^G\sum _{i=1}^N z_{ig} \log \left[ h(w_{ig\varvec{X}};{\varvec{\phi }}_{W_{\varvec{X}}|g})\right] + C_{\varvec{X}} \\&\quad -\frac{1}{2}\sum _{g=1}^G\sum _{i=1}^N z_{ig}\big [\log (|\varvec{\Sigma }_{\varvec{X}|g}|)+w_{ig{\varvec{X}}}\varvec{\alpha }_{\varvec{X}|g}'\varvec{\Sigma }_{\varvec{X}|g}^{-1}\varvec{\alpha }_{\varvec{X}|g} \\&\quad + \frac{1}{w_{ig\varvec{X}}}(\varvec{x}_i-{\varvec{\mu }}_{\varvec{X}|g})'\varvec{\Sigma }_{\varvec{X}|g}^{-1}(\varvec{x}_i-{\varvec{\mu }}_{\varvec{X}|g}) \\&\quad -(\varvec{x}_i-{\varvec{\mu }}_{\varvec{X}|g})'\varvec{\Sigma }_{\varvec{X}|g}^{-1}\varvec{\alpha }_{\varvec{X}|g}-\varvec{\alpha }_{\varvec{X}|g}'\varvec{\Sigma }_{\varvec{X}|g}^{-1}(\varvec{x}_i-{\varvec{\mu }}_{\varvec{X}|g})\big ], \end{aligned}$$

where $h(w_{ig\varvec{X}};{\varvec{\phi }}_{W_{\varvec{X}}|g})$ is the appropriate pdf for $W_{ig\varvec{X}}$ discussed in Sect. 2, with parameters notated as ${\varvec{\phi }}_{W_{\varvec{X}}|g}$, while $C_{\varvec{X}}$ is constant with respect to the parameters. On the other hand, if $\varvec{X}$ in component g, $g=1,\ldots ,G$, is normally distributed then

$$\begin{aligned} l_2({\varvec{\theta }}_{\varvec{X}}) = C_{\varvec{X}}-\frac{1}{2}\sum _{g=1}^G\sum _{i=1}^Nz_{ig}[\log (|\varvec{\Sigma }_{\varvec{X}|g}|)+(\varvec{x}_i-{\varvec{\mu }}_{\varvec{X}|g})'\varvec{\Sigma }_{\varvec{X}|g}^{-1}(\varvec{x}_i-{\varvec{\mu }}_{\varvec{X}|g})]. \end{aligned}$$

Similarly, if $\varvec{Y}|\varvec{x}$ in component g, $g=1,\ldots ,G$, is distributed according to one of the four skewed distributions,

$$\begin{aligned} l_3({\varvec{\theta }}_{\varvec{Y}}) =&\sum _{g=1}^G\sum _{i=1}^N z_{ig} \log \left[ h(w_{ig\varvec{Y}};{\varvec{\phi }}_{W_{\varvec{Y}}|g})\right] +C_{\varvec{Y}}\\&\quad -\frac{1}{2}\sum _{g=1}^G\sum _{i=1}^N z_{ig}\big [\log (|\varvec{\Sigma }_{\varvec{Y}|g}|)+w_{ig\varvec{Y}}\varvec{\alpha }_{\varvec{Y}|g}'\varvec{\Sigma }_{\varvec{Y}|g}^{-1}\varvec{\alpha }_{\varvec{Y}|g} \\&\quad + \frac{1}{w_{ig\varvec{Y}}}(\varvec{y}_i-{\varvec{B}_g'\varvec{x}_i^*})'\varvec{\Sigma }_{\varvec{Y}|g}^{-1}(\varvec{y}_i-{\varvec{B}_g'\varvec{x}_i^*})\\&\quad -(\varvec{y}_i-{\varvec{B}_g'\varvec{x}_i^*})'\varvec{\Sigma }_{\varvec{Y}|g}^{-1}\varvec{\alpha }_{\varvec{Y}|g}-\varvec{\alpha }_{\varvec{Y}|g}'\varvec{\Sigma }_{\varvec{Y}|g}^{-1}(\varvec{y}_i-{\varvec{B}_g'\varvec{x}_i^*})\big ], \end{aligned}$$

where $h(w_{ig\varvec{Y}};{\varvec{\phi }}_{W_{\varvec{Y}}|g})$ is the appropriate pdf for $W_{ig\varvec{Y}}$ discussed in Sect. 2, with parameters notationally compacted as ${\varvec{\phi }}_{W_{\varvec{Y}}|g}$, while $C_{\varvec{Y}}$ is constant with respect to the parameters. Conversely, if $\varvec{Y}|\varvec{x}$ in component g, $g=1,\ldots ,G$, is normally distributed,

$$\begin{aligned} l_3({\varvec{\theta }}_{\varvec{Y}})=C_{\varvec{Y}}-\frac{1}{2}\sum _{g=1}^G\sum _{i=1}^Nz_{ig}[\log (|\varvec{\Sigma }_{\varvec{Y}|g}|)+(\varvec{x}_i-{\varvec{\mu }}_{\varvec{Y}|g})'\varvec{\Sigma }_{\varvec{Y}|g}^{-1}(\varvec{x}_i-{\varvec{\mu }}_{\varvec{Y}|g})]. \end{aligned}$$

After initialization, the EM algorithm proceeds iterating the following two steps until convergence.

E-Step. The E-step requires the calculation of the conditional expectation of (22). Thus, we first need to calculate

$$\begin{aligned} \hat{z}_{ig}=\frac{\hat{\pi }_g f\left( \varvec{y}_i|\varvec{x}_i;\hat{{\varvec{\theta }}}_{\varvec{Y}|g}\right) f\left( \varvec{x}_i;\hat{{\varvec{\theta }}}_{\varvec{X}|g}\right) }{\displaystyle \sum _{h=1}^G\hat{\pi }_h f\left( \varvec{y}_i|\varvec{x}_i;\hat{{\varvec{\theta }}}_{\varvec{Y}|h}\right) f\left( \varvec{x}_i;\hat{{\varvec{\theta }}}_{\varvec{X}|h}\right) }, \end{aligned}$$

which corresponds to the posterior probability that the unlabeled observation $\left( \varvec{X}_{i}',\varvec{Y}_{i}'\right) '$ belongs to the gth component of the CWM. In addition, if the distribution of $\varvec{X}$ in component g, $g=1,\ldots ,G$, is skewed, the following values need to be updated:

$$\begin{aligned} \begin{aligned} \hat{l}_{ig\varvec{X}}&{:}{=}\, \mathbb {E}[W_{ig\varvec{X}}|z_{ig}=1,\varvec{x}_i,\hat{{\varvec{\phi }}}_{W_{\varvec{X}}|g}],\\ \hat{m}_{ig\varvec{X}}&{:}{=}\, \mathbb {E}[1/W_{ig\varvec{X}}|z_{ig}=1,\varvec{x}_i,\hat{{\varvec{\phi }}}_{W_{\varvec{X}}|g}],\\ \hat{n}_{ig\varvec{X}}&{:}{=}\, \mathbb {E}[\log (W_{ig\varvec{X}})|z_{ig}=1,\varvec{x}_i,\hat{{\varvec{\phi }}}_{W_{\varvec{X}}|g}].\\ \end{aligned} \end{aligned}$$

If the distribution of $\varvec{Y}|\varvec{x}$ in component g, $g=1,\ldots ,G$, is skewed, then the following values are also updated:

$$\begin{aligned} \begin{aligned} \hat{l}_{ig\varvec{Y}}&{:}{=}\, \mathbb {E}[W_{ig\varvec{Y}}|z_{ig}=1,\varvec{y}_i,\varvec{x}_i,\hat{{\varvec{\phi }}}_{W_{\varvec{Y}}|g}],\\ \hat{m}_{ig\varvec{Y}}&{:}{=}\, \mathbb {E}[1/W_{ig\varvec{Y}}|z_{ig}=1,\varvec{y}_i,\varvec{x}_i,\hat{{\varvec{\phi }}}_{W_{\varvec{Y}}|g}],\\ \hat{n}_{ig\varvec{Y}}&{:}{=}\, \mathbb {E}[\log (W_{ig\varvec{Y}})|z_{ig}=1,\varvec{y}_i,\varvec{x}_i,\hat{{\varvec{\phi }}}_{W_{\varvec{Y}}|g}].\\ \end{aligned} \end{aligned}$$

These updates depend on which of the skewed distributions is considered. However, as shown in Sect. 2.2, the conditional latent variables are all GIG distributed. Therefore, all of the required expectations can be calculated using (2)–(4).

M-Step. The M-step involves the maximization of the conditional expectation of the complete-data log-likelihood, allowing all parameters to be updated. Specifically, the update for $\pi _g$ is

$$\begin{aligned} \hat{\pi }_g = \frac{1}{N}\sum _{i=1}^N\hat{z}_{ig}. \end{aligned}$$

The parameters related to the distribution of $\varvec{X}$ in component g, $g=1,\ldots ,G$, are updated as follows. For skewed distributions, we have the following updates for ${\varvec{\mu }}_{\varvec{X}|g}$ and $\varvec{\alpha }_{\varvec{X}|g}$:

$$\begin{aligned} \hat{{\varvec{\mu }}}_{\varvec{X}|g}=\frac{\displaystyle \sum _{i=1}^N\hat{z}_{ig}\varvec{x}_i\left( \overline{l}_{\varvec{X}|g} \hat{m}_{ig\varvec{X}}-1\right) }{\displaystyle \sum _{i=1}^N\hat{z}_{ig}\overline{l}_{\varvec{X}|g} \hat{m}_{ig\varvec{X}}-T_g}\quad \text {and}\quad \hat{{\varvec{\alpha }}}_{\varvec{X}|g}=\frac{\displaystyle \sum _{i=1}^N\hat{z}_{ig}\varvec{x}_i\left( \overline{m}_{\varvec{X}|g}-\hat{m}_{ig\varvec{X}}\right) }{\displaystyle \sum _{i=1}^N\hat{z}_{ig}\overline{l}_{\varvec{X}|g} \hat{m}_{ig\varvec{X}}-T_g}, \end{aligned}$$

where $T_g=\sum _{i=1}^N\hat{z}_{ig}$, $\overline{l}_{\varvec{X}|g}=(1/T_g)\sum _{i=1}^N\hat{z}_{ig}\hat{l}_{ig\varvec{X}}$ and $\overline{m}_{\varvec{X}|g}=(1/T_g)\sum _{i=1}^N\hat{z}_{ig}\hat{m}_{ig\varvec{X}}$. The update for $\varvec{\Sigma }_{\varvec{X}|g}$ is

$$\begin{aligned} \hat{\varvec{\Sigma }}_{\varvec{X}|g} =&\frac{1}{T_g}\sum _{i=1}^N\hat{z}_{ig}\big [\hat{m}_{ig\varvec{X}}(\varvec{x}_i-\hat{{\varvec{\mu }}}_{\varvec{X}|g})(\varvec{x}_i-\hat{{\varvec{\mu }}}_{\varvec{X}|g})'\\&-(\varvec{x}_i-\hat{{\varvec{\mu }}}_{\varvec{X}|g})\hat{\varvec{\alpha }}_{\varvec{X}|g}'-\hat{\varvec{\alpha }}_{\varvec{X}|g}(\varvec{x}_i-\hat{{\varvec{\mu }}}_{\varvec{X}|g})' \\&+ \hat{l}_{ig\varvec{X}}\hat{\varvec{\alpha }}_{\varvec{X}|g}\hat{\varvec{\alpha }}_{\varvec{X}|g}'\big ]. \end{aligned}$$

Instead, for the normal distribution, we have

$$\begin{aligned} \hat{{\varvec{\mu }}}_{\varvec{X}|g}=\frac{1}{T_g}\sum _{g=1}^G\hat{z}_{ig}\varvec{x}_i, \qquad \hat{\varvec{\Sigma }}_{\varvec{X}|g}=\frac{1}{T_g}\sum _{g=1}^G\hat{z}_{ig}(\varvec{x}_i-\hat{{\varvec{\mu }}}_{\varvec{X}|g})(\varvec{x}_i-\hat{{\varvec{\mu }}}_{\varvec{X}|g})'. \end{aligned}$$

The parameters related to the distribution of $\varvec{Y}|\varvec{x}$ in component g, $g=1,\ldots ,G$, are updated as follows. For skewed distributions the updates for $\varvec{B}_g$ and $\varvec{\alpha }_{\varvec{Y}|g}$ are

$$\begin{aligned} \hat{\varvec{B}}_g=\varvec{P}_g^{-1}\varvec{R}_g \quad \text {and}\quad \hat{\varvec{\alpha }}_{\varvec{Y}|g}=\frac{1}{T_g\overline{l}_{\varvec{Y}|g}}\left( \sum _{i=1}^N\hat{z}_{ig}\varvec{y}_i-\varvec{R}_g'\varvec{P}_g^{-1}\sum _{i=1}^N\hat{z}_{ig}\varvec{x}_i^*\right) , \end{aligned}$$

where

$$\begin{aligned} \varvec{P}_g=\sum _{i=1}^N\hat{z}_{ig}\hat{m}_{ig\varvec{Y}}\varvec{x}_i^*{\varvec{x}_i^*}'-\frac{1}{T_g\overline{l}_{\varvec{Y}|g}}\left( \sum _{i=1}^N\hat{z}_{ig}\varvec{x}_i^*\right) \left( \sum _{i=1}^N\hat{z}_{ig}{\varvec{x}_i^*}'\right) \end{aligned}$$

and

$$\begin{aligned} \varvec{R}_g=\sum _{i=1}^N\hat{z}_{ig}\hat{m}_{ig\varvec{Y}}\varvec{x}_i^*{\varvec{y}_i}'-\frac{1}{T_g\overline{l}_{\varvec{Y}|g}}\left( \sum _{i=1}^N\hat{z}_{ig}\varvec{x}_i^*\right) \left( \sum _{i=1}^N\hat{z}_{ig}{\varvec{y}_i}'\right) , \end{aligned}$$

with $\overline{l}_{\varvec{Y}|g}=(1/T_g)\sum _{i=1}^N\hat{z}_{ig}\hat{l}_{ig\varvec{Y}}$. The update for $\varvec{\Sigma }_{\varvec{Y}|g}$ is

$$\begin{aligned} \hat{\varvec{\Sigma }}_{\varvec{Y}|g}= & {} \frac{1}{T_g}\sum _{i=1}^N\hat{z}_{ig}\Big [\hat{m}_{ig\varvec{Y}}\left( \varvec{y}-\hat{\varvec{B}}_g'\varvec{x}_i^*\right) \left( \varvec{y}-\hat{\varvec{B}}_g'\varvec{x}_i^*\right) '\\&-\left( \varvec{y}-\hat{\varvec{B}}_g'\varvec{x}_i^*\right) \hat{\varvec{\alpha }}_{\varvec{Y}|g}'-\hat{\varvec{\alpha }}_{\varvec{Y}|g}\left( \varvec{y}-\hat{\varvec{B}}_g'\varvec{x}_i^*\right) ' + \hat{l}_{ig\varvec{Y}}\hat{\varvec{\alpha }}_{\varvec{Y}|g}\hat{\varvec{\alpha }}_{\varvec{Y}|g}'\Big ]. \end{aligned}$$

Conversely, in the case of a a multivariate normal distribution, the updates for $\varvec{B}_g$ and $\varvec{\Sigma }_{\varvec{Y}|g}$ are

$$\begin{aligned}&\hat{\varvec{B}}_g=\left( \sum _{i=1}^N\hat{z}_{ig}\varvec{x}_i^*{\varvec{x}_i^*}'\right) ^{-1}\left( \sum _{i=1}^N\hat{z}_{ig}\varvec{x}_i^*{\varvec{y}_i}'\right) \quad \text {and} \\&\hat{\varvec{\Sigma }}_{\varvec{Y}|g}=\frac{1}{T_g}\sum _{i=1}^N\hat{z}_{ig}(\varvec{y}_i-\hat{\varvec{B}}_g'\varvec{x}_i)(\varvec{y}_i-\hat{\varvec{B}}_g'\varvec{x}_i)'. \end{aligned}$$

Finally, if either $\varvec{X}$ or $\varvec{Y}|\varvec{x}$ in component g, $g=1,\ldots ,G$, follows one of the skewed distributions, then there are the additional tailedness and, in the case of the GH distribution, the index parameters that need to be updated. The updates for each distribution are now given.

1.1 Skew-t distribution

In the case of the ST distribution, we need to update the degrees of freedom $\nu _g$. This update cannot be obtained in closed form, and thus needs to be performed numerically. For the covariates the update for $\nu _{\varvec{X}|g}$ is obtained by solving the equation

$$\begin{aligned} \log \left( \frac{\nu _{\varvec{X}|g}}{2}\right) +1-\varphi \left( \frac{\nu _{\varvec{X}|g}}{2}\right) -\frac{1}{T_g}\sum _{i=1}^N\hat{z}_{ig}(\hat{m}_{ig\varvec{X}}+\hat{n}_{ig\varvec{X}})=0, \end{aligned}$$

(23)

where $\varphi (\cdot )$ denotes the digamma function. When the responses are considered, the update for $\nu _{\varvec{Y}|g}$ is obtained via (23), after the replacement of $\nu _{\varvec{X}|g}$, $\hat{m}_{ig\varvec{X}}$ and $\hat{n}_{ig\varvec{X}}$ with $\nu _{\varvec{Y}|g}$, $\hat{m}_{ig\varvec{Y}}$ and $\hat{n}_{ig\varvec{Y}}$, respectively.

1.2 Generalized hyperbolic distribution

For the GH distribution, we would update $\lambda _g$ and $\omega _g$. These updates are derived from Browne and McNicholas (2015), and rely on the log convexity of $K_{s}(t)$ in both s and t (Baricz 2010). For notational purposes in this section, the superscript “prev” is used to distinguish the previous update from the current one. The resulting updates, when $\varvec{X}$ is considered, are

$$\begin{aligned} \hat{\lambda }_{\varvec{X}|g}&=\overline{n}_{\varvec{X}|g}\hat{\lambda }_{\varvec{X}|g}^{\text {prev}}\left[ \left. \frac{\partial }{\partial s}\log (K_{s}(\hat{\omega }_{\varvec{X}|g}^{\text {prev}}))\right| _{s=\hat{\lambda }_{\varvec{X}|g}^{\text {prev}}}\right] ^{-1}, \end{aligned}$$

(24)

$$\begin{aligned} \hat{\omega }_{\varvec{X}|g}&=\hat{\omega }_{\varvec{X}|g}^{\text {prev}}-\left[ \left. \frac{\partial }{\partial s}q(\hat{\lambda }_{\varvec{X}|g},s)\right| _{s=\hat{\omega }_{\varvec{X}|g}^{\text {prev}}}\right] \left[ \left. \frac{\partial ^2}{\partial s^2}q(\hat{\lambda }_{\varvec{X}|g},s)\right| _{s=\hat{\omega }_{\varvec{X}|g}^{\text {prev}}}\right] ^{-1}, \end{aligned}$$

(25)

where the derivative in (24) is calculated numerically,

$$\begin{aligned} q(\lambda _{\varvec{X}|g},\omega _{\varvec{X}|g})=\sum _{i=1}^Nz_{ig}\left[ \log (K_{\lambda _{\varvec{X}|g}}(\omega _{\varvec{X}|g}))-\lambda _{\varvec{X}|g}\overline{n}_{\varvec{X}|g}-\frac{1}{2}\omega _{\varvec{X}|g}\left( \overline{l}_{\varvec{X}|g}+\overline{m}_{\varvec{X}|g}\right) \right] , \end{aligned}$$

and $\overline{n}_{\varvec{X}|g}=({1}/{T_g})\sum _{i=1}^N\hat{z}_{ig}\hat{n}_{ig\varvec{X}}$. When $\varvec{Y}$ is considered, $\lambda _{\varvec{X}|g}$, $\omega _{\varvec{X}|g}$, $\overline{l}_{\varvec{X}|g}$, $\overline{m}_{\varvec{X}|g}$, and $\overline{n}_{\varvec{X}|g}$ are replaced with $\lambda _{\varvec{Y}|g}$, $\omega _{\varvec{Y}|g}$, $\overline{l}_{\varvec{Y}|g}$, $\overline{m}_{\varvec{Y}|g}$, and $\overline{n}_{\varvec{Y}|g}$, respectively, where $\overline{m}_{\varvec{Y}|g}=(1/T_g)\sum _{i=1}^N\hat{z}_{ig}\hat{m}_{ig\varvec{Y}}$ and $\overline{n}_{\varvec{Y}|g}=({1}/{T_g})\sum _{i=1}^N\hat{z}_{ig}\hat{n}_{ig\varvec{Y}}$.

1.3 Variance-gamma distribution

For the VG distribution, the update for $\psi _g$ cannot be obtained in closed form. When $\varvec{X}$ is considered, this update is obtained by solving the equation

$$\begin{aligned} \log \psi _{\varvec{X}|g}+1-\varphi (\psi _{\varvec{X}|g})+\overline{n}_{\varvec{X}|g}-\overline{l}_{\varvec{X}|g}=0. \end{aligned}$$

(26)

Clearly, when $\varvec{Y}$ is considered, $\psi _{\varvec{X}|g}$, $\overline{n}_{\varvec{X}|g}$ and $\overline{l}_{\varvec{X}|g}$ are replaced with $\psi _{\varvec{Y}|g}$, $\overline{n}_{\varvec{Y}|g}$ and $\overline{l}_{\varvec{Y}|g}$, respectively.

1.4 Normal inverse Gaussian distribution

The NIG distribution is the only having a closed form expression for its tailedness parameter. In detail, when we consider the covariates, the update of $\kappa _{\varvec{X}|g}$ is

$$\begin{aligned} \hat{\kappa }_{\varvec{X}|g}=\frac{1}{\overline{l}_{\varvec{X}|g}}. \end{aligned}$$

If the responses are considered, we replace $\kappa _{\varvec{X}|g}$ and $\overline{l}_{\varvec{X}|g}$ with $\kappa _{\varvec{Y}|g}$ and $\overline{l}_{\varvec{Y}|g}$, respectively.

1.5 Initialization of the algorithm

To initialize the EM algorithm, we followed the approach discussed in Dang et al. (2017). Specifically, the $z_{ig}$ are initialized in two different ways: 10 times using a random soft initialization and once with a k-means (hard) initialization. Therefore, for each G, the algorithms are run 11 times until convergence, and the solution producing the highest log-likelihood value is chosen. Notice that, for the k-means initialization, the initial $z_{ig}$ are selected from the best k-means clustering results from 10 random starting values, and it is implemented by using the kmeans() function of the R statistical software (R Core Team 2019).

Proof of Theorem 3.1

Proof

Suppose that

$$\begin{aligned} p_{\text {GH-GH}}\left( \varvec{x},\varvec{y}; \varvec{\vartheta }\right) =p_{\text {GH-GH}}\left( \varvec{x},\varvec{y}; \widetilde{\varvec{\vartheta }}\right) . \end{aligned}$$

(27)

Integrating out $\varvec{y}$ from each side of (27) yields an equality on the marginal distribution of $\varvec{X}$, i.e.,

$$\begin{aligned} \sum _{g=1}^G\pi _gf_{\text {GH}}\left( \varvec{x};{\varvec{\theta }}_{\varvec{X}|g}\right)&= \sum _{j=1}^{\widetilde{G}}\widetilde{\pi }_jf_{\text {GH}}\left( \varvec{x};\widetilde{{\varvec{\theta }}}_{\varvec{X}|j}\right) \nonumber \\ p_{\text {GH}}\left( \varvec{x}; \varvec{\pi },{\varvec{\theta }}_{\varvec{X}}\right)&= p_{\text {GH}}\left( \varvec{x}; \widetilde{\varvec{\pi }},\widetilde{{\varvec{\theta }}}_{\varvec{X}}\right) , \end{aligned}$$

(28)

where ${\varvec{\theta }}_{\varvec{X}}=\left\{ {\varvec{\theta }}_{\varvec{X}|g}; \, g=1,\ldots ,G\right\}$, $\widetilde{{\varvec{\theta }}}_{\varvec{X}}=\left\{ \widetilde{{\varvec{\theta }}}_{\varvec{X}|j}; \, j=1,\ldots ,\widetilde{G}\right\}$, $\varvec{\pi }=\{\pi _g; \, g=1,\ldots ,G \}$ and $\widetilde{\varvec{\pi }}=\{\widetilde{\pi }_j; \, j=1,\ldots ,\widetilde{G}\}$. Dividing the left-hand (right-hand) side of (27) by the left-hand (right-hand) side of (28) leads to

$$\begin{aligned} \sum _{g=1}^{\widetilde{G}} \frac{\pi _gf_{\text {GH}}\left( \varvec{x};{\varvec{\theta }}_{\varvec{X}|g}\right) }{p_{\text {GH}}\left( \varvec{x}; \varvec{\pi },{\varvec{\theta }}_{\varvec{X}}\right) } f_{\text {GH}}\left( \varvec{y}|\varvec{x};{\varvec{\theta }}_{\varvec{Y}|g}\right)&= \sum _{j=1}^{\widetilde{G}} \frac{\widetilde{\pi }_jf_{\text {GH}}\left( \varvec{x};\widetilde{{\varvec{\theta }}}_{\varvec{X}|j}\right) }{p_{\text {GH}}\left( \varvec{x}; \widetilde{\varvec{\pi }}, \widetilde{{\varvec{\theta }}}_{\varvec{X}}\right) } f_{\text {GH}}\left( \varvec{y}|\varvec{x};\widetilde{{\varvec{\theta }}}_{\varvec{Y}|j}\right) \nonumber \\ p_{\text {GH}}\left( \varvec{y}|\varvec{x};\varvec{\vartheta }\right)&= p_{\text {GH}}\left( \varvec{y}|\varvec{x};\widetilde{\varvec{\vartheta }}\right) . \end{aligned}$$

(29)

For each fixed value of $\varvec{x}$, $p_{\text {GH}}\left( \varvec{y}|\varvec{x};\varvec{\vartheta }\right)$ and $p_{\text {GH}}\left( \varvec{y}|\varvec{x};\widetilde{\varvec{\vartheta }}\right)$ are mixtures of $d_{\varvec{Y}}$-variate GH distributions for $\varvec{Y}$ (see Browne and McNicholas 2015).

Now, recall from Sect. 3.1 that the location parameter $\varvec{\mu }_{\varvec{Y}|g}$ of the $d_{\varvec{Y}}$-variate GH distribution of $\varvec{Y}$ in the gth mixture component is related to the covariates $\varvec{X}$, through the regression coefficients $\varvec{B}_g$, by the relation $\varvec{B}'_g \varvec{x}^*$, $g=1, \ldots , G$. Define the set of all covariate points $\varvec{x}$ which can be used to distinct different regression coefficients $\varvec{B}_g$ by different values of $\varvec{B}'_g \varvec{x}^*$, i.e.

$$\begin{aligned} \mathcal {X} := \Bigl \{ \varvec{x}\in IR^{d_{\varvec{X}}}:&\forall g,s \in \{1, \ldots , G \} \text { and } j,t \in \{1, \ldots , \widetilde{G}\},\Bigr . \nonumber \\&\Bigl . \varvec{B}'_g \varvec{x}^*=\varvec{B}'_s\varvec{x}^* \ \Rightarrow \ \varvec{B}_g=\varvec{B}_s, \Bigr . \nonumber \\&\Bigl . \varvec{B}'_g \varvec{x}^*=\widetilde{\varvec{B}}'_j\varvec{x}^* \ \Rightarrow \ \varvec{B}_g=\widetilde{\varvec{B}}_j , \Bigr . \nonumber \\&\Bigl . \widetilde{\varvec{B}}'_j\varvec{x}^*=\widetilde{\varvec{B}}'_t\varvec{x}^* \ \Rightarrow \ \widetilde{\varvec{B}}_j=\widetilde{\varvec{B}}_t\Bigr \} . \end{aligned}$$

Note that $\mathcal {X}$ is complement of a finite union of hyperplanes of $IR^{d_{\varvec{X}}}$. Therefore,

$$\begin{aligned} \int _{\mathcal {X}}p_{\text {GH}}\left( \varvec{x}; \varvec{\pi },{\varvec{\theta }}_{\varvec{X}}\right) d\varvec{x}=1. \end{aligned}$$

For $\varvec{x}\in \mathcal {X}$, all $\left\{ \varvec{B}'_g \varvec{x}^*,\varvec{\Sigma }_{\varvec{Y}|g},\varvec{\alpha }_{\varvec{Y}|g},\lambda _{\varvec{Y}|g},\omega _{\varvec{Y}|g}\right\}$, $g=1,\ldots ,G$, are pairwise distinct because all $\left\{ \varvec{B}_g,\varvec{\Sigma }_{\varvec{Y}|g},\varvec{\alpha }_{\varvec{Y}|g},\lambda _{\varvec{Y}|g},\omega _{\varvec{Y}|g}\right\}$, $g=1,\ldots ,G$, are pairwise distinct for the hypothesis of the theorem. As mentioned above, for each fixed value of $\varvec{x}$, $p_{\text {GH}}\left( \varvec{y}|\varvec{x};\varvec{\vartheta }\right)$ is a mixture of $d_{\varvec{Y}}$-variate GH distributions, which being identifiable (Browne and McNicholas 2015) implies that $G=\widetilde{G}$ and that, for each $g\in \left\{ 1,\ldots ,G\right\}$, there exists a $j\in \left\{ 1,\ldots ,G\right\}$ such that

$$\begin{aligned} \varvec{B}_g=\widetilde{\varvec{B}}_j, \quad \varvec{\Sigma }_{\varvec{Y}|g}=\widetilde{\varvec{\Sigma }}_{\varvec{Y}|j}, \quad \varvec{\alpha }_{\varvec{Y}|g}=\widetilde{\varvec{\alpha }}_{\varvec{Y}|j}, \quad \lambda _{\varvec{Y}|g}=\widetilde{\lambda }_{\varvec{Y}|j}, \quad \omega _{\varvec{Y}|g}=\widetilde{\omega }_{\varvec{Y}|j} \end{aligned}$$

and

$$\begin{aligned} \frac{\pi _gf_{\text {GH}}\left( \varvec{x};{\varvec{\theta }}_{\varvec{X}|g}\right) }{p_{\text {GH}}\left( \varvec{x}; \varvec{\pi },{\varvec{\theta }}_{\varvec{X}}\right) } = \frac{\widetilde{\pi }_jf_{\text {GH}}\left( \varvec{x};\widetilde{{\varvec{\theta }}}_{\varvec{X}|j}\right) }{p_{\text {GH}}\left( \varvec{x}; \widetilde{\varvec{\pi }},\widetilde{{\varvec{\theta }}}_{\varvec{X}}\right) }. \end{aligned}$$

(30)

Now, based on (28), the equality in (30) simplifies to

$$\begin{aligned} \pi _gf_{\text {GH}}\left( \varvec{x};{\varvec{\theta }}_{\varvec{X}|g}\right) =\widetilde{\pi }_jf_{\text {GH}}\left( \varvec{x};\widetilde{{\varvec{\theta }}}_{\varvec{X}|j}\right) ,\quad \forall \ \varvec{x}\in \mathcal {X}. \end{aligned}$$

(31)

Integrating (31) over $\varvec{x}\in \mathcal {X}$ yields $\pi _g=\widetilde{\pi }_j$. Therefore, the condition (31) further simplifies as

$$\begin{aligned} f_{\text {GH}}\left( \varvec{x};{\varvec{\theta }}_{\varvec{X}|g}\right) = f_{\text {GH}}\left( \varvec{x};\widetilde{{\varvec{\theta }}}_{\varvec{X}|j}\right) ,\quad \forall \ \varvec{x}\in \mathcal {X}. \end{aligned}$$

The equalities $\varvec{\mu }_{\varvec{X}|g}=\widetilde{\varvec{\mu }}_{\varvec{X}|j}$, $\varvec{\Sigma }_{\varvec{X}|g}=\widetilde{\varvec{\Sigma }}_{\varvec{X}|j}$, $\varvec{\alpha }_{\varvec{X}|g}=\widetilde{\varvec{\alpha }}_{\varvec{X}|j}$, $\lambda _{\varvec{X}|g}=\widetilde{\lambda }_{\varvec{X}|j}$, and $\omega _{\varvec{X}|g}=\widetilde{\omega }_{\varvec{X}|j}$ simply arise from the identifiability of the $d_{\varvec{X}}$-variate GH distribution, and this completes the proof. $\square$