Robust Bayesian seemingly unrelated regression model

Abstract

A robust Bayesian model for seemingly unrelated regression is proposed. By using heavy-tailed distributions for the likelihood, robustness in the response variable is attained. In addition, this robust procedure is combined with a diagnostic approach to identify observations that are far from the bulk of the data in the multivariate space spanned by all variables. The most distant observations are downweighted to reduce the effect of leverage points. The resulting robust Bayesian model can be interpreted as a heteroscedastic seemingly unrelated regression model. Robust Bayesian estimates are obtained by a Markov Chain Monte Carlo approach. Complications by using a heavy-tailed error distribution are resolved efficiently by representing these distributions as a scale mixture of normal distributions. Monte Carlo simulation experiments confirm that the proposed model outperforms its traditional Bayesian counterpart when the data are contaminated in the response and/or the input variables. The method is demonstrated on a real dataset.

Appendix

Theorem 1

We follow the lines of the proof of Theorem 2 in Peña et al. (2009). This means that we have to show that the weighted observations $z_i^\star$ are bounded which implies that the Kullback–Leibler divergence is bounded as well. Let Z denote the original dataset. After rearrangement of the observations, Z can be split into $Z=\left( {\begin{matrix} Z_1 \\ Z_2 \end{matrix}}\right)$. $Z_1$ contains the $\lceil (1-\alpha ) n \rceil$ observations that remain uncontaminated in $Z_\alpha$, while $Z_2$ contains the remaining $[\alpha n]$ observations that are replaced by outliers in $Z_\alpha$. The contaminated data $Z_\alpha$ can similarly be split into $Z_\alpha =\left( {\begin{matrix} Z_1 \\ Z_{2,\alpha } \end{matrix}}\right)$. Moreover, denote $Z_\alpha ^\star = W\,Z_\alpha$ with $W=\text {diag}(\omega _{1},\ldots ,\omega _{N})$ and $\omega _{i}=\omega (d_i)$ with $d_i=d_i(Z_\alpha )$ given by (7).

We first show that the constant a which is the $(1-\alpha )$ quantile of the distances $d_{i}$ is bounded. Since $\alpha <1/2$ and the coordinate-wise median T and the scatter matrix C both have breakdown point 1/2 under the conditions of the theorem, there exist bounds $B_T(Z_1)$ and $B_l(Z_1)$ such that $\Vert T(Z_\alpha )\Vert \le B_T(Z_1)\Vert <\infty$ and $0<B_l(Z_1) \le \lambda _{\min }(C(Z_\alpha ))\le \lambda _{\max }(C(Z_\alpha )) \le B_u(Z_1) < \infty$ where $\lambda _{\min }(C(Z_\alpha ))$ and $\lambda _{\max }(C(Z_\alpha ))$ are the smallest/largest eigenvalue of $C(Z_\alpha )$. For every $z_{\alpha ,i}$ in $Z_\alpha$ it then follows that

$$\begin{aligned} \Vert z_{\alpha ,i}-T(Z_\alpha )\Vert / \sqrt{B_u(Z_1)} \le d_i \le \Vert z_{\alpha ,i}-T(Z_\alpha )\Vert / \sqrt{B_l(Z_1)} \end{aligned}$$

For observations $z_{\alpha ,i}$ in $Z_1$, we have that $z_{\alpha ,i}=z_i$ and the corresponding distances $d_i$ have a finite lower and upper bound that only depends on $Z_1$. This implies immediately that the $(1-\alpha )$ quantile of the distances $d_{i}$ in $Z_\alpha$ is bounded (and positive) whatever the outliers are.

Now, consider $z_i^\star = \omega _{i}z_{\alpha ,i}$. If $d_i > a$, then we have that

$$\begin{aligned} d_i^\star&= \sqrt{(z_i^\star -T)^T C^{-1} (z_i^\star -T)}\\&\le \sqrt{(z_i^\star -\omega _{i}T)^T C^{-1} (z_i^\star -\omega _{i}T)}+(1-\omega _{i})\sqrt{T^TC^{-1}T}\\&\le d_i(1+d_i^2-a^2)^{-1/2} + \Vert T\Vert /\sqrt{B_l(Z_1)}\\&\le \max (1,a)+B_T(Z_1)/\sqrt{B_l(Z_1)}, \end{aligned}$$

which is bounded. If $d_i \le a$, then $\omega _{i}=1$ and thus $d_i^\star = d_i \le a$. Hence, $d_i^\star$ has a finite upper bound for all observations in $Z_\alpha$ and thus also $\Vert z_i^\star \Vert$ has a finite upper bound for the transformed observations.