Gini-PLS Regressions

Abstract

Data contamination and excessive correlations between regressors (multicollinearity) constitute a standard and major problem in econometrics. Two techniques enable solving these problems, in separate ways: the Gini regression for the former, and the PLS (partial least squares) regression for the latter. Gini-PLS regressions are proposed in order to treat extreme values and multicollinearity simultaneously.

Appendices

Appendix A: Proof of Proposition 2

1. Properties (o)–(vi) of PLS1: See Tenenhaus (1998).

2. Properties (o)–(vi) of Gini1-PLS1.

${(o) {t_1} \ \bot \ \cdots \ \bot {t_h}:}$

The proof is by mathematical induction. We follow Tenenhaus (1998, p. 101) for PLS1, except that in our case, $\hat{U}_{(0)}:=R(X)$ [the residuals are issued from the rank vectors Eq. (7)].On the one hand, let us show that ${t_1} \ \bot \ {t_2}$:

$$\begin{aligned} {t_1} \ \bot \ {t_2} \ \Longleftrightarrow \ t_1^{\intercal } t_2 = t_1^{\intercal } \underbrace{\hat{U}_{(1)}w_{2}}_{t_2} = 0 , \end{aligned}$$

since $t_1^{\intercal } \hat{U}_{(1)} = \mathbf {0}$, where $\mathbf {0}$ is the null (row) vector of size p. Suppose the following claim is true:

$$\begin{aligned} \mathbf{[h] :} {t_1} \ \bot \ {t_2} \ \bot \cdots \ \bot \ {t_h}. \end{aligned}$$

We have to show that [h+1] is true, i.e., $t_{h+1}$ is orthogonal to all components $t_1,\ldots ,t_{h}$. The relation [h] implies that $t_{h}^{\intercal }\hat{U}_{(h)} = \mathbf {0}$, hence

$$\begin{aligned} t_{h}^{\intercal } t_{h+1} = t_{h}^{\intercal }\hat{U}_{(h)} w_{(h+1)} =0. \end{aligned}$$

According to Steps 2-h, the partial regressions imply, for all $j=1,\ldots ,p$,

$$\begin{aligned} R(x_j) = \hat{\beta }_{1j}t_1 + \hat{u}_{(1)j} = \hat{\beta }_{1j}t_1 + \hat{\beta }_{2j}t_2 + \hat{u}_{(2)j} = \cdots = \sum _{r=1}^{h}\hat{\beta }_{rj}t_r + \hat{u}_{(h-1)j}. \end{aligned}$$

(11)

The relation (11) provides $\hat{U}_{(h)} = \hat{U}_{(h-1)} - t_h\hat{\beta }_{(h)}^{\intercal }$, where $t_h\hat{\beta }_{(h)}^{\intercal }$ is the $n\times p$ matrix containing $\hat{\beta }_{hj}t_h$ in columns, for all $j=1,\ldots ,p$. Since [h] implies that $t_{h-1}^{\intercal }\hat{U}_{(h-1)}=\mathbf {0}$ and that $t_{h-1}^{\intercal } t_{h} = 0$, we get

$$\begin{aligned} t_{h-1}^{\intercal } t_{h+1}&= t_{h-1}^{\intercal }\hat{U}_{(h)} w_{(h+1)} \\&= t_{h-1}^{\intercal }\left( \hat{U}_{(h-1)} - t_h\hat{\beta }_{(h)}^{\intercal } \right) w_{(h+1)} \\&=\left( t_{h-1}^{\intercal }\hat{U}_{(h-1)} - t_{h-1}^{\intercal }t_h\hat{\beta }_{(h)}^{\intercal }\right) w_{(h+1)} = 0 \ . \end{aligned}$$

Using [h], we find

$$\begin{aligned} t_{h-2}^{\intercal } t_{h+1}&= t_{h-2}^{\intercal }\left( \hat{U}_{(h-1)} - t_h\hat{\beta }_{(h)}^{\intercal } \right) w_{(h+1)} \\&= t_{h-2}^{\intercal }\left( \hat{U}_{(h-2)} - t_{h-1}\hat{\beta }_{(h-1)}^{\intercal } - t_h\hat{\beta }_{(h)}^{\intercal }\right) w_{(h+1)} \\&= \left( t_{h-2}^{\intercal }\hat{U}_{(h-2)} - t_{h-2}^{\intercal }t_{h-1}\hat{\beta }_{(h-1)}^{\intercal } - t_{h-2}^{\intercal }t_h\hat{\beta }_{(h)}^{\intercal } \right) w_{(h+1)} = 0 \ . \end{aligned}$$

Finally, [h] yields

$$\begin{aligned} t_{1}^{\intercal } t_{h+1}&= t_{1}^{\intercal }\left( \hat{U}_{(h-1)} - t_h\hat{\beta }_{(h)}^{\intercal } \right) w_{(h+1)} \\&= \left( t_{1}^{\intercal }\hat{U}_{1} - t_{1}^{\intercal }\sum _{r=2}^{h}t_{r}\hat{\beta }_{(r)}^{\intercal } \right) w_{(h+1)} = 0 \ . \end{aligned}$$

(i) ${ w_\ell ^\intercal \hat{\beta }_{\ell } = 1, \ \forall \ell \in \{2,\ldots ,h\}:}$

Let $\hat{\beta }_h$ be the column vector whose elements are $\hat{\beta }_{hj}$ for all $j=1,\ldots ,p$. The components $t_h$ are given by $w_h^{\intercal }\hat{U}_{(h-1)}^{\intercal }=t^{\intercal }_h$, and so

$$\begin{aligned} w_{(h)}^{\intercal } \hat{\beta }_{h} = w_h^{\intercal }\frac{\hat{U}_{(h-1)}^{\intercal }t_h}{t_h^{\intercal }t_h}. \end{aligned}$$

(12)

For $h=1$, we have $w_1^{\intercal }X^{\intercal }=t^{\intercal }_1$, and so $w_{1}^{\intercal } \hat{\beta }_{1}=w_1^{\intercal }\frac{R(X)^{\intercal } t_1}{t_1^{\intercal }t_1}\ne 1$ if $R(X)\ne X$. For $h > 1$, we get $w_h^{\intercal }\hat{U}^{\intercal }_{(h)}=t^{\intercal }_h$. From expression (11), we have

$$\begin{aligned} \hat{\beta }_{h} = \frac{\hat{U}^{\intercal }_{(h)} t_h}{t_h^{\intercal }t_h}, \end{aligned}$$

(13)

and so $w_{h}^{\intercal } \hat{\beta }_{h} = w_h^{\intercal }\frac{\hat{U}^{\intercal }_{(h)} t_h}{t_h^{\intercal }t_h} =\frac{t_h^{\intercal } t_h}{t_h^{\intercal }t_h} = 1$.

(ii) ${ w_h^{\intercal } \hat{U}^{\intercal }_{(\ell )} = \mathbf {0}, \ \forall \ell \geqslant h > 1:}$

Relation (11), $R(x_j)=\hat{\beta }_{1j}t_1 + \hat{\beta }_{2j}t_2 + \cdots + \hat{u}_{(h-1)j}$, yields

$$\begin{aligned} \hat{U}_{(h-1)} - t_h\hat{\beta }_{h}^{\intercal } = \hat{U}_{(h)}. \end{aligned}$$

(14)

For $h=\ell =1$, we get $R(X) \equiv \hat{U}_{(0)}=t_1\hat{\beta }_{1}^{\intercal }+\hat{U}_{(1)}$, and so

$$\begin{aligned} w_1^{\intercal } \hat{U}^{\intercal }_{(1)} = w_1^{\intercal }\left( R(X)^{\intercal }- \hat{\beta }_1 t_1^\intercal \right) = w_1^{\intercal }\left( \hat{\beta }_1 t_1^\intercal +\hat{U}_{(1)}^\intercal - \hat{\beta }_1 t_1^\intercal \right) = w_1^{\intercal }\hat{U}_{(1)}^\intercal \ne \mathbf {0}. \end{aligned}$$

For $h=\ell >1$, using (i), we deduce from (14) that

$$\begin{aligned} w_h^{\intercal } \hat{U}_h^{\intercal }&= w_h^{\intercal }\left( \hat{U}^{\intercal }_{(h-1)} - \hat{\beta }_h t_h^\intercal \right) \\&= w_h^{\intercal }\hat{U}^{\intercal }_{(h-1)} - w_h^{\intercal }\hat{\beta }_h t_h^\intercal \\&= t_h^\intercal - t_h^\intercal = \mathbf {0}. \end{aligned}$$

For all $\ell> h > 1$, expressions (i) and (13) yield

$$\begin{aligned} w_h^{\intercal } \hat{U}_{(\ell +1)}^{\intercal }&= w_h^{\intercal }\left( \hat{U}_{(\ell )}- t_{\ell +1} \hat{\beta }_{\ell +1}^\intercal \right) ^{\intercal } \\&= w_h^{\intercal } \hat{U}_{(\ell )}^{\intercal } - w_h^{\intercal } \frac{\hat{U}_{(\ell )}^{\intercal }t_{\ell +1}}{t_{\ell +1}^{\intercal }t_{\ell +1}}t_{\ell +1}^{\intercal } \\&= w_h^{\intercal } \hat{U}_{(\ell )}^{\intercal } - w_h^{\intercal } \hat{U}_{(\ell )}^{\intercal }(t_{\ell +1}t_{\ell +1}^{\intercal })^{-1}t_{\ell +1} t_{\ell +1}^{\intercal } \\&=\mathbf {0}. \end{aligned}$$

(iii) ${w_h^{\intercal } \hat{\beta }_\ell = 0, \ \forall \ell> h > 1:}$

Due to relations (i) and (13), we get

$$\begin{aligned} w_h^{\intercal } \hat{\beta }_\ell = w_h^{\intercal } \frac{\hat{U}_{(\ell )}^{\intercal } t_\ell }{t_\ell ^ {\intercal } t_\ell }. \end{aligned}$$

For $\ell> h > 1$, relation (ii) provides the result.

(iv) ${ w_h^{\intercal } w_\ell = 0, \ \forall \ell> h > 1:}$

Relation (ii) yields $w_h^{\intercal }\hat{U}_{(\ell )}^{\intercal }= \mathbf {0}$, for all $\ell> h > 1$. Thus

$$\begin{aligned} w_h^{\intercal } w_\ell = w_h^{\intercal } \frac{\hat{U}_{(\ell -1)} \hat{\varepsilon }_{\ell -1}}{\left( \sum _{j=1}^p \text {cog}^2 \left( \hat{u}_{(\ell -1)j}^{\intercal }, \hat{\varepsilon }_{\ell -1} \right) \right) ^{\frac{1}{2}}} = w_h^{\intercal } \frac{\hat{U}_{(\ell -1)} \hat{\varepsilon }_{\ell -1}}{\left\| \hat{U}_{(\ell -1)} \hat{\varepsilon }_{\ell -1} \right\| } =0. \end{aligned}$$

(v) ${ t_h^{\intercal }\hat{U}_\ell = \mathbf {0}, \ \forall \ell \geqslant h \geqslant 1:}$

Equations (14) and (11) imply

$$\begin{aligned} t_h^{\intercal } \hat{U}_{(\ell )}&= t_h^{\intercal } \left( \hat{U}_{(\ell -1)} - t_\ell \hat{\beta }_\ell ^{\intercal } \right) \\&= t_h^{\intercal } \left( R(X) - \sum _{\ell =1}^h t_\ell \hat{\beta }_\ell ^{\intercal } \right) \\&= t_h^{\intercal } \hat{U}_{(h)} = \mathbf {0}. \end{aligned}$$

(vi) ${ \hat{U}_{(h)} \ne \hat{U}_{(0)} \prod _{\ell =1}^{h} \left( \mathbb {I} - w_\ell \beta _\ell ^{\intercal }\right) , \ \forall h \geqslant 1:}$

Let $h = 1$, since $\hat{U}_{(0)}\equiv R(X)$, by Eq. (14) we have

$$\begin{aligned} \hat{U}_{(1)}&= R(X) - t_1 \hat{\beta }_1^{\intercal } \\&= R(X) - X w_1 \beta _1^{\intercal } \\&\ne R(X)\left( \mathbb {I} - w_1 \beta _1^{\intercal }\right) , \ \text {if} \ X\ne R(X) \\&\ne R(X) \prod _{\ell =1}^{h+1} \left( \mathbb {I} - w_\ell \beta _\ell ^{\intercal }\right) . \end{aligned}$$

3. Properties (o)–(vi) of Gini2-PLS1.

All properties (o)–(v) are obtained in the same manner as in the Gini1-PLS1 case. As to property (vi), let $h = 1$, since $\hat{U}_{(0)}\equiv X$:

$$\begin{aligned} \hat{U}_{(1)}&= R(X) - t_1 \hat{\beta }_1^{\intercal } \\&= X - X w_1 \beta _1^{\intercal } \\&= X\left( \mathbb {I} - w_1 \beta _1^{\intercal }\right) \\&= X \prod _{\ell =1}^{h+1} \left( \mathbb {I} - w_\ell \beta _\ell ^{\intercal }\right) . \end{aligned}$$

Appendix B

See Tables 13, 14, 15 and 16.

Table 13 Descriptive statistics of the database

Table 14 Database of cars data

Table 15 Detecting outliers (before contamination)

Table 16 Predictions with $t_1$ and $t_2$ (before contamination)