Random Projections for Large-Scale Regression

Abstract

Fitting linear regression models can be computationally very expensive in large-scale data analysis tasks if the sample size and the number of variables are very large. Random projections are extensively used as a dimension reduction tool in machine learning and statistics. We discuss the applications of random projections in linear regression problems, developed to decrease computational costs, and give an overview of the theoretical guarantees of the generalization error. It can be shown that the combination of random projections with least squares regression leads to similar recovery as ridge regression and principal component regression. We also discuss possible improvements when averaging over multiple random projections, an approach that lends itself easily to parallel implementation.

Appendix

In this section we give proofs of the statements from the section theoretical results. Theorem  1 ([10]) Assume fixed design and Rank(X) ≥ d, then the AMSE  4 can be bounded above by

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\|\mathbf{X}\beta -\mathbf{X}\hat{\beta }_{d}^{\boldsymbol{\phi }}\|_{ 2}^{2}]] \leq \sigma ^{2}d + \frac{\|\mathbf{X}\beta \|_{2}^{2}} {d} +\mathop{ \mathrm{trace}}\nolimits (\mathbf{X}^{{\prime}}\mathbf{X})\frac{\|\beta \|_{2}^{2}} {d}. }$$

(21)

Proof

(Sketch)

$$\displaystyle\begin{array}{rcl} \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\|\mathbf{X}\beta -\mathbf{X}\hat{\beta }_{d}^{\boldsymbol{\phi }}\|_{ 2}^{2}]]& =& \mathbb{E}_{\boldsymbol{\phi }}[\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }(\boldsymbol{\phi }^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}\boldsymbol{\phi })^{-1}\boldsymbol{\phi }^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}\beta \|_{ 2}^{2}] +\sigma ^{2}d {}\\ & \leq & \mathbb{E}_{\boldsymbol{\phi }}[\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }(\boldsymbol{\phi }^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}\boldsymbol{\phi })^{-1}\boldsymbol{\phi }^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}\beta \|_{ 2}^{2}] +\sigma ^{2}d {}\\ & =& \mathbb{E}_{\boldsymbol{\phi }}[\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}\beta \|_{ 2}^{2}] +\sigma ^{2}d. {}\\ \end{array}$$

Finally a rather lengthy but straightforward calculation leads to

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}\beta \|_{ 2}^{2}] = \frac{\|\mathbf{X}\beta \|_{2}^{2}} {d} +\mathop{ \mathrm{trace}}\nolimits (\mathbf{X}^{{\prime}}\mathbf{X})\frac{\|\beta \|_{2}^{2}} {d} }$$

(22)

and thus proving the statement above. □ 

Theorem  2 Assume Rank(X) ≥ d, then the AMSE (4) can be bounded above by

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\|\mathbf{X}\beta -\mathbf{X}\hat{\beta }_{d}^{\boldsymbol{\phi }}\|_{ 2}^{2}]] \leq \sigma ^{2}d +\sum _{ i=1}^{p}\beta _{ i}^{2}\lambda _{ i}w_{i} }$$

(23)

where

$$\displaystyle{ w_{i} = \frac{(1 + 1/d)\lambda _{i}^{2} + (1 + 2/d)\lambda _{i}\mathop{ \mathrm{trace}}\nolimits (\varSigma ) +\mathop{ \mathrm{trace}}\nolimits (\varSigma )^{2}/d} {(d + 2 + 1/d)\lambda _{i}^{2} + 2(1 + 1/d)\lambda _{i}\mathop{ \mathrm{trace}}\nolimits (\varSigma ) +\mathop{ \mathrm{trace}}\nolimits (\varSigma )^{2}/d}. }$$

(24)

Proof

We have for all $v \in \mathbb{R}^{p}$

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\mathop{\min }\limits_{\hat{\gamma }\in \mathbb{R}^{d}}\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }\hat{\gamma }\|_{ 2}^{2}] \leq \mathbb{E}_{\boldsymbol{\phi }}[\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}v\|_{ 2}^{2}]. }$$

Which we can minimize over the whole set $\mathbb{R}^{p}$:

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\mathop{\min }\limits_{\hat{\gamma }\in \mathbb{R}^{d}}\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }\hat{\gamma }\|_{ 2}^{2}] \leq \mathop{\min }\limits_{ v \in \mathbb{R}^{p}}\mathbb{E}_{\boldsymbol{\phi }}[\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}v\|_{ 2}^{2}]. }$$

This last expression we can calculate following the same path as in Theorem 1:

$$\displaystyle\begin{array}{rcl} \mathbb{E}_{\boldsymbol{\phi }}[\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}v\|_{ 2}^{2}]& =& \beta ^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}\beta - 2\beta ^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}\mathbb{E}_{\boldsymbol{\phi }}[\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}]v {}\\ & & +v^{{\prime}}\mathbb{E}_{\boldsymbol{\phi }}[\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}]v {}\\ & =& \beta ^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}\beta - 2\beta ^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}v {}\\ & & +(1 + 1/d)v^{{\prime}}\mathbf{X}^{{\prime}}\mathbf{X}v + \frac{\mathop{\mathrm{trace}}\nolimits (\varSigma )} {d} \|v\|_{2}^{2}, {}\\ \end{array}$$

where Σ = X ^′ X. Next we minimize the above expression w.r.t v. For this we take the derivative w.r.t. v and then we zero the whole expression. This yields

$$\displaystyle{ 2\Big(1 + \frac{1} {d}\Big)\varSigma v + 2\frac{\mathop{\mathrm{trace}}\nolimits (\varSigma )} {d} I_{p\times p}v - 2\varSigma \beta = 0. }$$

Hence we have

$$\displaystyle{ v =\Big (\Big(1 + \frac{1} {d}\Big)\varSigma + \frac{\mathop{\mathrm{trace}}\nolimits (\varSigma )} {d} I_{p\times p}\Big)^{-1}\varSigma \beta, }$$

which is element wise equal to

$$\displaystyle{ v_{i} = \frac{\beta _{i}\lambda _{i}} {(1 + 1/d)\lambda _{i} +\mathop{ \mathrm{trace}}\nolimits (\varSigma )/d}. }$$

Define the notation $s =\mathop{ \mathrm{trace}}\nolimits (\varSigma )$. We now plug this back into the original expression and get

$$\displaystyle\begin{array}{rcl} \mathop{\min }\limits_{v \in \mathbb{R}^{p}}\mathbb{E}_{\boldsymbol{\phi }}[\|\mathbf{X}\beta -\mathbf{X}\boldsymbol{\phi }\boldsymbol{\phi }^{{\prime}}v\|_{ 2}^{2}]& =& \beta ^{{\prime}}\varSigma \beta - 2\beta ^{{\prime}}\varSigma v {}\\ & & +(1 + 1/d)v^{{\prime}}\varSigma v + \frac{s} {d}\|v\|_{2}^{2} {}\\ & =& \sum _{i=1}^{p}\beta _{ i}^{2}\lambda _{ i} - 2\beta _{i}v_{i}\lambda _{i} + (1 + 1/d)v_{i}^{2}\lambda _{ i} + s/dv_{i}^{2} {}\\ & =& \sum _{i=1}^{p}\Big(\beta _{ i}^{2}\lambda _{ i} - 2\beta _{i}^{2}\lambda _{ i} \frac{\lambda _{i}} {(1 + 1/d)\lambda _{i} + s/d} {}\\ & & +\beta _{i}^{2}\lambda _{ i}(1 + 1/d) \frac{\lambda _{i}^{2}} {((1 + 1/d)\lambda _{i} + s/d)^{2}} {}\\ & & +\beta _{i}^{2}\lambda _{ i}\frac{s} {d} \frac{\lambda _{i}} {((1 + 1/d)\lambda _{i} + s/d)^{2}}\Big) {}\\ & =& \sum _{i=1}^{p}\beta _{ i}^{2}\lambda _{ i}w_{i}, {}\\ \end{array}$$

by combining the summands we get for w _i the expression mentioned in the theorem. □ 

Theorem  3 Assume Rank(X) ≥ d, then the MSE (4) equals

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\|\mathbf{X}\beta -\mathbf{X}\hat{\beta }_{d}^{\boldsymbol{\phi }}\|_{ 2}^{2}]] =\sigma ^{2}d +\sum _{ i=1}^{p}\beta _{ i}^{2}\lambda _{ i}\Big(1 -\frac{\lambda _{i}} {\eta _{i}}\Big). }$$

(25)

Furthermore we have

$$\displaystyle{ \sum _{i=1}^{p}\frac{\lambda _{i}} {\eta _{i}} = d. }$$

(26)

Proof

Calculating the expectation yields

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\|\mathbf{X}\beta -\mathbf{X}\hat{\beta }_{d}\|_{2}^{2}]] =\beta ^{{\prime}}\varSigma \beta - 2\beta ^{{\prime}}\varSigma T_{ d}^{\phi }\varSigma \beta + \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[Y ^{{\prime}}\mathbf{X}\phi _{ d}^{\mathbf{X}}\mathbf{X}^{{\prime}}Y ]]. }$$

Going through these terms we get:

$$\displaystyle\begin{array}{rcl} \beta ^{{\prime}}\varSigma \beta & =& \sum _{ i=1}^{p}\beta _{ i}^{2}\lambda _{ i} {}\\ \beta ^{{\prime}}\varSigma T_{ d}^{\phi }\varSigma \beta & =& \sum _{ i=1}^{p}\beta _{ i}^{2}\frac{\lambda _{i}^{2}} {\eta _{i}} {}\\ \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[Y ^{{\prime}}\mathbf{X}\phi _{ d}^{\mathbf{X}}\mathbf{X}^{{\prime}}Y ]]& =& \beta ^{{\prime}}\varSigma \mathbb{E}_{\boldsymbol{\phi }}[\phi _{ d}^{\mathbf{X}}]\varSigma \beta + \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\varepsilon ^{{\prime}}\mathbf{X}\phi _{ d}^{\mathbf{X}}\mathbf{X}^{{\prime}}\varepsilon ]]. {}\\ \end{array}$$

The first term in the last line equals ∑ _i = 1 ^p β _i ² λ _i ²∕η _i . The second can be calculated in two ways, both relying on the shuffling property of the trace operator:

$$\displaystyle\begin{array}{rcl} \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\varepsilon ^{{\prime}}\mathbf{X}\phi _{ d}^{\mathbf{X}}\mathbf{X}^{{\prime}}\varepsilon ]]& =& \mathbb{E}_{\varepsilon }[\varepsilon ^{{\prime}}\mathbf{X}T_{ d}^{\mathbf{X}}\mathbf{X}^{{\prime}}\varepsilon ]] =\sigma ^{2}\mathop{ \mathrm{trace}}\nolimits (\mathbf{X}T_{ d}^{\mathbf{X}}\mathbf{X}^{{\prime}}) {}\\ & =& \sigma ^{2}\mathop{ \mathrm{trace}}\nolimits (\varSigma T_{ d}^{\mathbf{X}}) =\sum _{ i=1}^{p}\frac{\lambda _{i}} {\eta _{i}}. {}\\ \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\varepsilon ^{{\prime}}\mathbf{X}\phi _{ d}^{\mathbf{X}}\mathbf{X}^{{\prime}}\varepsilon ]]& =& \sigma ^{2}\mathbb{E}_{\boldsymbol{\phi }}[\mathop{\mathrm{trace}}\nolimits (\mathbf{X}\phi _{ d}^{\mathbf{X}}\mathbf{X}^{{\prime}})] =\sigma ^{2}\mathbb{E}_{\boldsymbol{\phi }}[\mathop{\mathrm{trace}}\nolimits (\varSigma \phi _{ d}^{\mathbf{X}})] {}\\ & =& \sigma ^{2}\mathbb{E}_{\boldsymbol{\phi }}[\mathop{\mathrm{trace}}\nolimits (I_{ d\times d})] =\sigma ^{2}d. {}\\ \end{array}$$

Adding the first version to the expectation from above we get the exact expected mean-squared error. Setting both versions equal we get the equation

$$\displaystyle{ d =\sum _{ i=1}^{p}\frac{\lambda _{i}} {\eta _{i}}\,\,\,\,. }$$

 □ 

Theorem  4 Assume Rank(X) ≥ d, then there exists a real number τ ∈ [d ² ∕p,d] such that the AMSE of $\hat{\beta }_{d}$ can be bounded from above by

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\|\mathbf{X}\beta -\mathbf{X}\hat{\beta }_{d}\|_{2}^{2}]] \leq \sigma ^{2}\tau +\sum _{ i=1}^{p}\beta _{ i}^{2}\lambda _{ i}w_{i}^{2}, }$$

where the w _i ’s are given as

$$\displaystyle{ w_{i} = \frac{(1 + 1/d)\lambda _{i}^{2} + (1 + 2/d)\lambda _{i}\mathop{ \mathrm{trace}}\nolimits (\varSigma ) +\mathop{ \mathrm{trace}}\nolimits (\varSigma )^{2}/d} {(d + 2 + 1/d)\lambda _{i}^{2} + 2(1 + 1/d)\lambda _{i}\mathop{ \mathrm{trace}}\nolimits (\varSigma ) +\mathop{ \mathrm{trace}}\nolimits (\varSigma )^{2}/d} }$$

and

$$\displaystyle{ \tau \in [d^{2}/p,d]. }$$

Proof

First a simple calculation [10] using the closed form solution gives the following equation:

$$\displaystyle{ \mathbb{E}_{\boldsymbol{\phi }}[\mathbb{E}_{\varepsilon }[\|\mathbf{X}\beta -\mathbf{X}\hat{\beta }_{d}\|_{2}^{2}]] =\sigma ^{2}\sum _{ i=1}^{p}\Big(\frac{\lambda _{i}} {\eta _{i}}\Big)^{2} +\sum _{ i=1}^{p}\beta _{ i}^{2}\lambda _{ i}\Big(1 -\frac{\lambda _{i}} {\eta _{i}}\Big)^{2}. }$$

(27)

Now using the corollary from the last section we can bound the second term by the following way:

$$\displaystyle{ \Big(1 -\frac{\lambda _{i}} {\eta _{i}}\Big)^{2} \leq w_{ i}^{2}. }$$

(28)

For the first term we write

$$\displaystyle{ \tau =\sum _{ i=1}^{p}\Big(\frac{\lambda _{i}} {\eta _{i}}\Big)^{2}. }$$

(29)

Now note that since λ _i ∕η _i  ≤ 1 we have

$$\displaystyle{ \Big(\frac{\lambda _{i}} {\eta _{i}}\Big)^{2} \leq \frac{\lambda _{i}} {\eta _{i}} }$$

(30)

and thus we get the upper bound by

$$\displaystyle{ \sum _{i=1}^{p}\Big(\frac{\lambda _{i}} {\eta _{i}}\Big)^{2} \leq \sum _{ i=1}^{p}\frac{\lambda _{i}} {\eta _{i}} = d. }$$

(31)

For the lower bound of τ we consider an optimization problem. Denote $t_{i} = \frac{\lambda _{i}} {\eta _{i}}$, then we want to find $t \in \mathbb{R}^{p}$ such that

$$\displaystyle{ \sum _{i=1}^{p}t_{ i}^{2}\text{ is minimal } }$$

under the restrictions that

$$\displaystyle{\sum _{i=1}^{p}t_{ i} = d\text{ and }0 \leq t_{i} \leq 1.}$$

The problem is symmetric in each coordinate and thus t _i  = c. Plugging this into the linear sum gives c = d∕p and we calculate the quadratic term to give the result claimed in the theorem. □