Regularization

Abstract

In statistics, we assume that the number of samples N is larger than the number of variables p. Otherwise, linear regression will not produce any least squares solution, or it will find the optimal variable set by comparing the information criterion values of the $2^p$ subsets of the cardinality p. Therefore, it is difficult to estimate the parameters. In such a situation, regularization is often used. In the case of linear regression, we add a penalty term to the squared error to prevent the coefficient value from increasing. When the regularization term is a constant $\lambda$ times the L1 and L2 norms of the coefficient, the method is called lasso and ridge, respectively. In the case of lasso, as the constant $\lambda$ increases, some coefficients become 0; finally, all coefficients become 0 when $\lambda$ is infinity. In that sense, lasso plays a role of model selection. In this chapter, we consider the principle of lasso and compare it with ridge. Finally, we learn how to choose the constant $\lambda$.

Exercise 49–56

• 49. Let $N, p\ge 1$. For $X\in {\mathbb R}^{N\times p}$ and $y\in {\mathbb R}^N,\ \lambda \ge 0$, we wish to obtain $\beta \in {\mathbb R}^{p}$ that minimizes

  $$\frac{1}{N}\Vert y-X\beta \Vert ^2+\lambda \Vert \beta \Vert _2^2\ ,$$

  where for $\beta =(\beta _1,\ldots ,\beta _p)$, we denote $\Vert \beta \Vert _2:=\sqrt{\sum _{j=1}^p\beta _j^2}$. Suppose $N<p$. Show that such a solution always exists and that it is equivalent to $\lambda >0$. Hint: In order to show a necessary and sufficient condition, both directions should be proved.

• 50.

  • (a) Suppose that a function $f: {\mathbb R}\rightarrow {\mathbb R}$ is convex and differentiable at $x=x_0$. Show that a z exists for an arbitrary $x\in {\mathbb R}$ such that $f(x)\ge f(x_0)+z(x-x_0)$ (subderivative) and that it coincides with the differential coefficient $f'(x_0)$ at $x=x_0$.

  • (b) Show that $-1\le z\le 1$ is equivalent to $zx \le |x|$ for all $x\in {\mathbb R}$.

  • (c) Find the set of z defined in (a) for function $f(x)=|x|$ and $x_0\in {\mathbb R}$. Hint: Consider the cases $x_0>0$, $x_0<0$, and $x_0=0$.

  • (d) Compute the subderivatives of $f(x)=x^2-3x+|x|$ and $f(x)=x^2+x+2|x|$ for each point, and find the maximal and minimal values for each of the two functions.

• 51. Write an R program soft.th(lambda,x) of the function $S_\lambda (x)$, $\lambda >0$, $x\in {\mathbb R}$ defined by

  $$S_\lambda (x):=\left\{ \begin{array}{ll} x-\lambda ,&{}x>\lambda \\ 0,&{}|x|\le \lambda \\ x+\lambda ,&{}x<-\lambda \end{array} \right. $$

  and execute the following.

  

  Hint: Use pmax rather than max.

• 52. We wish to find the $\beta \in {\mathbb R}$ that minimizes

  $$L=\frac{1}{2N}\sum _{i=1}^N(y_i-x_i\beta )^2+\lambda |\beta |$$

  given $(x_i,y_i)\in {\mathbb R}\times {\mathbb R},\ i=1,\ldots ,N,\ \lambda >0$, where we assume that $x_1,\ldots ,x_N$ have been scaled so that $\frac{1}{N}\sum _{i=1}^N x_i^2=1$. Express the solution by $z:=\frac{1}{N}\sum _{i=1}^N x_iy_i$ and function $\mathcal{S}_\lambda (\cdot )$.

• 53. For $p>1$ and $\lambda >0$, we estimate the coefficients $\beta _0\in {\mathbb R}$ and $\beta \in {\mathbb R}^p$ as follows: Initially, we randomly give the coefficients $\beta \in {\mathbb R}^p$. Then, we update $\beta _j$ by $\displaystyle \mathcal{S}_\lambda \left( \sum _{i=1}^N \frac{x_{i,j}r_{i,j}}{N}\right)$, where $\displaystyle r_{i,j}:=y_i-\sum _{k\not =j}x_{i,j}\beta _j$. We repeat this process for $j=1,\ldots ,p$ and repeat the cycle until convergence. The function lasso below is used to scale the sample-based variance to one for each of the p variables before estimation of $(\beta _0,\beta )$. Fill in the blanks and execute the procedure.

  

• 54. Transform Problem 53 (Lasso) into the setting in Problem 49 (Ridge) and execute it. Hint: Replace the line of eps and the while loop in the function lasso by

  

  and change the function name to ridge. Blank (4) should be ridge rather than lasso.

• 55. Look up the meanings of glmnet and cv.glmnet and find the optimal $\lambda$ and $\beta$ for the data below. Which variables are selected among the five variables?

  

  Hint: The coefficients are displayed via fit$beta. If a coefficient is nonzero, we consider it to be selected.

• 56. Given $x_{i,1},x_{i,2},y_i\in {\mathbb R},\ i=1,\ldots ,N$, let $\hat{\beta }_1,\hat{\beta }_2$ be the $\beta _1,\beta _2$ that minimize $\displaystyle S:=\sum _{i=1}^N(y_i-\beta _1x_{i,1}-\beta _2x_{i,2})^2$ given $\hat{\beta }_1x_{i,1}+\hat{\beta }_{2}x_{i,2}$, $\hat{y}_i$, $(i=1,\ldots ,N)$. Show the following three equations.

  • (a)
    $$\sum _{i=1}^Nx_{i,1}(y_i-\hat{y}_i)=\sum _{i=1}^Nx_{i,2}(y_i-\hat{y}_i)=0.$$

    For arbitrary $\beta _1,\beta _2$,

    $$y_i-\beta _1x_{i,1}-\beta _2x_{i,2}=y_i-\hat{y}_i-(\beta _1-\hat{\beta }_1)x_{i,1}-(\beta _2-\hat{\beta }_2)x_{i,2}.$$

    For arbitrary $\beta _1,\beta _2$, $\displaystyle \sum _{i=1}^N(y_i-\beta _1x_{i,1}-\beta _2x_{i,2})^2$ can be expressed by

    $$\begin{aligned}&(\beta _1-\hat{\beta }_1)^2\sum _{i=1}^Nx_{i,1}^2+2(\beta _1-\hat{\beta }_1)(\beta _2-\hat{\beta }_2)\sum _{i=1}^Nx_{i,1}x_{i,2}+(\beta _2-\hat{\beta }_2)^2\sum _{i=1}^Nx_{i,2}^2\\&+\sum _{i=1}^N(y_i-\hat{y}_i)^2. \end{aligned}$$

  • (b) We consider the case $\displaystyle \sum _{i=1}^Nx_{i,1}^2=\sum _{i=1}^Nx_{i,2}^2=1,\ \sum _{i=1}^Nx_{i,1}x_{i,2}=0$. In the standard least squares method, we choose the coefficients as $\beta _1=\hat{\beta }_1$, $\beta _2=\hat{\beta }_2$. However, under the constraint that $|\beta _1|+|\beta _2|$ is less than a constant, we choose $(\beta _1,\beta _2)$ at which the circle with center $(\hat{\beta }_1,\hat{\beta }_2)$ and the smallest radius comes into contact with the rhombus. Suppose that we grow the radius of the circle with center $(\hat{\beta }_1,\hat{\beta }_2)$ until it comes into contact with the rhombus that connects $(1,0),(0,1),(-1,0),(0,-1)$. Show the region of the centers such that one of coordinates $(\hat{\beta }_1$ and $\hat{\beta }_2)$ is zero.

  • (c) What if the rhombus in (b) is replaced by a unit circle?