Variable selection for linear regression in large databases: exact methods

Abstract

This paper analyzes the variable selection problem in the context of Linear Regression for large databases. The problem consists of selecting a small subset of independent variables that can perform the prediction task optimally. This problem has a wide range of applications. One important type of application is the design of composite indicators in various areas (sociology and economics, for example). Other important applications of variable selection in linear regression can be found in fields such as chemometrics, genetics, and climate prediction, among many others. For this problem, we propose a Branch & Bound method. This is an exact method and therefore guarantees optimal solutions. We also provide strategies that enable this method to be applied in very large databases (with hundreds of thousands of cases) in a moderate computation time. A series of computational experiments shows that our method performs well compared to well-known methods in the literature and with commercial software.

Appendices

Appendix 1. Corollary used by our Branch & Bound methods

1.1 Corollary

∀p, p ’ ∈ {1, …, n }, if p < p’ then g(p) ≤ g(p’).

Proof:

By simplifying, we can define p ’ = p + 1, and ${S}_p^{\ast }=\left\{1,..,p\right\}$.

Let us also define S ’ = {1, …, p, p + 1}. Obviously, ${S}_p^{\ast}\subset {S}^{\prime }$. Therefore

$$ g(p)=f\left({S}_p^{\ast}\right)\le f\left(S'\right)\le \mathit{\max}\ \left\{f(S):S\subset V,\left|S\right|=p+1\right\}=g\left(p+1\right). $$

Appendix 2. Calculation of the objective function

1.1 Pre-process for calculating the objective function

To facilitate the calculation of the objective function f(S) a pre-calculation is initially performed before beginning to execute the algorithms. The matrix of independent variables X and the vector of the dependent variable Y are considered, as defined in section 2. In other words

X = (x_ij)_{i = 1, …, m; j = 1, …, n} e Y = (y_i)_{i = 1, …, m}

The pre-process consists of the following steps:

- The matrix ${X}^{\ast }={\left({x}_{ij}^{\ast}\right)}_{i=1,\dots, m;j=1,\dots, n}$ is calculated where ${x}_{ij}^{\ast }=\left(\frac{x_{ij}-{\overline{x}}_j}{\sqrt{m}\cdotp {s}_j}\right)$, i = 1, …, m; j = 1, …, n

and ${\overline{x}}_j$ and ${s}_j^2$ are respectively the mean and the sample variance of the variable j; j = 1, …, n.

- The matrix ${Y}^{\ast }={\left({y}_i^{\ast}\right)}_{i=1,\dots, m}$ is calculated where ${y}_i^{\ast }=\left(\frac{y-\overline{y}}{\sqrt{m}\cdotp {s}_y}\right)$, i = 1, …, m; and $\overline{y}$ and ${s}_y^2$ are respectively the mean and the sample variance of the variable Y.

- The matrix R = X^∗′ · X^∗ and the vector H = X^∗′ · Y^∗ are calculated. Note that R is the matrix of correlations of the independent variables. We denote the elements of R, j, j ′ = 1, …, n, as r_jj′ l; and the elements of H, j = 1, …, n, as h_j.

The matrix R and the vector H will be used in calculating f(S) for the various sets S in the algorithms proposed in this paper. This pre-process requires Θ(n² · m) operations. However, it is executed only once.

1.2 Calculation of the objective function f(S)

Let there be a set S ⊂ V of size p. We denote S = {s(1), s(2), …, s(p)}. The calculation of f(S) consists of the following steps:

- The matrix ${R}^S=\left({r}_{jj\prime}^S\right)$ is constructed, where ${r}_{jj\prime}^S={r}_{S(j)S\left({j}^{\prime}\right)}$, j, j^′ = 1, …, p;

- The vector ${H}^S=\left({h}_j^S\right)$, where ${h}_j^S={h}_{S(j)}$ j = 1, …, p;

- Find the inverse of the matrix R^S: (R^S)⁻¹

- Calculate the vector of coefficients Β = (R^S)⁻¹ · H^S. We denote the elements of Β, j = 1, …, p as β_j.

- Calculate the value of $(S)={\sum}_{j=1}^p{\sum}_{j^{\prime }=1}^p{\beta}_j\cdotp {\beta}_{j^{\prime }}\cdotp {r}_{jj\prime}^S$ .

This calculation requires Θ(p²) operations and is therefore independent of the number of cases m and of the initial number of variables n.

Appendix 3. Analysis of the complexity of our Branch & Bound methods

As described in Section 3, the Branch & Bound methods are based on a recursive exploration in the set of solutions. This set is represented by a search tree. Each node of this tree corresponds with a subset of solutions. In the exploration of the set of solutions corresponding to a node (Pseudocode 1), a variable a is selected and the corresponding set is divided into two subsets: one with the variable a fixed and another with the variable a forbidden. The process is then repeated with each subset. Since at most p₀ variables are selected (p₀ fixed variables), there are p₀ divisions until a solution with maximum size p₀ is reached. Therefore, $\theta \left({2}^{p_0}\right)$ nodes are explored. On the other hand, the number of variables that are examined to be selected is limited by n, so determining the variable a assumes θ(n) calculations of the function f (line 5 of Pseudocode 1). Therefore, the Branch & Bound methods calculate the objective function of $\theta \left(n\cdotp {2}^{p_0}\right)$ solutions S, with |S| ≤ p₀. As the calculation of each f(S) requires θ(|S|²) operations, the complexity of our methods is $\theta \left({p_0}^2\cdotp n\cdotp {2}^{p_0}\right)$.

Appendix 4. Basic ideas of traditional methods and our Branch & Bound methods

- Our Branch & Bound methods find the global optimum solution to the problem defined by (1)–(3) in Section 2.

- The Forward Method finds a solution to the same problem defined by (1)–(3), but this solution is and approximate solution (not necessarily the global optimal). The method works as follows:

1. Do S = ∅

Repeat

2. Determine i^∗ = argmax { f(S ∪ {i}) : i ∈ V − S}

3. Make S = S ∪ {i^∗}

$$ \left(\mathrm{Until}\right)\left|S\right|=p $$

Note that the problem defined by (1)–(3) can also be defined compactly as:

$$ {\mathit{\min}}_{\beta }\ {\left\Vert Y-{X}^{\ast}\beta \right\Vert}_2 $$

subject to:

$$ {\left\Vert \beta \right\Vert}_0=p $$

where

X^∗ = (1| X) and 1^t = (1, 1, …1) ∈ R^m, that is X^∗ is matrix X extended with vector 1

β^t = (β₀, β₁, ⋯, β_n) is the vector of the coefficients of the variables and the independent coefficient

‖_r = r - Norme

- The LASSO Method solves the following model:

$$ {\mathit{\min}}_{\beta}\kern0.5em {\left\Vert Y-{X}^{\ast}\beta \right\Vert}_2+\lambda \cdotp {\left\Vert \beta \right\Vert}_1 $$

where λ is a previously established positive parameter. The higher the value of λ the fewer the variables with a coefficient β_i ≠ 0 are selected – that is, the fewer variables are selected. To find the solution, methods based on the Coordinate Descent algorithm are usually used.

- The LARS method solves the same model as LASSO but using strategies analogous to the Forward method – that is, selecting in each step a variable to enter in the solution.

- The GARROTE method calculates its coefficients ${\beta}_i^g$ as follows:

$$ {\beta}_i^g={\left[1-\frac{\lambda }{{\left({\beta}_i^r\right)}^2}\right]}_{+}\cdotp {\beta}_i^r $$

where the values ${\beta}_i^r$ are the coefficients of the linear regression model with all variables, [z]₊ is the positive part of a real number z, and λ is a previously established positive parameter. As in previous model, the higher the value of λ the fewer the variables with coefficient β_i ≠ 0 – that is, the fewer variables are selected.