VANISH regularization for generalized linear models

Abstract

Marketers increasingly face modeling situations where the number of independent variables is large and possibly approaching or exceeding the number of observations. In this setting, covariate selection and model estimation present significant challenges to usual methods of inference. These challenges are exacerbated when covariate interactions are of interest. Most extant regularization methods make no distinction between main and interaction terms in estimation. The linear VANISH model is an exception to these methods. The linear VANISH model is a regularization method for models with interaction terms that ensures proper model hierarchy by enforcing the heredity principle. We derive the generalized VANISH model for nonlinear responses, including duration, discrete choice, and count models widely used in marketing applications. In addition, we propose a VANISH model that allows to account for unobserved consumer heterogeneity via a mixture approach. In three empirical applications we demonstrate that our proposed model outperforms main effects models as well as other methods that include interaction terms.

Appendix

We detail the steps in our VANISH regularization approach for the VANISH Hazard, VANISH Choice and VANISH Poisson models. For more information on the derivation of the full conditional distributions for the VANISH parameters (τ, ω, λ₁, λ₂) please see the Web Appendix.

1.1 Hazard model

• 1) Generate α and γ using a random-walk Metropolis-Hastings (MH) sampler based on the likelihood given by:

$$ L=\prod \limits_{i=1}^n\prod \limits_{t=1}^{t_i}{\Pr}_i{\left(t,{x}_{it}\right)}^{d_{it}}{\left(1-{\Pr}_i\left(t,{x}_{it}\right)\right)}^{1-{d}_{it}}, $$

where

$$ {\Pr}_i\left(t,{x}_{it}\right)=1-\frac{S_i\left(t,{x}_{it}\right)}{S_i\left(t-1,{x}_{it}\right)}=1-\exp \left(-\exp \left({x}_{it}\beta \right)\underset{t-1}{\overset{t}{\int }}{h}_i(u) du\right) $$

where d_it is 1 if the life event occurs and zero otherwise.

• 2) Generateβ using a random-walk MH sampler based on the likelihood given by:

$$ L=\prod \limits_{i=1}^n\prod \limits_{t=1}^{t_i}{\Pr}_i{\left(t,{x}_{it}\right)}^{d_{it}}{\left(1-{\Pr}_i\left(t,{x}_{it}\right)\right)}^{1-{d}_{it}} $$

the VANISH prior given by:

$$ {\beta}_j\mid ...\propto \exp \left[-\frac{{\left(\sum \limits_{k=j+1}^p|{\beta}_{jk}|\right)}^2}{2{\tau}_j^2}-\frac{\beta_j+\sum \limits_{k:k\ne j}{\beta}_{jk}}{2{\omega}_j^2}\right]. $$

• 3) Generate $\frac{1}{\tau^2}$

$$ \frac{1}{\tau^2}={\gamma}_j\mid ...\sim InverseGaussian\left(\sqrt{\frac{\lambda_1^2}{{\left(\sum \limits_{k=j+1}^p|{\beta}_{jk}|\right)}^2}},{\lambda}_1^2\right)I\left({\gamma}_j>0\right), $$

where I(⋅) is the indicator function.

• 4) Generate $\frac{1}{\omega^1}$

$$ \frac{1}{\omega^1}={\varphi}_j\mid ...\sim InverseGaussian\left(\sqrt{\frac{\lambda_2^2}{{\left({\beta}_j\right)}^2+\sum \limits_{k:k\ne j}{\left({\beta}_{jk}\right)}^2}},{\lambda}_2^2\right)I\left({\varphi}_j>0\right), $$

where I(⋅) is the indicator function.

• 5) Generate ${\lambda}_1^2\ \mathrm{and}\ {\lambda}_2^2$

$$ {\displaystyle \begin{array}{l}{\lambda}_1^2\mid ...\sim gamma\left(\frac{K}{2}+r,\frac{1}{2}\sum \limits_{j=1}^p{\tau}_j^2+s\right),\mathrm{K}=\#\mathrm{main}\ \mathrm{effects}+\#\mathrm{interaction}\ \mathrm{effects}\\ {}{\lambda}_2^2\mid ...\sim gamma\left(\frac{H}{2}+r,\frac{1}{2}\sum \limits_{j=1}^p{\omega}_j^2+s\right),\mathrm{H}=2\ast \#\mathrm{main}\ \mathrm{effects}+\#\mathrm{interaction}\ \mathrm{effects}\end{array}} $$

where r = 1,s = 0.1.

1.2 Choice model

• 1) Generate β^s using a random-walk MH sampler based on the likelihood given by (11) and (12) and the VANISH prior given by:

$$ {\beta}_{jk}^s\mid ...\propto \exp \left[-\frac{{\left(\sum \limits_{k=j+1}^p|{\beta}_{jk}^s|\right)}^2}{2{\left({\tau}_j^s\right)}^2}-\frac{\beta_j^s+\sum \limits_{k:k\ne j}{\beta}_{jk}^s}{2{\left({\omega}_j^s\right)}^s}\right]. $$

• 2) Generate $\frac{1}{{\left({\tau}^s\right)}^2}$

$$ \frac{1}{{\left({\tau}^s\right)}^s}={\gamma}_j^s\mid ...\sim InverseGaussian\left(\sqrt{\frac{{\left({\lambda}_1^s\right)}^2}{{\left(\sum \limits_{k=j+1}^p|{\beta}_{jk}^s|\right)}^2}},{\left({\lambda}_1^s\right)}^2\right)I\left({\gamma}_j^s>0\right), $$

where I(⋅)is the indicator function.

• 3) Generate $\frac{1}{{\left({\omega}^s\right)}^2}$

$$ \frac{1}{{\left({\omega}^s\right)}^2}={\varphi}_j^s\mid ...\sim InverseGaussian\left(\sqrt{\frac{{\left({\lambda}_2^s\right)}^s}{{\left({\beta}_j^s\right)}^2+\sum \limits_{k:k\ne j}{\left({\beta}_{jk}^s\right)}^2}},{\left({\lambda}_2^s\right)}^s\right)I\left({\varphi}_j^s>0\right), $$

where I(⋅) is the indicator function.

• 4) Generate ${\left({\lambda}_1^s\right)}^2\ \mathrm{and}\ {\left({\lambda}_2^s\right)}^2$

$$ {\displaystyle \begin{array}{l}{\left({\lambda}_1^s\right)}^2\mid ...\sim gamma\left(\frac{K}{2}+{r}^{lam},\frac{1}{2}\sum \limits_{j=1}^p{\left({\tau}_j^s\right)}^2+{s}^{lam}\right),\mathrm{K}=\#\mathrm{main}\ \mathrm{effects}+\#\mathrm{interaction}\ \mathrm{effects}\\ {}{\left({\lambda}_2^s\right)}^s\mid ...\sim gamma\left(\frac{H}{2}+{r}^{lam},\frac{1}{2}\sum \limits_{j=1}^p{\left({\omega}_j^s\right)}^2+{s}^{lam}\right),\mathrm{H}=2\ast \#\mathrm{main}\ \mathrm{effects}+\#\mathrm{interaction}\ \mathrm{effects}\end{array}} $$

where r^lam1, s^lam = 0.1.

• 5) Generate π

• $\pi \mid Z,\rho \sim Dir\left[\left({\tilde{\rho}}_1...{\tilde{\rho}}_S\right)\right]$, where ${\tilde{\rho}}_s={\rho}_s+\sum \limits_{i=1}^nI\left({Z}_i=s\right)$ with prior ρ = (1, ..., 1).

• 6) Generate Z_i

• Z_i ∣ θ, μ, V, π~multinomial(1, [LR₁(βⁱ), ..., LR_S(βⁱ)]),

where $L{R}_l\left({\beta}^i\right)=\frac{\pi_lL\left({\beta}^i\right)}{\sum \limits_{s=1}^S{\pi}_sL\left({\beta}^i\right)}$ and L is the likelihood given by (11) and (12).

• 7) Choose a permutation ${\xi}_t^{+}$ and relabel draws according to ${\xi}_t^{+}$

$$ {\xi}_t^{+}=\underset{\xi_t}{\mathrm{argmin}}\sum \limits_{i=1}^n\sum \limits_{s=1}^S{p}_{is}\left({\xi}_t\left({\mu}^t,{V}^t\right)\right)\log \left[\frac{p_{is}\left({\xi}_t\left({\mu}^t,{V}^t\right)\right)}{{\hat{q}}_{is}^{t-1}}\right]. $$

• 8) Set

$$ {\hat{Q}}^t=\frac{t{\hat{Q}}^{t-1}+P\left({\xi}_t^{\ast}\left({\mu}^t,{V}^t\right)\right)}{t+1}. $$

1.3 Poisson model

$$ {\theta}_t={x}_t^{sea}{\beta}^{sea}+{x}_t^{imp}{\beta}^{imp}+{x}_t^{pos}{\beta}^{pos}+{x}_t^{cpc}{\beta}^{cpc}+\sum \limits_j{x}_{jt}^{txt}{\beta}^{txt}+\sum \limits_{j<k}{x}_{jt}^{txt}{x}_{kt}^{txt}{\beta}_{jk}^{txt} $$

• 1) Generate [β^seaβ^impβ^posβ^cpc] using a random-walk Metropolis-Hastings (MH) sampler based on the likelihood given by:

$$ L\propto \prod \limits_{t=1}^n\frac{e^{-{\lambda}_t}{\lambda}_t^{y_t}}{y_t!}. $$

• 2) Generate β^txt using a random-walk MH sampler based on the likelihood given by:

$$ L\propto \prod \limits_{t=1}^n\frac{e^{-{\lambda}_t}{\lambda}_t^{y_t}}{y_t!} $$

the VANISH prior given by:

$$ {\beta}_{jk}\mid ...\propto \exp \left[-\frac{{\left(\sum \limits_{k=j+1}^p|{\beta}_{jk}|\right)}^2}{2{\tau}_j^2}-\frac{\beta_j+\sum \limits_{k:k\ne j}{\beta}_{jk}}{2{\omega}_j^2}\right]. $$

• 3) Generate $\frac{1}{\tau^2}$

$$ \frac{1}{\tau^2}={\gamma}_j\mid ...\sim InverseGaussian\left(\sqrt{\frac{\lambda_1^2}{{\left(\sum \limits_{k=j+1}^p|{\beta}_{jk}|\right)}^2}},{\lambda}_1^2\right)I\left({\gamma}_j>0\right), $$

where I(⋅) is the indicator function.

• 4) Generate $\frac{1}{\omega^1}$

$$ \frac{1}{\omega^1}={\varphi}_j\mid ...\sim InverseGaussian\left(\sqrt{\frac{\lambda_2^2}{{\left({\beta}_j\right)}^2+\sum \limits_{k:k\ne j}{\left({\beta}_{jk}\right)}^2}},{\lambda}_2^2\right)I\left({\varphi}_j>0\right), $$

where I(⋅) is the indicator function.

• 5) Generate ${\lambda}_1^2\ \mathrm{and}\ {\lambda}_2^2$

$$ {\displaystyle \begin{array}{l}{\lambda}_1^2\mid ...\sim gamma\left(\frac{K}{2}+r,\frac{1}{2}\sum \limits_{j=1}^p{\tau}_j^2+s\right),\mathrm{K}=\#\mathrm{main}\ \mathrm{effects}+\#\mathrm{interaction}\ \mathrm{effects}\\ {}{\lambda}_2^2\mid ...\sim gamma\left(\frac{H}{2}+r,\frac{1}{2}\sum \limits_{j=1}^p{\omega}_j^2+s\right),\mathrm{H}=2\ast \#\mathrm{main}\ \mathrm{effects}+\#\mathrm{interaction}\ \mathrm{effects}\end{array}} $$

where r = 1,s = 0.1.