Modular regression - a Lego system for building structured additive distributional regression models with tensor product interactions

Abstract

Semiparametric regression models offer considerable flexibility concerning the specification of additive regression predictors including effects as diverse as nonlinear effects of continuous covariates, spatial effects, random effects, or varying coefficients. Recently, such flexible model predictors have been combined with the possibility to go beyond pure mean-based analyses by specifying regression predictors on potentially all parameters of the response distribution in a distributional regression framework. In this paper, we discuss a generic concept for defining interaction effects in such semiparametric distributional regression models based on tensor products of main effects. These interactions can be assigned anisotropic penalties, i.e. different amounts of smoothness will be associated with the interacting covariates. We investigate identifiability and the decomposition of interactions into main effects and pure interaction effects (similar as in a smoothing spline analysis of variance) to facilitate a modular model building process. The decomposition is based on orthogonality in function spaces which allows for considerable flexibility in setting up the effect decomposition. Inference is based on Markov chain Monte Carlo simulations with iteratively weighted least squares proposals under constraints to ensure identifiability and effect decomposition. One important aspect is therefore to maintain sparse matrix structures of the tensor product also in identifiable, decomposed model formulations. The performance of modular regression is verified in a simulation on decomposed interaction surfaces of two continuous covariates and two applications on the construction of spatio-temporal interactions for the analysis of precipitation on the one hand and functional random effects for analysing house prices on the other hand.

Appendices

Appendix

Mixed model decomposition

For the further developments, it is useful to study functional effects with partially improper prior to determine which parts of the function are assigned an informative prior and which ones are assigned a flat prior. While this is hidden in the multivariate normal prior (1), it can be made explicit by reparameterising the basis functions utilising the mixed model representation of structured additive regression terms. This reparameterisation can be obtained from the spectral decomposition of the prior precision matrix as

$$\begin{aligned} {{\varvec{K}}}= {\varvec{\Gamma }}{\varvec{\Omega }}{\varvec{\Gamma }}' \end{aligned}$$

where

$$\begin{aligned} {\varvec{\Omega }}= {{\,\mathrm{diag}\,}}(\omega _1,\ldots ,\omega _D), \qquad \omega _1\le \ldots \le \omega _D \end{aligned}$$

comprises the eigenvalues (in ascending order) and

$$\begin{aligned} {\varvec{\Gamma }}'{\varvec{\Gamma }}={{\varvec{I}}}\end{aligned}$$

is the orthonormal matrix of corresponding eigenvectors. Let now $M = \dim ({\varvec{\gamma }})-{{\,\mathrm{rk}\,}}({{\varvec{K}}}) = D - R > 0$ denote the rank deficiency of ${{\varvec{K}}}$, then the first M eigenvalues are equal to zero, i.e.  $\omega _1=\ldots =\omega _M=0$. Accordingly, we split the spectral decomposition into

$$\begin{aligned} {\varvec{\Gamma }}=[{\varvec{\Gamma }}_1,{\varvec{\Gamma }}_2], \qquad {\varvec{\Omega }}= {{\,\mathrm{blockdiag}\,}}({\varvec{\Omega }}_1,{\varvec{\Omega }}_2) \end{aligned}$$

where ${\varvec{\Omega }}_1={{\,\mathrm{diag}\,}}(\omega _1,\ldots ,\omega _M)$ and ${\varvec{\Omega }}_2={{\,\mathrm{diag}\,}}(\omega _{M+1},\ldots ,\omega _D)$ comprise zero and nonzero eigenvalues, respectively, while ${\varvec{\Gamma }}_1$ and ${\varvec{\Gamma }}_2$ are the corresponding matrices of eigenvectors of dimension $D\times M$ and $D\times R$. We then obtain a reduced representation of the precision matrix as

$$\begin{aligned} {{\varvec{K}}}={\varvec{\Gamma }}_2{\varvec{\Omega }}_2{\varvec{\Gamma }}_2'. \end{aligned}$$

Based on ${\tilde{{\varvec{X}}}}={\varvec{\Gamma }}_1$ and ${\tilde{{\varvec{Z}}}}={\varvec{\Gamma }}_2{\varvec{\Omega }}_2^{-1/2}$, we can now reparameterise ${\varvec{\gamma }}$ as

$$\begin{aligned} {\varvec{\gamma }}= {\tilde{{\varvec{X}}}}{\varvec{\beta }}+ {\tilde{{\varvec{Z}}}}{\varvec{\alpha }}\end{aligned}$$

where the new regression parameters ${\varvec{\beta }}$ and ${\varvec{\alpha }}$ (which are of dimension M and R, respectively) follow the prior specifications

$$\begin{aligned} p({\varvec{\beta }})\propto {{\,\mathrm{const}\,}}\qquad {\varvec{\alpha }}\sim {{\,\mathrm{N}\,}}(0,\tau ^2{{\varvec{I}}}_R). \end{aligned}$$

This can be interpreted as follows: A flat, noninformative prior is assigned to the vector of regression coefficients ${\varvec{\beta }}$ which therefore represents the part of the function that is not affected by the prior specification while ${\varvec{\alpha }}$ follows an i.i.d. Gaussian prior. In mixed model terminology, this means that ${\varvec{\beta }}$ comprises fixed effects while ${\varvec{\alpha }}$ are i.i.d. Gaussian random effects. Note that ${\tilde{{\varvec{X}}}}$ defines a basis of the null space of the prior precision matrix ${{\varvec{K}}}$ and in fact any alternative basis of this null space can be considered as well. In particular, for specific types of effects one can deduce bases that entail an easier interpretation.

In matrix notation, the mixed model reparameterisation induces a similar decomposition of the vector of function evaluations, i.e.

$$\begin{aligned} {{\varvec{f}}}= {{\varvec{X}}}{\varvec{\beta }}+ {{\varvec{Z}}}{\varvec{\alpha }}\end{aligned}$$

where ${{\varvec{X}}}={{\varvec{B}}}{\tilde{{\varvec{X}}}}$ and ${{\varvec{Z}}}={{\varvec{B}}}{\tilde{{\varvec{Z}}}}$. For the individual function evaluations, we obtain

$$\begin{aligned} f(\nu )= & {} x_1\beta _1 + \ldots + x_M\beta _M + \tilde{f}(\nu )\\= & {} {{\varvec{x}}}'{\varvec{\beta }}+ {{\varvec{z}}}(\nu )'{\varvec{\alpha }}\end{aligned}$$

where ${{\varvec{x}}}'={{\varvec{b}}}(\nu ){\tilde{{\varvec{X}}}}$ and ${{\varvec{z}}}(\nu )'={{\varvec{b}}}(\nu )'{\tilde{{\varvec{Z}}}}$. Following the mixed model representation above this means that $x_1,\ldots ,x_M$ are covariates with fixed effects, whereas $\tilde{f}(\nu )$ is a function that is obtained from $f(\nu )$ by removing these fixed effects by means of a reparameterisation. For the new function $\tilde{f}(\nu )$, the basis functions are defined in the vector ${{\varvec{z}}}(\nu )$.

Note that by construction the reparameterisation is a one-to-one transformation, and therefore, explicit expressions for the reparameterised regression coefficients can be obtained from

$$\begin{aligned} ({\tilde{{\varvec{X}}}},{\tilde{{\varvec{Z}}}})^{-1} = \begin{pmatrix}{\varvec{\Gamma }}_1'\\ {\varvec{\Omega }}_2^{1/2}{\varvec{\Gamma }}_2'\end{pmatrix} \end{aligned}$$

and are then given by

$$\begin{aligned} {\varvec{\beta }}= {\varvec{\Gamma }}_1'{\varvec{\gamma }}\qquad {\varvec{\alpha }}= {\varvec{\Omega }}_2^{1/2}{\varvec{\Gamma }}_2'{\varvec{\gamma }}. \end{aligned}$$

Algorithm for constrained sampling

As shown in Rue and Held (2005, Algorithm 2.6), a sample ${\varvec{\gamma }}$ can be modified to fulfil the constraint ${{\varvec{A}}}{\varvec{\gamma }}=\mathbf {0}$ applying the following steps:

• Compute the $D\times a$ matrix ${{\varvec{V}}}={{\varvec{P}}}^{-1}{{\varvec{A}}}'$ by solving the equation systems ${{\varvec{P}}}{{\varvec{V}}}={{\varvec{A}}}'$ for each of the columns of ${{\varvec{V}}}$, see Rue and Held (2005, Algorithm 2.1).

• Compute the $a\times a$ matrix ${{\varvec{W}}}={{\varvec{A}}}{{\varvec{V}}}$.

• Compute the $a\times D$ matrix ${{\varvec{U}}}={{\varvec{W}}}^{-1}{{\varvec{V}}}'$ by solving the equation systems ${{\varvec{W}}}{{\varvec{U}}}={{\varvec{V}}}'$ for each of the columns of ${{\varvec{U}}}$.

• Compute the constrained sample ${\varvec{\gamma }}^*={\varvec{\gamma }}-{{\varvec{U}}}'{{\varvec{A}}}{\varvec{\gamma }}$ where ${\varvec{\gamma }}$ is an unconstrained sample from ${{\,\mathrm{N}\,}}({\varvec{\mu }},{{\varvec{P}}}^{-1})$. Note that all four steps of the sampling scheme have to be conducted in each iteration of the MCMC algorithm since typically the precision matrix ${{\varvec{P}}}$ changes over the iterations.