A Systematic Study into the Factors that Affect the Predictive Accuracy of Multilevel VAR(1) Models

Abstract

The use of multilevel VAR(1) models to unravel within-individual process dynamics is gaining momentum in psychological research. These models accommodate the structure of intensive longitudinal datasets in which repeated measurements are nested within individuals. They estimate within-individual auto- and cross-regressive relationships while incorporating and using information about the distributions of these effects across individuals. An important quality feature of the obtained estimates pertains to how well they generalize to unseen data. Bulteel and colleagues (Psychol Methods 23(4):740–756, 2018a) showed that this feature can be assessed through a cross-validation approach, yielding a predictive accuracy measure. In this article, we follow up on their results, by performing three simulation studies that allow to systematically study five factors that likely affect the predictive accuracy of multilevel VAR(1) models: (i) the number of measurement occasions per person, (ii) the number of persons, (iii) the number of variables, (iv) the contemporaneous collinearity between the variables, and (v) the distributional shape of the individual differences in the VAR(1) parameters (i.e., normal versus multimodal distributions). Simulation results show that pooling information across individuals and using multilevel techniques prevent overfitting. Also, we show that when variables are expected to show strong contemporaneous correlations, performing multilevel VAR(1) in a reduced variable space can be useful. Furthermore, results reveal that multilevel VAR(1) models with random effects have a better predictive performance than person-specific VAR(1) models when the sample includes groups of individuals that share similar dynamics.

Appendix

Performance Measures to Evaluate Estimation Accuracy. While we cannot directly estimate the mean squared bias or variance of the predictions in our CV setting, an easy way to shed some light on the relation between predictive accuracy and estimation accuracy is to investigate how well person-specific and multilevel VAR(1) models can accurately estimate the elements of the true transition matrix $\varvec{\Psi }_i$, underlying our simulated data. Specifically, using the complete data sets (i.e., without further splitting them in a training and test part), we can compute the mean squared estimation error for the autoregressive effects as follows:

$$\begin{aligned} \text {MSE}_{\psi _{jj}} = \frac{1}{N} \sum _{i=1}^N \left( \frac{1}{P} \sum _{j=1}^{P} \big ({\hat{\psi }}_{i j j} - \psi _{i j j}\big )^2 \right) \end{aligned}$$

(12)

where $\psi _{i j j}$ and ${\hat{\psi }}_{i j j}$ denote the true and estimated autoregressive effects. Additionally, we can compute the mean squared estimation errors for the cross-regressive effects:

$$\begin{aligned} \text {MSE}_{\psi _{jk}} = \frac{1}{N} \sum _{i=1}^N \left( \frac{1}{P(P-1)} \sum _{j=1}^{P} \sum _{\begin{array}{c} k=1 \\ k\ne j \end{array}}^{P} \big ({\hat{\psi }}_{i j k} - \psi _{i j k}\big )^2 \right) \end{aligned}$$

(13)

where $\psi _{i j k}$ and ${\hat{\psi }}_{i j k}$ denote the true and estimated cross-regressive effects. We expect that these two MSE measures are strongly correlated to how well a considered modeling approach estimates the target function $f(\mathbf{Y }_{it-1})$. We will do that for Study I and report the results below (Tables 9 and 10). Figures 5, 6, 7, 8, 9, 10, 11, and 12 display the histograms of the distribution of the auto- and cross-regressive effects across persons when data are generated from a multilevel VAR(1) model with random effects, a person-specific VAR(1) model, or a cluster-specific VAR(1) model.

Table 9 Simulation results of study I. The mean squared estimation error (standard deviation in parentheses) for the autoregressive effects when data are generated using person-specific and multilevel VAR(1) models.

Table 10 Simulation results of study I. The mean squared estimation error (standard deviation in parentheses) for the cross-regressive effects when data are generated using person-specific and multilevel VAR(1) models.

Fig. 5

Distribution of the elements of the transition matrix for a multilevel VAR(1) model with random effects with 4 variables and 60 individuals. The vertical lines represent the fixed effects.

Fig. 6

Distribution of the elements of the transition matrix for a person-specific VAR(1) model with 4 variables and 60 individuals.

Fig. 7

Distribution of the elements of the transition matrix for a cluster-specific VAR(1) model with random effects with 4 variables, 2 clusters of equal size, and 60 individuals. The vertical lines represent the fixed effects for each cluster.

Fig. 8

Distribution of the elements of the transition matrix for a cluster-specific VAR(1) model with random effects with 4 variables, 2 clusters with one cluster including 10% of the persons, and 60 individuals. The vertical lines represent the fixed effects for each cluster.

Fig. 9

Distribution of the elements of the transition matrix for a cluster-specific VAR(1) model with random effects with 4 variables, 2 clusters with one cluster including 60% of the persons, and 60 individuals. The vertical lines represent the fixed effects for each cluster.

Fig. 10

Distribution of the elements of the transition matrix for a cluster-specific VAR(1) model with random effects with 4 variables, 4 clusters of equal size, and 60 individuals. The vertical lines represent the fixed effects for each cluster.

Fig. 11

Distribution of the elements of the transition matrix for a cluster-specific VAR(1) model with random effects with 4 variables, 4 clusters with one cluster including 10% of the persons, and 60 individuals. The vertical lines represent the fixed effects for each cluster.

Fig. 12

Distribution of the elements of the transition matrix for a cluster-specific VAR(1) model with random effects with 4 variables, 4 clusters with one cluster including 60% of the persons, and 60 individuals. The vertical lines represent the fixed effects for each cluster.