Concealed correlations meta-analysis: A new method for synthesizing standardized regression coefficients

Abstract

The synthesis of standardized regression coefficients is still a controversial issue in the field of meta-analysis. The difficulty lies in the fact that the standardized regression coefficients belonging to regression models that include different sets of covariates do not represent the same parameter, and thus their direct combination is meaningless. In the present study, a new approach called concealed correlations meta-analysis is proposed that allows for using the common information that standardized regression coefficients from different regression models contain to improve the precision of a combined focal standardized regression coefficient estimate. The performance of this new approach was compared with that of two other approaches: (1) carrying out separate meta-analyses for standardized regression coefficients from studies that used the same regression model, and (2) performing a meta-regression on the focal standardized regression coefficients while including an indicator variable as a moderator indicating the regression model to which each standardized regression coefficient belongs. The comparison was done through a simulation study. The results showed that, as expected, the proposed approach led to more accurate estimates of the combined standardized regression coefficients under both random- and fixed-effect models.

The synthesis of standardized regression coefficients has received a lot of attention over the last few decades (e.g., Becker & Wu, 2007; Hanushek, 1974; Kim, 2011; Novick, Jackson, Thayer, & Cole, 1972; Peterson & Brown, 2005; Stanley & Jarrell, 1989), because standardized regression coefficients are effect sizes commonly used in various domains, including medicine (e.g., Nieminen, Lehtiniemi, Vähäkangas, Huusko, & Rautio, 2013; Paul, Lipps, & Madden, 2006), psychology (e.g., Pratt et al., 2009), and educational sciences (e.g., Bowman, 2012). By definition, a standardized regression coefficient (also called a beta weight) represents the estimated number of standard deviations of change in the outcome variable for one standard deviation unit change in the predictor, while controlling for other predictors. The meta-analysis of standardized regression coefficients has the potential to yield a more accurate estimate of the effect of a predictor variable on a dependent variable after controlling for other variables that might also be related to the outcome variable.

Standardizing regression coefficients is especially convenient in fields in which the dependent variable can be measured on several scales, since thanks to the standardization the coefficients become comparable. For example, in the meta-analysis by Nieminen et al. (2013), the relationship between maternal exposure to polychlorinated biphenyls and the birth weights of newborns was explored. They affirmed that beta weights are “natural measures of interest” (p. 3), not only because the exposure levels in the original studies were measured in different units, but also because of many confounding variables (e.g., smoking, age, body mass index of the mother, etc.) that needed to be controlled for. Another example is the meta-analysis by Yin, Schmidt, and Besag (2006), in which the combination of standardized regression slopes was useful because it permitted “aggregation and comparison across states using different assessment tests” (p. 53). Another example is found in the meta-analysis by Ferguson (2015a), in which the influence of video games on children’s and adolescent’s aggression, mental health, and other variables was studied. In Ferguson’s words, “Controlling for gender as well as other theoretically critical factors—such as trait aggression, family violence, peer delinquency, and mental health—is essential,” (p. 648) since these are potentially confounding variables. For these reasons, many researchers advocate the use of standardized regression coefficients as effect sizes (e.g., Ferguson, 2015b; Pratt et al., 2009). Nevertheless, the use of beta weights as effect sizes also has its detractors (see, e.g., Greenland, Maclure, Schlesselman, Poole, & Morgenstern, 1991), because of beta weights’ incomparability across studies when the same dependent variable has dissimilar variances in different studies or when the regression models from which the beta weights are taken include different covariates.

Nowadays, many methods for synthesizing regression coefficients exist. Most of these methods are of one of two kinds: univariate methods to combine regression coefficients for each predictor separately, such as classical weighted least squares methods (Hedges & Olkin, 1985; used by, e.g., Bini, Coelho, & Diniz-Filho, 2001), and multivariate methods to combine the vectors of regression coefficients, such as the generalized least squares approach described by Becker and Wu (2007). Unfortunately, all of these methods have some limitations: (1) The multivariate approaches assume that the meta-analyst “knows” in advance the covariance between the beta weights for the predictors in a study’s model, whereas primary studies rarely give information on covariances between the regression coefficients, and (2) no approach allows for combining standardized regression coefficients that come from regression models that include different covariates. A recent method proposed by Yoneoka and Henmi (2017) extends the multivariate approach of Becker and Wu and allows regression coefficients that come from different regression models to be incorporated in the analysis. However, regression models with more than three covariates are difficult to incorporate, and furthermore, this method requires all primary studies to report the sample size, mean, and variance of all covariates, the regression coefficients and their variances, and the standard error of the regression; reporting all this information in all primary studies, however, seems unlikely. Alternative approaches, if researchers have access to the raw data of all included studies, are the iterative least squares (Hanushek, 1974) and multivariate Bayesian (Novick et al., 1972) approaches, but this situation is exceptional, since meta-analysts of group-comparison experimental design studies can normally count on having only the descriptive and inferential statistics, not the raw data.

Recent methodological research has focused on synthesizing other effect sizes that are functions of the beta weights rather than of the standardized regression coefficients themselves. For example, Peterson and Brown (2005) proposed using an approximation of the bivariate Pearson correlation coefficient based on knowledge of the standardized regression coefficient, and then meta-analyzing the correlation coefficients instead of the beta weights. However, although this approximation is based on a huge amount of real data, it is based only on empirical data, and not on any statistical theory or simulation study. Moreover, recent research (Aloe, 2015) has shown that in many scenarios the Peterson and Brown index does not approximate the bivariate correlation well. In addition, this transformation of betas into correlation coefficients normally responds to a false perception that correlation coefficients provide a more “homogeneous” effect size than do standardized regression coefficients, making them perceived as better effect sizes to combine. However, correlation coefficients can be as heterogeneous as beta weights, because the correlation coefficients from different studies are obtained with different samples, different measures, and so forth.

On the other hand, other authors have proposed the use of other effect sizes from the correlation family, such as the semipartial correlation, instead of the standardized regression coefficient (Aloe & Becker, 2009, 2012). The semipartial correlation coefficient represents the unique effect of the jth predictor on the outcome variable, after partialing out from X_j the effects of all other predictors in the model. The context in which semipartial correlations are used in primary studies is not exactly the same one in which standardized regression coefficients are used. Normally, researchers calculate the (squared) semipartial correlation in primary studies when the main interest is to estimate the amount of unique variance that a predictor variable X explains of Y. In contrast, researchers focus more on standardized regression coefficients when the interest is in the effect that X has on Y, controlling for the presence of other variables that could be related to the outcome variable Y. Furthermore, because they refer to different parameters, standardized regression coefficients and semipartial correlations should not be combined in a single analysis. Also note that semipartial correlations depend on what predictors are partialed out of the focal predictor, X, which means that the semipartial correlations reported from various studies are not necessarily immediately comparable, which is the same problem encountered with combining beta weights.

Several approaches to pooling standardized regression coefficients are statistically correct. For instance, it is possible to perform meta-analyses of standardized regression coefficients (of the focal predictor) that arise from the same regression model (i.e., doing meta-analysis separately for the slopes from each model defined by a set of included predictors). A second strategy was suggested by Aloe and Becker (2012), consisting of coding an indicator or dummy variable that indicates the presence or absence of a specific covariate or a set of covariates in the regression model. By coding the presence of a specific covariate/set of covariates as 0 and the absence as 1, and then performing a meta-regression with these indicator variables as predictors, the estimated intercept can be interpreted as the combined effect size when there are no covariates, whereas the regression coefficient(s) of the indicator variable(s) would represent the expected change in the combined effect size when the specific covariate(s) is/are included in the regression model.

Although these approaches are feasible, the present study aims to propose a new approach that starts from the following idea: We can use the common information shared by the standardized regression coefficients from different models to improve the precision of a focal pooled beta-weight estimate. By “common information shared,” we refer to the underlying correlation coefficients: Population standardized regression coefficients from different regression models can be written in terms of population correlation coefficients. Observed regression weights therefore give information on the underlying correlation coefficients, and therefore also on the other population regression weights, even if they stem from other regression models. Using the new approach, we can get estimates of the mean underlying correlation coefficients, but also of the standardized regression weights of interest. This is a first advantage of the approach: The meta-analyst can decide which effect size to report, on the basis of the specific goal of the meta-analysis. This could, for instance, be the bivariate correlation between X₁ and Y, but also the standardized weight of X₁ after accounting for X₂, or the weight of X₁ after accounting for X₂ and X₃. Another advantage is that we expect to get more accurate estimates because more information will be used for estimating a focal combined standardized regression coefficient, which is highly important, since a central tenet in the discipline of meta-analysis is to increase the precision of the effect size estimates through the accumulation of evidence. A third advantage of the proposed method is that it does not require that the primary studies report correlation coefficients: Only the standardized regression coefficients and their standard errors are used as input. Because the approach is based on the relation between the regression weights and the underlying correlation coefficients, we have named it the concealed correlations meta-analysis approach. This approach is explained in greater detail in the following section.

The concealed correlations meta-analysis approach

Let Y, X₁, …, X_j be random variables, where Y represents a dependent variable and X₁, …, X_j represent the j predictor variables. Let us consider the example of a researcher who wants to calculate the pooled effect that a predictor variable X₁ has on an outcome Y—that is, a combined estimate of the standardized regression weight β₁—and suppose j = 3. The investigator finds 40 primary studies that used regression analysis to explore the effect of X₁ on Y. Ten out of the 40 studies explored the model Y_i = β₁₍₁₎X_i1 + ε_i (Model 1), where i refers to a study participant and β_j(p) is the standardized regression coefficient for X_j for model p. Ten studies included a covariate in the model, X₂, resulting in the within-study regression model Y_i = β₁₍₂₎X_i1 + β₂₍₂₎X_i2 + ε_i (Model 2). Another ten studies explored Model 3, Y_i = β₁₍₃₎X_i1 + β₃₍₃₎X_i3 + ε_i, and the last ten of the studies included the predictor and the other two covariates, resulting in Model 4: Y_i = β₁₍₄₎X_i1 + β₂₍₄₎X_i2 + β₃₍₄₎X_i3 + ε_i. The error term, represented as ε_i, follows a normal distribution, ε_i ~ $N\left(0,{\sigma}_i^2\right).$ The ordinary least squares estimates of the standardized regression coefficients of X₁ for the four models, b₁₍₁₎, b₁₍₂₎ , b₁₍₃₎, and b₁₍₄₎, are calculated with Eqs. 1–4, respectively:

$$ {b}_{1(1)}={r}_{Y1}, $$

(1)

$$ {b}_{1(2)}=\frac{r_{Y1}-{r}_{Y2}{r}_{12}}{1-{r}_{12}^2}, $$

(2)

$$ {b}_{1(3)}=\frac{r_{Y1}-{r}_{Y3}{r}_{13}}{1-{r}_{13}^2}, $$

(3)

$$ {b}_{1(4)}=\frac{\left(1-{r}_{23}^2\right){r}_{Y1}+\left({r}_{13}\ {r}_{23}-{r}_{12}\right){r}_{Y2}+\left({r}_{12}{r}_{23}-{r}_{13}\right){r}_{Y3}}{1-{r}_{12}^2-{r}_{13}^2-{r}_{23}^2+2{r}_{12}{r}_{13}{r}_{23}}, $$

(4)

where r_Y1, r_Y2, r_Y3, r₁₂, r₁₃, and r₂₃ are the sample correlations between the variables Y , X₁, X₂, and X₃, respectively. The sample correlations are the estimates of the population correlation ρ_Y1, ρ_Y2, ρ_Y3, ρ₁₂, ρ₁₃, and ρ₂₃, respectively. From these equations, we can see that the estimated standardized regression coefficient b₁ refers to a different parameter when the covariates included in the regression model are not the same.

Our proposal is to estimate the standardized regression coefficients of each of the four models (in this case, one combined estimate for the each of the four parameters: β₁₍₁₎, β₁₍₂₎ , β₁₍₃₎ and β₁₍₄₎), but doing so using a single model based on the data from all studies. The main idea is to take advantage of the common information that these four parameters contain in estimating β₁₍₁₎, β₁₍₂₎, β₁₍₃₎, and β₁₍₄₎. From the example above, we see that the four estimates of the regression weight of X₁ are based on the correlation r_Y1, and therefore contain information about the population parameter ρ_Y1. In the same way, Eqs. 2 and 4 both contain the correlations r_Y2 and r₁₂, which represent the population parameters ρ_Y2 and ρ₁₂, and Eqs. 3 and 4 contain the correlations r_Y3 and r₁₃ (which represent the population parameters ρ_Y3 and ρ₁₃, respectively). If we were to synthesize only the standardized regression coefficients from the studies that used Model 1, we would then ignore the information about parameter ρ_Y1 contained in studies using Models 2, 3, and 4. Similarly, when we synthesize only the standardized regression coefficients from fitting Model 2 [b₁₍₂₎], then we ignore the information about parameter ρ_Y1 contained in the first type of studies, as well as the information about ρ₁₂ and ρ_Y2 contained in the fourth set of studies (Model 4). To take advantage of all common information that the beta weights share, we can first describe the distributions of the observed regression weights b in terms of their underlying correlation coefficients ρ. These distributions change as a function of the regression model. For instance, following the previous example, we can say that b₁₍₁₎ follows an approximately normal distribution with mean β₁₍₁₎ (which is equal to the population correlation ρ_Y1) and variance (1 − ρ_Y1²)²/n_k, where n_k is the sample size of study k. In the case of b₁₍₂₎, where the covariates X₁ and X₂ are included in the model, b₁₍₂₎ and b₂₍₂₎ follow a multivariate normal distribution described by Becker (1992):

$$ \left[\begin{array}{c}{b}_{1(2)}\\ {}{b}_{2(2)}\end{array}\right]\sim \boldsymbol{N}\left(\left[\begin{array}{c}{\beta}_{1(2)}\\ {}{\beta}_{2(2)}\end{array}\right],A\ {\Sigma}_k{A}^{\prime}\right), $$

where β₁₍₂₎= $\frac{\rho_{Y1}-{\rho}_{Y2}{\rho}_{12}}{1-{\rho}_{12}^2}$ and β₂₍₂₎= $\frac{\rho_{Y2}-{\rho}_{Y1}{\rho}_{12}}{1-{\rho}_{12}^2}$. The formulas for obtaining matrices A and Σ_k can be found in Becker (1992). Similarly, the standardized regression coefficients of Model 3 follow a multivariate normal distribution:

$$ \left[\begin{array}{c}{b}_{1(3)}\\ {}{b}_{3(3)}\end{array}\right]\sim \boldsymbol{N}\left(\left[\begin{array}{c}{\beta}_{1(3)}\\ {}{\beta}_{3(3)}\end{array}\right],A\ {\Sigma}_k{A}^{\prime}\right), $$

where β₁₍₃₎= $\frac{\rho_{Y1}-{\rho}_{Y3}{\rho}_{13}}{1-{\rho}_{13}^2}$ and β₃₍₃₎= $\frac{\rho_{Y3}-{\rho}_{Y1}{\rho}_{13}}{1-{\rho}_{13}^2}$. Finally, the beta weights of Model 4 are distributed according the following multivariate normal distribution:

$$ \left[\begin{array}{c}{b}_{1(4)}\\ {}{b}_{2(4)}\\ {}{b}_{3(4)}\end{array}\right]\sim \boldsymbol{N}\left(\left[\begin{array}{c}{\beta}_{1(4)}\\ {}{\beta}_{2(4)}\\ {}{\beta}_{3(4)}\end{array}\right],A\ {\Sigma}_k{A}^{\prime}\right), $$

where

$$ {\beta}_{1(4)}=\frac{\left(1-{\rho}_{23}^2\right){\rho}_{Y1}+\left({\rho}_{13}\ {\rho}_{23}-{\rho}_{12}\right){\rho}_{Y2}+\left({\rho}_{12}{\rho}_{23}-{\rho}_{13}\right){\rho}_{Y3}}{1-{\rho}_{12}^2-{\rho}_{13}^2-{\rho}_{23}^2+2{\rho}_{12}{\rho}_{13}{\rho}_{23}}; $$

(5)

$$ {\beta}_{2(4)}=\frac{\left(1-{\rho}_{13}^2\right){\rho}_{Y2}+\left({\rho}_{13}\ {\rho}_{23}-{\rho}_{12}\right){\rho}_{Y1}+\left({\rho}_{12}{\rho}_{23}-{\rho}_{13}\right){\rho}_{Y3}}{1-{\rho}_{12}^2-{\rho}_{13}^2-{\rho}_{23}^2+2{\rho}_{12}{\rho}_{13}{\rho}_{23}}, $$

(6)

and

$$ {\beta}_{3(4)}=\frac{\left(1-{\rho}_{12}^2\right){\rho}_{Y3}+\left({\rho}_{12}\ {\rho}_{13}-{\rho}_{23}\right){\rho}_{Y2}+\left({\rho}_{12}{\rho}_{23}-{\rho}_{13}\right){\rho}_{Y1}}{1-{\rho}_{12}^2-{\rho}_{13}^2-{\rho}_{23}^2+2{\rho}_{12}{\rho}_{13}{\rho}_{23}}. $$

(7)

Once we have defined the distribution of the standardized regression coefficients existing in our data and described the expected means in terms of population correlation coefficients, we can proceed to estimate the parameters of interest more precisely thanks to these known relationships. The parameter of interest can be either a standardized regression coefficient of the focal predictor (i.e., β₁₍₁₎, β₁₍₂₎, β₁₍₃₎, or β₁₍₄₎) or the correlation coefficients (i.e., ρ_Y1, ρ_Y2, ρ_Y3, ρ₁₂, ρ₁₃, ρ₂₃).

The estimation of the population standardized regression coefficients and correlation coefficients can be done with any program or procedure that allows the user to constrain the relationships between the parameters that need to be estimated (namely the βs and the ρs). For example, one option is to carry out the estimation using software for Bayesian data analysis, for instance the WinBUGS software (Lunn, Thomas, Best, & Spiegelhalter, 2000). Another option is to use the PROC NLMIXED procedure in SAS, which is what we used in the present study. It is also possible to fit a random-effect model rather than a fixed-effect model, allowing each study to estimate its own population correlation coefficient ρ_Y1. The variation between population correlation coefficients is reflected in the estimated between-study variance (τ²), and its correct estimation is highly relevant, because it is incorporated in the random-effect weights and therefore has a direct influence on the estimation of the combined effect size. An SAS syntax for fitting the parameters of fixed- and random-effect models is given in Appendix A.

Although this last approach makes sense conceptually, given the complexity of its implementation it is worth assessing empirically how well it works and comparing its functioning with alternative approaches. The purpose of this study was therefore to compare the estimates obtained by three different approaches. These approaches include (1) the concealed correlations meta-analysis approach, (2) the approach in which the standardized regression coefficients are combined separately for each type of model, and (3) Aloe and Becker’s (2012) approach using a single model that includes a categorical variable distinguishing to which of the four types of models a standardized regression coefficient belongs. We expected that the estimates obtained using the concealed correlations meta-analysis method would be more accurate, since we were using more information than in the second approach and were explicitly modeling the relation between the regression weights, as apposed to the third approach.

Method

A simulation study was performed in order to evaluate the parameter recovery of the new concealed correlations meta-analysis approach and to compare it to both an approach involving separate meta-analyses for model type and Aloe and Becker’s (2012) single regression model approach. Next, we compared these approaches using a real meta-analytic dataset.

Data generation

To evaluate the performance of these approaches, meta-analytic datasets were generated under different scenarios, using a fixed- or random-effect model to generate and analyze the data, and varying the total number of studies (K = 10, 20, and 40) within the meta-analysis, the sample size of primary studies (n = 50 and 200), and the population correlation matrix from which data were generated. In a first step, raw scores for Y, X₁, X₂, and X₃ were randomly drawn from a multivariate normal distribution for each study. In the fixed-effect case, the raw data were drawn from a multivariate normal distribution with mean (0, 0, 0, 0) and a correlation matrix of either $\left[\begin{array}{c}\begin{array}{ccc}1& .5& .5\ \\ {}.5& 1& .25\ \\ {}.5& .25& 1\ \end{array}\begin{array}{c}.5\\ {}.25\\ {}.25\end{array}\\ {}.5\kern0.5em .25\kern0.5em \begin{array}{cc}.25& 1\end{array}\end{array}\right]$ or $\left[\begin{array}{c}\begin{array}{ccc}1& .25& .25\ \\ {}.25& 1& .25\ \\ {}.25& .25& 1\ \end{array}\begin{array}{c}.25\\ {}.25\\ {}.25\end{array}\\ {}.25\kern0.5em \begin{array}{cc}.25& \begin{array}{cc}.25& 1\end{array}\end{array}\end{array}\right]$. When a random-effect model was considered, random variation was only introduced in the correlation between the criterion variable Y and the covariate of interest X₁ (ρ_Y1). Random deviations of the study-specific correlations from the overall correlation were taken from a uniform distribution between the values – .20 and .20, so the per-study ρ_Y1 ranged between either .30 and .70 or .05 and .45.

In a second step, each study’s dataset was analyzed using the relevant regression model, and the standardized regression weights and corresponding standard errors were saved. As is summarized in Table 1, for a subset of the studies’ data, the first model was used, in which only one variable was included (X₁). For other studies, we fitted the second model, in which Y was regressed on two variables (X₁ and X₂). For a third set of studies, Y was regressed on X₁ and X₃, and in the final set of studies, we fitted a model with three predictor variables (X₁, X₂, and X₃). The expected standardized regression weights were computed from the correlation matrices using the formulas given by Cooley and Lohnes (1971).

Table 1 Overview of the regression models fitted in the studies

The analysis used afterward matched the way in which the data were generated, meaning that if the data were generated using a fixed-effect model, they were after analyzed using a fixed-effect approach, and the same applied for random effects. Thus, in total there were 2 × 2 × 2 × 3 conditions (fixed vs. random, three numbers of studies, two numbers of sample sizes, and two population correlation matrices). For each of these 24 conditions, 3,000 datasets were simulated. Table 1 describes the models that were used in these analyses.

Data analysis

Once the standardized regression coefficients were obtained, we analyzed them according to the three different approaches. For the concealed correlations meta-analysis approach, analyses were done using the PROC NLMIXED procedure in SAS (see Appendix A for the code). We followed the proposal of Kalaian and Raudenbush (1996), to use Cholesky factorization to make the transformed coefficients within studies independent before analyzing them in a meta-analysis. In this way, the standardized regression coefficients belonging to the same study become independent (and their variance became 1), and therefore we could specify univariate distributions (instead of a multivariate one, as proposed by Becker, 1992) for the beta weights, simplifying the estimation procedure. Kalaian and Raudenbush (1996) showed that this transformation does not affect the model parameters of interest in a meta-analysis (i.e., the mean effect size, the moderator effects, or the between-study covariance matrix).

For the approach in which the standardized regression coefficients were combined separately for each model, classical meta-analytic techniques were applied in order to obtain precision-weighted average standardized regression coefficients (Hedges & Vevea, 1998). Meta-analyses in this approach were performed using PROC MIXED from SAS, using restricted maximum likelihood estimation when a random-effect model was fitted (see Appendix B for the code). From now on, this approach will be called the “separate meta-analysis approach.”

For the last approach, which will be called “regression meta-analysis,” the regression coefficients of the focal predictor were regressed on an indicator variable with four categories, which was created with values of 1 through 4 representing Models 1 through 4, respectively. By dropping the intercept from this model, we got estimates for the regression weight of the focal predictor for the four types of models (i.e., b₁₍₁₎, b₁₍₂₎, b₁₍₃₎, b₁₍₄₎). This model specification is very similar to the proposal of Aloe and Becker (2012), which consists of using a meta-regression with two dummy variables that represent the presence (0) or absence (1) of the covariates X₂ and X₃ in the model. The main difference between the method applied in this study and the original proposal of Aloe and Becker (2012) is that in the latter, the intercept of the meta-regression would be interpreted as the combined effect size corrected for the effect of covariates X₂ and X₃, whereas the estimated coefficients of the dummy variables would be interpreted as the units of change in the effect size when a certain variable is excluded from the regression model. On the basis of these coefficients, we can derive those we would expect for the four types of models (b₁₍₁₎, b₁₍₂₎, b₁₍₃₎, b₁₍₄₎). If the effect of the absence of both X₂ and X₃ were exactly equal to the sum of the effect of the absence of X₂ and the effect of the absence of X₃, both models would, however, be equivalent.

Evaluation of performance

For each condition, the fixed-effect and (co)variance parameter estimates, as well as the standard error estimates from the three approaches, were summarized across the 3,000 iterations. Several aspects were considered when evaluating the estimation of the aggregated standardized regression coefficients: the bias, relative bias (RB), precision, and mean squared error (MSE) of the estimates. To evaluate the accuracy of the estimation of the fixed effects, bias was approximated by calculating the average of the deviations of the estimated value from the true value. The precision of the standardized regression coefficients was evaluated by looking at the standard deviations of the estimates: the lower the standard deviation, the higher the precision of the estimate. Finally, the MSE was approximated by calculating the average squared deviation of the estimates from the true values. For evaluating the estimation of the underlying correlation coefficients (only obtained through the new approach), the bias and RB were also calculated, but using the median instead of the average, because we obtained a few outlying correlation coefficient estimates that had a great influence on the average but not on the median estimates.

For evaluating the fixed-effect parameters’ standard error estimates, we estimated the bias by subtracting the standard deviation of the fixed-parameter estimates from the median estimated standard error value. The median of the standard errors was preferred over the mean because these parameters tend to have skewed distributions (the same procedure had been used in other simulation studies for evaluating standard error estimates, as in Van den Noortgate, López-López, Marín-Martínez, & Sánchez-Meca, 2014).

Afterward, the RB of the estimated beta weights, their standard errors, and the correlation coefficients were calculated by dividing the estimated bias by the true value of the parameter (or, for the standard errors, by dividing the estimated bias by the standard deviation of the fixed parameter estimates). For evaluating the RB values, we followed the cutoffs proposed by Hoogland and Boomsma (1998): An RB is moderate but acceptable if its value is between 5% and 10%, whereas an RB is considered unacceptable if its value is above 10%.¹

Finally, analyses of variance (ANOVAs) were conducted to detect the extent to which simulation design factors were related to the precision of the estimates (i.e., the standard deviation of fixed parameter estimates). From these analyses, we looked at the value of eta-squared to know which simulation conditions were more relevant.

Results

Bias and relative parameter bias

An important first result is that the standardized regression coefficients were always estimated without relevant bias, independently of condition or procedure (Table 2).

Table 2 Average estimates of the pooled standardized regression coefficients

Regarding the estimation of the correlation coefficients (Table 3), we found that practically all median correlations were close to the true values when the sample size of the primary studies was 200. When the sample size of the primary studies was smaller (n = 50), the correlations between the criterion and the covariates were estimated without bias, but some correlations between covariates were overestimated when a fixed-effect model was fitted and the number of studies was small (K=10), and when a random-effect model was fitted and the number of studies was large (K=20 or 40).

Table 3 Median estimates of the underlying correlation coefficients

In the following section the precision and MSE of the fixed parameter estimates will be discussed, as well as the estimation of the standard errors. We start with the condition in which a fixed-effect model was used to generate and analyze the data, and continue with the case in which the correlation between X₁ and Y was generated with a random component, and therefore a random-effect model was used.

MSE and relative standard error bias

Fixed-effect model estimates

When a fixed-effect model was used, the pooled beta weights for each model were estimated more precisely through the concealed correlations meta-analysis method, since the standard deviation of the estimates was lower (mean = .039) than the standard deviation of the estimates obtained when meta-analyses were performed separately or when a regression model was fitted (both with mean = .044). In addition, the MSE was lower when the new approach was applied (Table 4). However, an ANOVA performed on the true standard deviations showed stronger effects of the number of studies included in the meta-analysis (η²= .32) and the sample size (η²= .51) than of the method used (η²= .01). As expected, the more studies and the larger their sample sizes, the higher the precision of the pooled beta weights, and hence the smaller the standard deviation.

Table 4 Mean squared errors

With respect to the RB of the standard error estimates, we found no substantial RB when the correlations between the criterion variable and the predictors were small (i.e., .25), and therefore in Table 5 only the results of the condition in which the correlations between criterion and predictor variables were large (i.e., .50) are shown. In this condition, no large differences between the approaches were observed: All procedures substantially overestimated the standard error of b₁₍₁₎. This overestimation was likely due to the fact that b₁₍₁₎ equals the correlation coefficient between the criterion and the predictor variable X₁ (r_Y1), and the variance of correlation coefficient estimates depends on the correlation’s value, which can lead to biased estimates of their variances and standard errors (Meng, Rosenthal, & Rubin, 1992). Therefore, we ran another simulation (with just 1,000 iterations per condition) in which the value of b₁₍₁₎ was transformed to a Fisher’s Z score before performing the three approaches (see the formulas on p. 231 of Borenstein, 1994). These results are shown at the bottom of Table 5: The standard error of the combined b₁₍₁₎ increased, and the relative standard error bias was below 10% for all three approaches, meaning that the initial overestimation of the standard error of b₁₍₁₎ was likely related to the bias in the estimation of the variances of the correlation coefficients. Also in this case, in which b₁₍₁₎ was transformed, the concealed correlations meta-analysis approach led to smaller estimates of the mean standard deviation of the mean estimate. Furthermore, all standardized regression coefficients (i.e., b₁₍₁₎, b₁₍₂₎, b₁₍₃₎, and b₁₍₄₎) were accurately estimated with the new method, despite the use of ${Z}_{b_{1(1)}}$ instead of b₁₍₁₎ in the analyses. Finally, the relative standard error biases of the rest of the pooled beta weights were within the acceptable thresholds (within ± 8%) in all conditions and with all methods.

Table 5 Standard deviations of the beta weights estimates and relative bias obtained when a fixed-effect model was fitted and when the correlations between the dependent variable and the covariates were set to .50

Random-effect model estimates

When a random-effect model was applied, the new approach also showed a smaller standard deviation (.056) than did separate meta-analyses (.074) or the regression meta-analysis approach (.073), reflecting a gain in precision. The MSE was also smaller when the new approach was used (Table 4). This effect was more pronounced than when we used a fixed-effect model. The one-way ANOVA indeed showed a substantial effect of the approach used on the true standard error estimates (η² = .12), in addition to strong effects of the number of studies within the meta-analysis (η² = .63) and of the sample size (η² = .15). Just as when a fixed-effect model was fitted, the larger the sample size and the number of studies included in the meta-analysis, the smaller the standard deviation.

Regarding the RB of the standard error estimates, the separate meta-analysis approach exhibited relative standard error bias above the recommended cutoffs when the number of studies within the meta-analysis was small (K = 10), whereas all relative standard error biases obtained with the regression meta-analysis approach were within the recommended thresholds (see Table 6). The concealed-correlations meta-analysis approach yielded underestimated standard errors when the sample size and the number of studies were both small and when the sample size was large but only 10 or 20 studies were included in the meta-analysis. We also performed an additional simulation study under a random-effect model (1,000 iterations per condition) to verify whether the transformation of b₁₍₁₎ to Fisher’s Z scores before running the analysis of the three approaches had reduced the amount of RB, but the results remained similar. Therefore, the Fisher-transformed results are not included in the table.

Table 6 Standard deviations of the beta weights estimates and relative bias obtained when a random-effect model was fitted and when the correlations between the dependent variable and the covariates were set to .50

Real dataset

To illustrate the concealed correlations meta-analysis approach, we used part of the data from the meta-analysis by Yang, Aloe, and Feeley (2014). In this meta-analysis, the authors aimed to assess the overall effects of the risk information seeking and processing (RISP) model. Specifically, they were using meta-analysis to explore the relationships among the variables within the RISP model, including information insufficiency, risk judgment, worry, informational subjective norms, risk information seeking, systematic processing, and so forth. For our illustration, we used a subset of ten studies. Because the primary studies reported correlation matrices, we computed the standardized regression coefficients, their standard errors, and the covariance between the beta weights using the formulas of Cooley and Lohnes (1971).

The datasets from the primary studies were set to one of four models, listed in Table 7. The focal predictor or the effect we aimed to synthesize in this example was the standardized regression coefficient for informational subjective norms (ISN) as a predictor of risk information seeking (RIS). The predictors in each of the four models were as follows:

• Model 1: RIS = β₁ · ISN,

• Model 2: RIS = β₁ · ISN + β₂ · Current Knowledge (CK),

• Model 3: RIS = β₁ · ISN + β₃ · Perceived Information Gathering Capacity (PIGC),

• Model 4: RIS = β₁ · ISN + β₂ · CK + β₃ · PIGC,

Table 7 Summary of the studies used in the example using real data

where ISN, CK, and PIGC constituted the predictor variables X₁, X₂, and X₃ respectively. A random-effect meta-analytic model was fitted first. As is shown in Table 8, whereas the separate meta-analysis and the regression meta-analysis approach led to similar estimates of the combined standardized regression coefficients, the estimates obtained through the concealed correlations meta-analysis approach differed. In general, standard errors were smaller when the concealed correlations meta-analysis approach was applied, except for the standard error for b₁₍₁₎, which was lower when the studies were combined separately. When a fixed-effect model was fitted (bottom of Table 8), the results were more in line with the three main results extracted from the simulation study. First, the combined standardized regression coefficients were similar across approaches. Second, the standard errors were lower when all studies were analyzed using the concealed correlations meta-analysis approach, reflecting again the gain in precision. Third, when the standardized regression coefficients from Model 1 (b₁₍₁₎) were transformed into Fisher’s Z scores, the standard errors obtained through the three approaches increased, but the estimate obtained through the concealed correlations meta-analysis approach remained the most precise. The conclusion of this meta-analysis is that ISN significantly contributes to the explanation of RIS, and this contribution is larger if the current-knowledge variable (X₂) is included in the regression model, as can be deduced from Models 2 and 4.

Table 8 Standardized regression coefficient estimates by model and approach when both random- and fixed-effect models were fitted

Regarding estimation of the underlying correlation coefficients (Table 9), we saw that when a fixed-effect model was fitted, some estimates were out of range (below - 1 or above 1), and therefore were constrained to – 1 or 1. When a random-effect model was fitted, the estimates were within the acceptable range, but the sign of the correlation between the criterion variable Y and the predictor variable X₂ was negative, whereas in the real correlation matrices we see that none of the r_Y2 estimates were negative.

Table 9 Estimated correlation coefficients

Discussion

In this article we have described, illustrated, and provided SAS code for a new technique for combining standardized regression coefficients in the context of meta-analysis. This new approach, called concealed correlations meta-analysis, not only allows researchers to use standardized regression coefficients coming from studies that include different sets of up to j = 3 covariates, but it also improves the precision of the overall focal regression weight estimates, and increasing the precision of effect size estimates is one of the main goals in the meta-analytic discipline.

From the results of the simulation study, we can extract some general conclusions: (1) The estimation of the combined beta weights for each model is equally unbiased with all procedures, but (2) the estimates of the beta weights using the concealed correlations meta-analysis approach are more precise than the estimates from doing either separate meta-analysis for each type of model or regression analyses, since the standard deviation and MSE are smaller. These findings corroborate our initial hypothesis: By using information from the beta weights of different kinds of models, the precision in the estimates of the combined standardized regression coefficients increases. However, it should be noted that the proposed approach slightly underestimates the standard errors when a random-effect model is fitted, and overestimates the standard errors when a fixed-effect model is fitted under some circumstances. In the case of the fixed-effect model, the overestimations of the standard errors are similar in the three procedures and are corrected by transforming the standardized regression coefficients of Model 1 into Fisher’s Z scores. Therefore, the transformation of the beta weights of simple regression models into Fisher’s Z scores is highly recommended, independent of the approach used, because the standard error estimates are more accurate. In the case of the random-effect model, the new approach underestimates the standard errors of some parameters, especially when the number of participants within the primary studies is large, whereas the estimates obtained through the regression meta-analysis approach exhibit RB within the recommended thresholds. When the number of participants is 50, the relative standard error bias obtained with the concealed correlations meta-analysis approach is close to the limit of what we can consider acceptable. Thus, although the proposed approach does not work perfectly in this condition (n = 50), it does not seem to work substantially worse than the other two approaches, and it has the additional advantage of estimating the standardized regression coefficients more precisely. An additional observation is that when a random-effect model was fitted, the RB of the beta weights belonging to Model 1 did not decrease with the transformation into Fisher’s Z scores.

Regarding the real-dataset analysis, a number of results are worth discussing. When a random-effect model was fitted, the standard error of the combined effect size was lower when a separate meta-analysis approach was used, whereas we would have expected the concealed correlations meta-analysis approach to lead to a lower standard error estimate. A probably explanation is that the estimated between-study variance for Model 1 was zero under the separate meta-analysis approach, although some variation between studies was found under Models 2, 3, and 4. If we combined the data for all models together, as we did in both the concealed correlations meta-analysis and regression meta-analysis approaches, there was just one estimate of the between-study variance for ρ_Y1 that was based on all studies. Combining all data gives evidence for between-study variance in ρ_Y1. Because the standard error of r_Y1 takes into account the additional uncertainty of the population value due to the systematic between-study variance, the standard error was larger. Differences in the estimates of the between-studies variance can also explain why the fixed-effect estimates for the separate and combined approaches differed quite substantially. This was confirmed by fitting a fixed-effect model, since the fixed-effect estimates were more similar for the three approaches because the weights used to calculate the combined beta weights were the same.

An important advantage when using the concealed correlations meta-analysis approach is that the researcher gets not only more precise estimates of the standardized regression coefficients, but also an estimation of the underlying correlations between the variables. The researcher can then use the effect size that is more suitable to the specific goal of the meta-analysis. In the simulation study, we found that all correlation coefficients were estimated accurately across all conditions when the sample size of the primary studies was 200. However, when the number of studies was ten and the sample size of the primary studies was small, and especially under a random-effect model, in some conditions the correlations between covariates were on average slightly overestimated. Furthermore, when the average correlations were calculated, we found that some of the correlations between the predictor variables were out of their theoretical range. Some of these estimation problems were reflected in the real-dataset example, in which all the correlations between covariates were out of range. Besides the expected underperformance due to the small number of studies, there might have been other reasons why the estimates were not accurate. For instance, in this study we considered a scenario in which only the parameter ρ_Y1 varied over studies, and we assumed that all the other underlying correlations did not vary across studies. Nevertheless, not accounting for possible between-study variability of correlation coefficients different from the focal one can negatively affect the estimation of correlation coefficients.

The concealed correlations meta-analysis approach can be extended in several ways. A first extension would be the inclusion of moderator variables to explain between-study variance. Another extension would be the use of additional levels of variation, to account for possible dependencies in the data (Van den Noortgate et al., 2014). Finally, whereas in this article we focused on four kinds of models, differing in whether no, one, or two covariates were included, it is possible to extend the approach to situations in which other covariates might be included in some studies, or occasions in which the regression models are not nested. We also would note that in principle, adding a study that does not include the focal predictor but includes other predictors could further (indirectly) improve the estimates of the correlation between the focal predictor and the criterion (or a standardized regression weight of the focal predictor). At the same time, we see a risk: Studies that do not include the focal predictor might systematically differ from studies that include that predictor (e.g., in the kind of population that is studied, the way the criterion variable is defined, or the context). In that situation, including these kinds of studies might induce bias when estimating the effect of the focal predictor.

Although the new approach is promising, it is not free of limitations. First, when there are too many predictors and/or when too many different models appear in different studies, deriving and coding the relation between the standardized regression coefficients and the correlation coefficients can become very cumbersome, which is similar to the problem faced when using the method of Yoneoka and Henmi (2017). If we had a model with four covariates, there would be ten correlation coefficients, and all beta weights would be a function of these ten correlation coefficients, so the formulas would be longer and more complicated than the ones we specified in this article. Therefore, the use of a more generic syntax should be explored in the future. Second, adding additional covariates could also lead to computational problems: If we have more covariates, many more models that can potentially be composed using these covariates, and therefore we would have many more parameters to estimate. When there are too many different models, it is likely that we would not have enough studies available to estimate all the parameters. Third, for getting (unique) estimates of all intercorrelations, the number of different regression coefficients for which we have observed data (eight, in our real example) has to be at least as high as the number of unknown correlation coefficients (six, in the real example). Thus, as the number of different covariates increases among the models, the complexity of executing this approach increases, as well. A fourth limitation is that the approach assumes that the between-study variation in a correlation coefficient is the same for all types of models. Although this seems a reasonable assumption, it is unclear when this assumption might be violated and what consequences it might have for the results. Fifth, although the simulation study showed that the median of the correlation estimates was quite close to the true correlation, the real-data example (which included only ten studies) illustrates that individual estimates can be outside the theoretical range of – 1 to 1. Finally, another limitation is that the simulated data structure was relatively simple, in the sense that only ρ_Y1 could vary over studies, but all other correlations were considered fixed. More complex but more realistic models could be formulated that account for between-study heterogeneity for other correlations, as well. Given that these models include a larger number of parameters, and that some of these are variance parameters for correlations about which only a subpart of the studies would give information, we expect that many more studies would be required in order to accurately and precisely fit these models. More simulation research will be needed to explore the performance of such models.

Appendices

Appendix A

All the codes within this appendix are specific to the example shown in this article.

The data used to perform the analysis with the new concealed correlations meta-analysis approach and its disposition are shown below. The raw data are shown first, and then transformed using Cholesky factorization.

Raw data

data RISP_model;

input study model estimate Stderr D1 D2 D3 D4 D5 D6 D7 D8 sample_size ;

datalines;

1	1	0.404	0.033	1	0	0	0	0	0	0	0	634
2	1	0.391	0.039	1	0	0	0	0	0	0	0	456
3	2	0.651	0.020	0	1	0	0	0	0	0	0	828
3	2	0.090	0.028	0	0	1	0	0	0	0	0	828
4	2	0.441	0.048	0	1	0	0	0	0	0	0	296
4	2	-.002	0.054	0	0	1	0	0	0	0	0	296
5	2	0.635	0.021	0	1	0	0	0	0	0	0	804
5	2	0.083	0.027	0	0	1	0	0	0	0	0	804
6	3	0.308	0.040	0	0	0	1	0	0	0	0	500
6	3	0.105	0.042	0	0	0	0	1	0	0	0	500
7	3	0.294	0.040	0	0	0	1	0	0	0	0	500
7	3	0.028	0.042	0	0	0	0	1	0	0	0	500
8	3	0.539	0.026	0	0	0	1	0	0	0	0	736
8	3	-.091	0.030	0	0	0	0	1	0	0	0	736
9	4	0.560	0.030	0	0	0	0	0	1	0	0	645
9	4	0.041	0.033	0	0	0	0	0	0	1	0	645
9	4	0.082	0.032	0	0	0	0	0	0	0	1	645
10	4	0.758	0.019	0	0	0	0	0	1	0	0	1007
10	4	0.041	0.023	0	0	0	0	0	0	1	0	1007
10	4	-.048	0.024	0	0	0	0	0	0	0	1	1007

;run;

The value 1 in the dummy variables (D1 through D8) indicates which standardized regression coefficients refer to the same population parameter. In our data, eight different types of standardized regression coefficients refer to eight different parameters: b₁₍₁₎, b₁₍₂₎, b₂₍₂₎, b₁₍₃₎, b₃₍₃₎, b₁₍₄₎, b₂₍₄₎, b₃₍₄₎. The number between brackets refers to the type of model to which each beta weight belongs. In the overview of the regression models fitted in the simulation study (Table 1), similar notation is used for the parameters β. The second variable (called Model) also indicates the model to which each standardized regression model belongs.

Transformed data (Cholesky factorization)

data RISP_model_transformed;

input study model estimate D1 D2 D3 D4 D5 D6 D7 D8 Stderr;

datalines;

1	1	12.12	30.01	0	0	0	0	0	0	0	1
2	1	9.83	25.15	0	0	0	0	0	0	0	1
3	2	37.46	0	54.83	19.42	0	0	0	0	0	1
3	2	3.41	0	0	37.86	0	0	0	0	0	1
4	2	9.41	0	21.35	5.48	0	0	0	0	0	1
4	2	-0.04	0	0	18.50	0	0	0	0	0	1
5	2	33.79	0	50.90	17.64	0	0	0	0	0	1
5	2	3.04	0	0	36.51	0	0	0	0	0	1
6	3	8.10	0	0	0	25.08	3.42	0	0	0	1
6	3	2.49	0	0	0	0	23.75	0	0	0	1
7	3	7.21	0	0	0	24.48	-0.52	0	0	0	1
7	3	0.65	0	0	0	0	23.40	0	0	0	1
8	3	22.47	0	0	0	39.88	-10.39	0	0	0	1
8	3	-2.96	0	0	0	0	32.34	0	0	0	1
9	4	21.48	0	0	0	0	0	37.59	11.21	-0.75	1
9	4	1.44	0	0	0	0	0	0	29.91	2.35	1
9	4	2.54	0	0	0	0	0	0	0	30.80	1
10	4	54.17	0	0	0	0	0	0	26.56	31.42	1
10	4	1.22	0	0	0	0	0	0	44.96	13.56	1
10	4	-1.94	0	0	0	0	0	0	0	39.77	1

;run;

The following codes were used to run the concealed correlations meta-analysis approach.

Fixed-effect model

Proc nlmixed data= RISP_model_transformed qpoints= 40 MAXITER= 1000 MAXFUNC= 4000 ;

mean = (D1*RY1) +

(D2* (RY1-RY2*R12)/( 1- (R12** 2 ))) +

(D3*(RY2-RY1*R12)/( 1- (R12** 2 ))) +

(D4*(RY1-RY3*R13)/( 1- (R13** 2 ))) +

(D5*(RY3-RY1*R13)/( 1- (R13** 2 ))) +

(D6*((1-(R23**2))*RY1+(R13*R23-R12)*RY2+(R12*R23-R13)*RY3)/(1-(R12**2)-(R13**2)-(R23**2)+2*R12*R13*R23)) +

(D7*(( 1- (R13** 2 ))*RY2+(R13*R23-R12)*RY1+(R12*R13-R23)*RY3)/( 1- (R12** 2 )-(R13** 2 )-(R23** 2 )+ 2 *R12*R13*R23))+

(D8*(( 1- (R12** 2 ))*RY3+(R12*R13-R23)*RY2+(R12*R23-R13)*RY1)/( 1- (R12** 2 )-(R13** 2 )-(R23** 2 )+ 2 *R12*R13*R23));

model estimate ~ Normal(mean,Stderr** 2 );

estimate 'b1' RY1;

estimate 'b21' (RY1-RY2*R12)/( 1- (R12** 2 ));

estimate 'b22'(RY2-RY1*R12)/( 1- (R12** 2 ));

estimate 'b31' (RY1-RY3*R13)/( 1- (R13** 2 ));

estimate 'b33'(RY3-RY1*R13)/( 1- (R13** 2 ));

estimate 'b41' (( 1- (R23** 2 ))*RY1+((R13*R23-R12)*RY2)+((R12*R23-R13)*RY3))/( 1- (R12** 2 )-(R13** 2 )-(R23** 2 )+( 2 *R12*R13*R23));

estimate 'b42'(( 1- (R13** 2 ))*RY2+((R13*R23-R12)*RY1)+((R12*R13-R23)*RY3))/( 1- (R12** 2 )-(R13** 2 )-(R23** 2 )+( 2 *R12*R13*R23));

estimate 'b43' (( 1- (R12** 2 ))*RY3+((R12*R13-R23)*RY2)+((R12*R23-R13)*RY1))/( 1- (R12** 2 )-(R13** 2 )-(R23** 2 )+( 2 *R12*R13*R23));

parms r12= .1 r13= .1 r23= .1 ;

bounds ry2>- 1 ;

run ;

In the first command, mean, we define the formulas that lead to each of the eight types of standardized regression coefficients. The variables RY1, RY2, RY2, R12, R13, and R23 refers to ρ_Y1, ρ_Y2, ρ_Y3, ρ₁₂, ρ₁₃, and ρ₂₃, respectively. In the model statement, we specify the distribution of each of the beta weights. In the estimate statements, the relationship between the standardized regression coefficients and the underlying correlation coefficients is specified. In the parms statement, we give some initial values, and finally, with the bound statement, we set some boundaries for the estimates.

Random-effect model

proc nlmixed data= RISP_model_transformed qpoints= 40 MAXITER= 1000 MAXFUNC= 4000 ;

mean = (D1*(RY1+u1)) +

(D2* ((RY1+u1)-((RY2)*(R12)))/( 1- ((R12)** 2 ))) +

(D3*((RY2)-(RY1+u1)*(R12))/( 1- ((R12)** 2 ))) +

(D4*((RY1+u1)-(RY3)*(R13))/( 1- ((R13)** 2 ))) +

(D5*((RY3)-(RY1+u1)*(R13))/( 1- ((R13)** 2 ))) +

(D6*(( 1- ((R23)** 2 ))*(RY1+u1)+((R13)*(R23)-(R12))*(RY2)+(R12*R23-R13)*(RY3))/( 1- ((R12)** 2 )-((R13)** 2 )-((R23)** 2 )+ 2 *(R12)*(R13)*(R23))) + (D7*(( 1- ((R13)** 2 ))*(RY2)+((R13)*(R23)-(R12))*(RY1+u1)+((R12)*(R13)-(R23))*(RY3))/( 1- ((R12)** 2 )-((R13)** 2 )-((R23)** 2 )+ 2 *(R12)*(R13)*(R23)))+ (D8*(( 1- ((R12)** 2 ))*(RY3)+((R12)*(R13)-(R23))*(RY2)+((R12)*(R23)-(R13))*(RY1+u1))/( 1- ((R12)** 2 )-((R13)** 2 )-((R23)** 2 )+ 2 *(R12)*(R13)*(R23)));

model estimate ~ Normal(mean,Stderr** 2 );

random u1 ~Normal( 0 ,tau1) subject = study;

estimate 'b1' RY1;

estimate 'b21' (RY1-RY2*R12)/( 1- (R12** 2 ));

estimate 'b22'(RY2-RY1*R12)/( 1- (R12** 2 ));

estimate 'b31' (RY1-RY3*R13)/( 1- (R13** 2 ));

estimate 'b33'(RY3-RY1*R13)/( 1- (R13** 2 ));

estimate 'b41' (( 1- (R23** 2 ))*RY1+((R13*R23-R12)*RY2)+((R12*R23-R13)*RY3))/( 1- (R12** 2 )-(R13** 2 )-(R23** 2 )+( 2 *R12*R13*R23));

estimate 'b42'(( 1- (R13** 2 ))*RY2+((R13*R23-R12)*RY1)+((R12*R13-R23)*RY3))/( 1- (R12** 2 )-(R13** 2 )-(R23** 2 )+( 2 *R12*R13*R23));

estimate 'b43' (( 1- (R12** 2 ))*RY3+((R12*R13-R23)*RY2)+((R12*R23-R13)*RY1))/( 1- (R12** 2 )-(R13** 2 )-(R23** 2 )+( 2 *R12*R13*R23));

parms tau1= .02 r12= .1 r13= .1 r23= .1 ;

bounds r23>- 1 ;

run

The code for the random-effect model has two differences from the previous code. First, in the mean statement a random component (u1) is added only to the correlation between the criterion and the X₁ variables, RY1, while the others correlations are kept fixed. Second, the distribution of this random component is specified in the random statement, and the estimation of its variance (tau1) is the estimate of the between-study variance of RY1.

Appendix B

All the codes within this appendix are specific to the example showed in this article.

For performing the separate meta-analysis approach, the precision was obtained through calculation of the inverse of the variance. Then the code below was used four times, changing each time the content of the where statement of the PROX MIXED procedure in SAS. For combining the standardized regression coefficients belonging to Model 1 (b₁₍₁₎), the following where statement was used:

where model = 1 and D1=1;

For combining the standardized regression coefficients belonging to Model 2 (b₁₍₂₎):

where model = 2 and D2=1;

For combining the beta weights from Model 3 (b₁₍₃₎):

where model = 3 and D4=1;

and finally, for Model 4 (b₁₍₄₎):

where model = 4 and D6=1;

For instance, the following code was used for meta-analyzing the beta weights belonging to Model 1.

Fixed-effect model

proc mixed data= RISP_model;

where model = 1 and D1=1;

class study;

weight prec;

model estimate=/SOLUTION ;

parms 1/hold = (1);

run;

Random-effect model

proc mixed data= RISP_model;

where model = 1 and D1=1;

class study;

weight prec;

model estimate=/SOLUTION ;

Random intercept /SUB=study;

parms .5 1/hold = (2);

run;

Appendix C

All the codes within this appendix are specific to the example showed in this article.

For performing the regression meta-analysis approach (Aloe & Becker, 2012), the following SAS codes were used.

Fixed-effect model

proc mixed data= RISP_model;

where D1=1 or D2=1 or D4=1 OR D6=1;

class model ;

weight prec;

model estimate= model/ noint SOLUTION;

parms 1/ hold = (1);

run;

Random-effect model

proc mixed data= RISP_model;

where D1=1 or D2=1 or D4=1 OR D6=1;

class study model;

weight prec;

model estimate= model /noint SOLUTION ddfm=sat;

Random intercept /SUB=study;

parms .5 1/hold = (2);

run;

Author note

This research was supported by the Research Foundation–Flanders (FWO), through Grant G.0798.15N to the University of Leuven, Belgium. Ariel M. Aloe contribution was partially supported by a grant from the National Science Foundation (DRL-1550169). The opinions expressed are those of the authors and do not represent the views of the FWO or the National Science Foundation. For the simulations, we used equipment at the VSC–Flemish Supercomputer Center, funded by the Hercules Foundation and the Flemish Government, Department EWI.