Abstract
In metaanalysis, dependent effect sizes are very common. An example is where in one or more studies the effect of an intervention is evaluated on multiple outcome variables for the same sample of participants. In this paper, we evaluate a threelevel metaanalytic model to account for this kind of dependence, extending the simulation results of Van den Noortgate, LópezLópez, MarínMartínez, and SánchezMeca Behavior Research Methods, 45, 576–594 (2013) by allowing for a variation in the number of effect sizes per study, in the betweenstudy variance, in the correlations between pairs of outcomes, and in the sample size of the studies. At the same time, we explore the performance of the approach if the outcomes used in a study can be regarded as a random sample from a population of outcomes. We conclude that although this approach is relatively simple and does not require prior estimates of the sampling covariances between effect sizes, it gives appropriate mean effect size estimates, standard error estimates, and confidence interval coverage proportions in a variety of realistic situations.
Similar content being viewed by others
Notes
Three outlying standardized mean differences ( >2) were not included in the analysis because of their substantial impact on the parameter estimates, especially on the variance estimates. The analysis therefore is based on 584 observed effect sizes from 39 studies.
References
Ahn, S., Ames, A. J., & Myers, N. D. (2012). A review of metaanalyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82, 436–476.
Arends, L. R., Voko, Z., & Stijnen, T. (2003). Combining multiple outcome measures in a metaanalysis: An application. Statistics in Medicine, 22, 1335–1353.
Becker, B. J. (2000). Multivariate metaanalysis. In H. E. A. Tinsley & E. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 499–525). Orlando, FL: Academic Press.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixedeffect and randomeffects models for metaanalysis. Research Synthesis Methods, 1, 97–111.
Cheung, M.W.L. (2013). Modelling dependent effect sizes with threelevel metaanalyses: A structural equation modelling approach. Psychological Methods. Advance online publication.
Cheung, S.F., & Chan, D.K.S. (2014). Metaanalyzing dependent correlations: AnSPSS macro and an R script. Behavior Research, 46, 331–345.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Geeraert, L., Van den Noortgate, W., Grietens, H., & Onghena, P. (2004). The effects of early prevention programs for families with young children at risk for physical child abuse and neglect. A metaanalysis. Child Maltreatment, 9, 277–291.
Gleser, L. J., & Olkin, I. (1994). Stochastically dependent effect sizes. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 339–355). New York: Russell Sage Foundation.
Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107–128.
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation of metaregression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009). A reevaluation of randomeffects metaanalysis. Journal of the Royal Statistical Society, Series A, 172, 137–159.
Hox, J. (2002). Multilevel analysis. Techniques and applications. Mahwah, NJ: Erlbaum.
Ishak, K. J., Platt, R. W., Joseph, L., & Hanley, J. A. (2008). Impact of approximating or ignoring withinstudy covariances in multivariate metaanalyses. Statistics in Medicine, 27, 670–686.
Jackson, D., Riley, R., & White, I. R. (2011). Multivariate metaanalysis: Potential and promise. Statistics in Medicine, 30, 2481–2498.
Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for metaanalysis. Psychological Methods, 1, 227–235.
Konstantopoulos, S. (2011). Fixed effects and variance components estimation in threelevel metaanalysis. Research Synthesis Methods, 2, 61–76.
Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberger, O. (2006). SAS® system for mixed models (2nd ed.). Cary, NC: SAS Institute Inc.
Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 85–91.
Mood, A. M., Graybill, F. A., & Boes, D. C. (1974). Introduction to the theory of statistics. New York: McGrawHill.
Osburn, H. G., & Callender, J. C. (1992). A note on the sampling variance of the mean uncorrected correlation in metaanalysis and validity generalization. Journal of Applied Psychology, 77, 115–122.
Raudenbush, S. W. (2009). Analyzing effect sizes: Random effects models. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and metaanalysis (2nd ed., pp. 295–315). New York: Russell Sage Foundation.
Raudenbush, S. W., Becker, B. J., & Kalaian, H. A. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111–120.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). London: Sage Publications.
Riley, R. D. (2009). Multivariate metaanalysis: The effect of ignoring withinstudy correlation. Journal of the Royal Statistical Society, Series A, 172, 789–811.
RosaAlcázar, A. I., SánchezMeca, J., GómezConesa, A., & MarínMartínez, F. (2008). Psychological treatment of obsessivecompulsive disorder: A metaanalysis. Clinical Psychology Review, 28, 1310–1325.
Scammacca, N., Roberts, G., & Stuebing, K.K. (2013). Metaanalysis with complex research designs: Dealing with dependence from multiple measures and multiple group comparisons. Review of Educational Research. Advance online publication.
Stevens, J. R., & Taylor, A. M. (2009). Hierarchical dependence in metaanalysis. Journal of Educational and Behavioral Statistics, 34, 46–73.
Van den Noortgate, W., LópezLópez, J. A., MarínMartínez, F., & SánchezMeca, J. (2013). Three level metaanalyses of dependent effect sizes. Behavior Research Methods, 45, 576–594.
Van den Noortgate, W., & Onghena, P. (2003). Multilevel metaanalysis: A comparison with traditional metaanalytical procedures. Educational and Psychological Measurement, 63, 765–790.
Van den Noortgate, W., & Onghena, P. (2005). Parametric and nonparametric bootstrap methods for metaanalysis. Behavior Research Methods, 37, 11–22.
van Houwelingen, H. C., Arends, L. R., & Stijnen, T. (2002). Advanced methods in metaanalysis: Multivariate approach and meta regression. Statistics in Medicine, 21, 589–624.
Acknowledgments
The simulation study was performed on the High Performance Cluster of the Flemish Supercomputer Centre.
Author information
Authors and Affiliations
Corresponding author
Appendices
Appendix A: Proof that the variance at the study level is the covariance
Suppose that two observed effect sizes, d _{ jk } and d _{ j’k }, stem from the same study, study k. According to Equation 6, d _{ jk } = γ _{00} + u _{0k } + v _{ jk } + r _{ jk } and d _{ j ' k } = γ _{00} + u _{0k } + v _{ j ' k } + r _{ j ' k }
Therefore,
Because γ _{00} is a constant, and adding a constant to one or both random variables does not affect their covariance, this covariance equals:
The covariance between two linear combinations is described by Mood, Graybill and Boes (1974, p. 179):
Hence:
Because in a multilevel model two residuals at the same level are assumed to be independent, as are residuals at two different levels,
Appendix B: SAS Codes for the Example
Data set format
For the multilevel analyses, the data set should contain one row for each observed effect size. For our example, we prepared such a data set, called abuse, with the following variables (the dataset is available upon request from the first author):

study: a study indicator with values from 1 to 39,

outcome: an outcome indicator with values from 1 to 587,

ES: the effect size expressed as bias corrected standardized mean differences,

W: the inverse of the estimated sampling variance for each observed effect size, and

X: an indicator variable for the two groups of outcomes (1 refers to the outcomes directly related to child abuse and neglect, 2 to outcomes related to risk factors).
The first ten rows are given below:
Twolevel metaanalysis
For the random effects twolevel analysis, the following code is run:
The Proc Mixedcommand calls the mixed procedure for multilevel or linear mixed models. The data set is defined, and we ask for using the restricted maximum likelihood (REML) estimation procedure.
The Classstatement is used to define the categorical variables of our model, in our case the study and outcome indicators. In the Modelstatement we define the model: the dependent variable (ES) on the left side of the equality sign, the predictor or moderator variables on the right. An intercept is included by default. In this random effects model, there are no moderator variables. The Solutionoption requests the parameter regression coefficient estimates and tests in the output. The ddfm=Satterthwaiteoption performs a general Satterthwaite approximation for the denominator degrees of freedom for the tests of the regression coefficients.
We use W, the inverse of the sampling variance, to weight the observed effect sizes in the analysis. However, weights that are used in the multilevel analysis will not only be based on the sampling variance, but also will automatically account for the estimated population variance(s) defined further. More specifically, the weights are equal to the inverse of the sum of the different sources of variance. The Randomstatement specifies that the intercept varies randomly over outcomes. In the Parmsstatement, we give starting values for the population variance of this intercept as well as for the residual variance. We use the Holdoption to fix the second parameter to the starting value of 1. In this way and by using the inverse of the sampling variance as weights, the levelone variance is automatically fixed at the sampling variances that we defined. Using a more realistic starting value for the first parameter (e.g., the estimate from a previous analysis) can speed up the estimation. Finally, we close the code using the Runstatement, and we submit the code.
For the mixed effects twolevel analysis, the code is adapted as follows:
First, the Xvariable is defined as a categorical variable by means of the Classstatement. Second, in the Modelstatement we include the Xvariable as a predictor. To get an estimate of the mean effect for each level of the Xvariable, we drop the intercept, by using noint as an option in the Modelstatement. Finally, to estimate and test the difference between both groups of outcomes in the expected effect, we use the Estimatestatement. The differenceparameter is labeled ‘group’, and is defined by a contrast with weights 1 and 1.1
Threelevel metaanalysis
Because in the threelevel random effects model we assume that the intercept might not simply vary over the 587 outcomes, but that there might be systematic differences between studies due to covariation between effect sizes from the same study, we include a second Randomstatement:
We now have three sources of variance: between studies, between outcomes within studies, and sampling variance. In the Parmsstatement, we define starting values for the three variances, and constrain the last one to 1.
The code for the random effects model is extended to the code for a mixed effects model in much the same way as for the twolevel models.
Rights and permissions
About this article
Cite this article
Van den Noortgate, W., LópezLópez, J.A., MarínMartínez, F. et al. Metaanalysis of multiple outcomes: a multilevel approach. Behav Res 47, 1274–1294 (2015). https://doi.org/10.3758/s1342801405272
Published:
Issue Date:
DOI: https://doi.org/10.3758/s1342801405272