Abstract
In prevention science and related fields, large meta-analyses are common, and these analyses often involve dependent effect size estimates. Robust variance estimation (RVE) methods provide a way to include all dependent effect sizes in a single meta-regression model, even when the exact form of the dependence is unknown. RVE uses a working model of the dependence structure, but the two currently available working models are limited to each describing a single type of dependence. Drawing on flexible tools from multilevel and multivariate meta-analysis, this paper describes an expanded range of working models, along with accompanying estimation methods, which offer potential benefits in terms of better capturing the types of data structures that occur in practice and, under some circumstances, improving the efficiency of meta-regression estimates. We describe how the methods can be implemented using existing software (the “metafor” and “clubSandwich” packages for R), illustrate the proposed approach in a meta-analysis of randomized trials on the effects of brief alcohol interventions for adolescents and young adults, and report findings from a simulation study evaluating the performance of the new methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Supplementary analysis and replication materials for this study are available on the Open Science Framework at https://osf.io/x8yre/.
Notes
This section describes RVE at a heuristic level. Section S1 of the supplementary materials includes exact mathematical details using matrix notation.
These assumptions define a certain structure for the variance–covariance matrices, \({{\varvec{\Phi}}}_{1},\dots ,{{\varvec{\Phi}}}_{J}\). See Section S2.1 of the supplementary materials for further details.
Section S2.2 of the supplementary materials provides further details on the structure of the variance–covariance matrices, \({{\varvec{\Phi}}}_{1},\dots ,{{\varvec{\Phi}}}_{J}\) under this model.
Section S2.3 of the supplementary materials gives form of the variance–covariance matrices implied by the CHE model.
Section S2.4 of the supplementary materials give the form of the variance–covariance matrices in the SCE model.
More details about the specification of CHE + are given in Section S3.1 of the supplementary materials.
For example, a meta-analysis might include some experiments with multiple treatment conditions, where the effect size estimates represent treatment effects compared to a common control condition. The formula for the covariance between pairs of effect sizes estimated from a common control group can be calculated from information that is usually reported in primary studies (Gleser & Olkin, 2009; Wei & Higgins, 2013), so that there is no need to make arbitrary assumptions about the correlation between sampling errors.
See Supplementary Table S1 for further details.
See Supplementary Table S2 for further details.
Section S4.2 of the supplementary materials provides further details.
Section S5 of the supplementary materials include a full description of the simulation design and results, including graphical depictions of the main findings. Supplementary materials also include the full computer code and complete numerical results.
References
Baudry, C., Tarabulsy, G. M., Atkinson, L., Pearson, J., & St-Pierre, A. (2017). Intervention with adolescent mother–child dyads and cognitive development in early childhood: a meta-analysis. Prevention Science, 18, 116–130. https://doi.org/10.1007/s11121-016-0731-7
Chhin, C. S., Taylor, K. A., & Wei, W. S. (2018). Supporting a culture of replication: an examination of education and special education research grants funded by the Institute of Education Sciences. Educational Researcher, 47, 594–605. https://doi.org/10.3102/0013189X18788047
Conley, C. S., Durlak, J. A., & Kirsch, A. C. (2015). A meta-analysis of universal mental health prevention programs for higher education students. Prevention Science, 16, 487–507. https://doi.org/10.1007/s11121-015-0543-1
Fisher, Z., Tipton, E., & Zhipeng, H. (2017). robumeta: Robust Variance Meta-Regression (2.0) [Computer software]. https://github.com/zackfisher/robumeta
Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The Handbook of Research Synthesis and Meta-Analysis (2nd ed., pp. 357–376). Russell Sage Foundation.
Hedberg, E. C. (2014). ROBUMETA: Stata module to perform robust variance estimation in meta-regression with dependent effect size estimates. Statistical Software Components S457219, Boston College Department of Economics.
Hedges, L. V., & Olkin, I. (1982). Statistical methods for meta-analysis. Academic press.
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65. https://doi.org/10.1002/jrsm.5
Hennessy, E. A., & Tanner-Smith, E. E. (2015). Effectiveness of brief school-based interventions for adolescents: A meta-analysis of alcohol use prevention programs. Prevention Science, 16, 463–474. https://doi.org/10.1007/s11121-014-0512-0
Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. The ANNALS of the American Academy of Political and Social Science, 587, 69–81. https://doi.org/10.1177/0002716202250791
Pustejovsky, J. E. (2020). clubSandwich: Cluster-Robust (Sandwich) Variance Estimators with Small-Sample Corrections (0.4.2) [R package]. https://github.com/jepusto/clubSandwich
Skeen, S., Laurenzi, C. A., Gordon, S. L., Toit, S. du, Tomlinson, M., Dua, T., Fleischmann, A., Kohl, K., Ross, D., Servili, C., Brand, A. S., Dowdall, N., Lund, C., Westhuizen, C. van der, Carvajal-Aguirre, L., Carvalho, C. E. de, & Melendez-Torres, G. J. (2019). Adolescent mental health program components and behavior risk reduction: A meta-analysis. Pediatrics, 144. https://doi.org/10.1542/peds.2018-3488
Tanner-Smith, E. E., & Lipsey, M. W. (2015). Brief Alcohol Interventions for Adolescents and Young Adults: A Systematic Review and Meta-Analysis. Journal of Substance Abuse Treatment, 51, 1–18. https://doi.org/10.1016/j.jsat.2014.09.001
Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 5, 13–30. https://doi.org/10.1002/jrsm.1091
Tanner-Smith, E. E., Tipton, E., & Polanin, J. R. (2016). Handling complex meta-analytic data structures using robust variance estimates: A tutorial in R. Journal of Developmental and Life-Course Criminology, 2, 85–112. https://doi.org/10.1007/s40865-016-0026-5
Tipton, E. (2013). Robust variance estimation in meta-regression with binary dependent effects. Research Synthesis Methods, 4, 169–187. https://doi.org/10.1002/jrsm.1070
Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20, 375–393. https://doi.org/10.1037/met0000011
Tipton, E., & Pustejovsky, J. E. (2015). Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. Journal of Educational and Behavioral Statistics, 40, 604–634. https://doi.org/10.3102/1076998615606099
Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019a). A history of meta-regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10, 161–179. https://doi.org/10.1002/jrsm.1338
Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019b). Current practices in meta-regression in psychology, education, and medicine. Research Synthesis Methods, 10, 180–194. https://doi.org/10.1002/jrsm.1339
Trikalinos, T. A., & Olkin, I. (2012). Meta-analysis of effect sizes reported at multiple time points: A multivariate approach. Clinical Trials, 9, 610–620. https://doi.org/10.1177/1740774512453218
Tyszler, M., Pustejovsky, J. E., & Tipton, E. (2017). REG_SANDWICH: Stata module to compute cluster-robust (sandwich) variance estimators with small-sample corrections for linear regression. In Statistical Software Components. Boston College Department of Economics. https://ideas.repec.org/c/boc/bocode/s458352.html
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016). Methods to estimate the between-study variance and its uncertainty in meta-analysis. Research Synthesis Methods, 7, 55–79. https://doi.org/10.1002/jrsm.1164
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1–48. https://doi.org/10.18637/jss.v036.i03
Wei, Y., & Higgins, J. P. (2013). Estimating within-study covariances in multivariate meta-analysis with multiple outcomes. Statistics in Medicine, 32, 1191–1205. https://doi.org/10.1002/sim.5679
Funding
Partial support for this research was provided by NSF Award 1937633 and IES Award R305B170019 to Elizabeth Tipton.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethics Approval
Not applicable. This study includes a re-analysis of data from a previously published systematic review, based on published summary data.
Conflict of Interest
The authors declare they have no financial interests. James Pustejovsky is the author of the clubSandwich package for R. Elizabeth Tipton is an author of the robumeta packages for R and Stata. Neither author receives compensation from use of these software packages.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Pustejovsky, J.E., Tipton, E. Meta-analysis with Robust Variance Estimation: Expanding the Range of Working Models. Prev Sci 23, 425–438 (2022). https://doi.org/10.1007/s11121-021-01246-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-021-01246-3