Advertisement

Behavior Research Methods

, Volume 46, Issue 2, pp 331–345 | Cite as

Meta-analyzing dependent correlations: An SPSS macro and an R script

  • Shu Fai Cheung
  • Darius K.-S. Chan
Article

Abstract

The presence of dependent correlation is a common problem in meta-analysis. Cheung and Chan (2004, 2008) have shown that samplewise-adjusted procedures perform better than the more commonly adopted simple within-sample mean procedures. However, samplewise-adjusted procedures have rarely been applied in meta-analytic reviews, probably due to the lack of suitable ready-to-use programs. In this article, we compare the samplewise-adjusted procedures with existing procedures to handle dependent effect sizes, and present the samplewise-adjusted procedures in a way that will make them more accessible to researchers conducting meta-analysis. We also introduce two tools, an SPSS macro and an R script, that researchers can apply to their meta-analyses; these tools are compatible with existing meta-analysis software packages.

Keywords

Meta-analysis Dependent effect sizes 

Notes

Author note

This research was supported by Grant No. MYRG047(Y1-L1)-FSH11-CSF, a Multi-Year Research Grant from the University of Macau.

Supplementary material

13428_2013_386_MOESM1_ESM.zip (8 kb)
ESM 1 (ZIP 7.74 KB)

References

  1. Baltes, B. B., Briggs, T. E., Huff, J. W., Wright, J. A., & Neuman, G. A. (1999). Flexible and compressed workweek schedules: A meta-analysis of their effects on work-related criteria. Journal of Applied Psychology, 84, 496–513.CrossRefGoogle Scholar
  2. Becker, B. J. (1992). Using results from replicated studies to estimate linear models. Journal of Educational and Behavioral Statistics, 17, 341–362. doi: 10.3102/10769986017004341 Google Scholar
  3. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 2, 97–111. doi: 10.1002/jrsm.12 CrossRefGoogle Scholar
  4. Cheung, M.W.L. (2013) metaSEM: Meta-analysis using structural equation modeling. Retrieved from http://courses.nus.edu.sg/course/psycwlm/Internet/metaSEM/metaSEM.pdf
  5. Cheung, M. W.-L. (2013a). metaSEM: Meta-analysis: A structural equation modeling approach (R package version 0.8-4) [Software]. Retrieved from http://courses.nus.edu.sg/course/psycwlm/internet/metaSEM/index.html
  6. Cheung, M. W.-L. (2013b). Multivariate meta-analysis as structural equation models. Structural Equation Modeling, 20, 429–454. doi: 10.1080/10705511.2013.797827 CrossRefGoogle Scholar
  7. Cheung, M. W.-L. (2013c). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods. doi: 10.1037/a0032968
  8. Cheung, S. F., & Chan, D. K.-S. (2004). Dependent effect sizes in meta-analysis: incorporating the degree of interdependence. Journal of Applied Psychology, 89, 780–791. doi: 10.1037/0021-9010.89.5.780 PubMedCrossRefGoogle Scholar
  9. Cheung, S. F., & Chan, D. K.-S. (2008). Dependent correlations in meta-analysis: The case of heterogeneous dependence. Educational and Psychological Measurement, 68, 760–777. doi: 10.1177/0013164408315263 CrossRefGoogle Scholar
  10. Cooper, H. M. (2009). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Thousand Oaks, CA: Sage.Google Scholar
  11. Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). New York, NY: Russell Sage.Google Scholar
  12. Hayes, A. F. (2012). PROCESS: A versatile computational tool for observed variable mediation, moderation, and conditional process modeling [White paper]. Retrieved from www.afhayes.com/public/process2012.pdf
  13. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando FL: Academic.Google Scholar
  14. Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65. doi: 10.1002/jrsm.5 CrossRefGoogle Scholar
  15. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504.CrossRefGoogle Scholar
  16. Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558. doi: 10.1002/sim.1186 PubMedCrossRefGoogle Scholar
  17. Huffcutt, A. I., Conway, J. M., Roth, P. L., & Stone, N. J. (2001). Identification and meta-analytic assessment of psychological constructs measured in employment interviews. Journal of Applied Psychology, 86, 897–913.PubMedCrossRefGoogle Scholar
  18. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.Google Scholar
  19. Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1, 227–235. doi: 10.1037/1082-989X.1.3.227 CrossRefGoogle Scholar
  20. Konstantopoulos, S. (2011). Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2, 61–76. doi: 10.1002/jrsm.35 CrossRefGoogle Scholar
  21. Madden, T. J., Ellen, P. S., & Ajzen, I. (1992). A comparison of the theory of planned behavior and the theory of reasoned action. Personality and Social Psychology Bulletin, 18, 3–9. doi: 10.1177/0146167292181001 CrossRefGoogle Scholar
  22. Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79, 1290–1326. doi: 10.3102/0034654309334143 CrossRefGoogle Scholar
  23. Martinussen, M., & Bjørnstad, J. F. (1999). Meta-analysis calculations based on independent and nonindependent cases. Educational and Psychological Measurement, 59, 928–950. doi: 10.1177/00131649921970260 CrossRefGoogle Scholar
  24. Muenchen, R. A. (2012). The popularity of data analysis software [Webpage]. Retrieved from http://r4stats.com/articles/popularity/
  25. R Development Core Team. (2012). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Downloaded from www.R-project.org/ Google Scholar
  26. Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111–120. doi: 10.1037/0033-2909.103.1.111 CrossRefGoogle Scholar
  27. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.Google Scholar
  28. Schmidt, F. L., Law, K., Hunter, J. E., & Rothstein, H. R. (1993). Refinements in validity generalization methods: Implications for the situational specificity hypothesis. Journal of Applied Psychology, 78, 3–12.CrossRefGoogle Scholar
  29. Schmidt, F. L., Le, H., & Oh, I.-S. (2009). Correcting for the distorting effects of study artifacts in meta-analysis. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 317–333). New York, NY: Russell Sage.Google Scholar
  30. Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45, 576–594. doi: 10.3758/s13428-012-0261-6 PubMedCrossRefGoogle Scholar
  31. Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3). Retrieved from www.jstatsoft.org/v36/i03/paper
  32. Viswesvaran, C., Sanchez, J. L., & Fisher, J. (1999). The role of social support in the process of work stress: A meta-analysis. Journal of Vocational Behavior, 54, 314–334.CrossRefGoogle Scholar
  33. Whitener, E. M. (1990). Confusion of confidence intervals and credibility intervals in meta-analysis. Journal of Applied Psychology, 75, 315–321. doi: 10.1037/0021-9010.75.3.315 CrossRefGoogle Scholar
  34. Wilson, D. B. (2005). Macro for SPSS/Win Version 6.1 or Higher (Version 2005.05.23) [Computer Software]. Retrieved from http://mason.gmu.edu/~dwilsonb/ma.html

Copyright information

© Psychonomic Society, Inc. 2013

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of MacauMacao SARChina
  2. 2.Department of PsychologyChinese University of Hong KongHong Kong SARChina
  3. 3.Department of Psychology, Faculty of Social SciencesUniversity of MacauMacao SARChina

Personalised recommendations