Skip to main content
SpringerLink
Log in

A Guide to Conducting a Meta-Analysis

  • Review
  • Published:
Neuropsychology Review Aims and scope Submit manuscript

Abstract

Meta-analysis is widely accepted as the preferred method to synthesize research findings in various disciplines. This paper provides an introduction to when and how to conduct a meta-analysis. Several practical questions, such as advantages of meta-analysis over conventional narrative review and the number of studies required for a meta-analysis, are addressed. Common meta-analytic models are then introduced. An artificial dataset is used to illustrate how a meta-analysis is conducted in several software packages. The paper concludes with some common pitfalls of meta-analysis and their solutions. The primary goal of this paper is to provide a summary background to readers who would like to conduct their first meta-analytic study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aytug, Z. G., Rothstein, H. R., Zhou, W., & Kern, M. C. (2012). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15(1), 103–133. doi:10.1177/1094428111403495.

    Article  Google Scholar 

  • Bax, L., Yu, L.-M., Ikeda, N., & Moons, K. G. (2007). A systematic comparison of software dedicated to meta-analysis of causal studies. BMC Medical Research Methodology, 7(1), 40. doi:10.1186/1471-2288-7-40.

    Article  PubMed  PubMed Central  Google Scholar 

  • Borenstein, M. (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–235). New York: Russell Sage Foundation.

    Google Scholar 

  • Borenstein, M., Hedges, L. V., & Rothstein, H. R. (2005). Comprehensive meta-analysis (version 2). Englewood NJ: Biostat.

    Google Scholar 

  • Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, West Sussex, U.K.; Hoboken: John Wiley & Sons.

    Book  Google Scholar 

  • Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1(2), 97–111. doi:10.1002/jrsm.12.

    Article  PubMed  Google Scholar 

  • Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12–37. doi:10.1177/0163278702025001003.

    Article  Google Scholar 

  • Cheung, M. W.-L. (2013). Multivariate meta-analysis as structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 20(3), 429–454. doi:10.1080/10705511.2013.797827.

    Article  Google Scholar 

  • Cheung, M. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychological Methods, 19(2), 211–229. doi:10.1037/a0032968.

    Article  PubMed  Google Scholar 

  • Cheung, M. W.-L. (2015a). Meta-analysis: A structural equation modeling approach. Chichester, West Sussex: John Wiley & Sons, Inc..

    Book  Google Scholar 

  • Cheung, M. W.-L. (2015b). metaSEM: an R package for meta-analysis using structural equation modeling. Frontiers in Psychology, 5(1521). doi:10.3389/fpsyg.2014.01521.

  • Cheung, M. W.-L., Ho, R. C. M., Lim, Y., & Mak, A. (2012). Conducting a meta-analysis: Basics and good practices. International Journal of Rheumatic Diseases, 15(2), 129–135. doi:10.1111/j.1756-185X.2012.01712.x

  • Cooper, H., & Hedges, L. V. (2009). Research synthesis as a scientific process. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 3–16). New York: Russell Sage Foundation.

    Google Scholar 

  • Cooper, H., Maxwell, S., Stone, A., & Sher, K. (2008). Reporting standards for research in psychology why do we need them? What might they be? American Psychologist, 63(9), 839–851. doi:10.1037/0003-066X.63.9.839.

    Article  Google Scholar 

  • Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

    Google Scholar 

  • Davey, J., Turner, R. M., Clarke, M. J., & Higgins, J. P. (2011). Characteristics of meta-analyses and their component studies in the Cochrane database of systematic reviews: a cross-sectional, descriptive analysis. BMC Medical Research Methodology, 11, 160. doi:10.1186/1471-2288-11-160.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dewey, M. (2015). CRAN task view: Meta-analysis. Retrieved from http://CRAN.R-project.org/view=MetaAnalysis

  • Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987). Publication bias and clinical trials. Controlled Clinical Trials, 8(4), 343–353. doi:10.1016/0197-2456(87)90155-3.

    Article  CAS  PubMed  Google Scholar 

  • Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991). Publication bias in clinical research. The Lancet, 337(8746), 867–872. doi:10.1016/0140-6736(91)90201-Y.

    Article  CAS  Google Scholar 

  • Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33(5), 517.

    Article  Google Scholar 

  • Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17(1), 120–128. doi:10.1037/a0024445.

    Article  PubMed  Google Scholar 

  • Fioravanti, M., Carlone, O., Vitale, B., Cinti, M. E., & Clare, L. (2005). A meta-analysis of cognitive deficits in adults with a diagnosis of schizophrenia. Neuropsychology Review, 15(2), 73–95. doi:10.1007/s11065-005-6254-9.

    Article  PubMed  Google Scholar 

  • Fleiss, J. L., & Berlin, J. A. (2009). Effect sizes for dichotomous data. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 237–253). New York: Russell Sage Foundation.

    Google Scholar 

  • Francis, G. (2013). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57(5), 153–169. doi:10.1016/j.jmp.2013.02.003.

    Article  Google Scholar 

  • Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8. doi:10.2307/1174772.

    Article  Google Scholar 

  • Harwell, M. (1997). An empirical study of Hedge’s homogeneity test. Psychological Methods, 2(2), 219–231. doi:10.1037/1082-989X.2.2.219.

    Article  Google Scholar 

  • Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

    Google Scholar 

  • Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203–217. doi:10.1037/1082-989X.6.3.203.

    Article  CAS  PubMed  Google Scholar 

  • Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9(4), 426–445. doi:10.1037/1082-989X.9.4.426.

    Article  PubMed  Google Scholar 

  • Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. doi:10.1002/sim.1186.

    Article  PubMed  Google Scholar 

  • Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327(7414), 557–560.

    Article  PubMed  PubMed Central  Google Scholar 

  • Huedo-Medina, T. B., Sanchez-Meca, J., Marin-Martinez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis. Q statistic or I2 index? Psychological Methods June 2006, 11(2), 193–206.

    PubMed  Google Scholar 

  • Hunt, M. (1997). How science takes stock: The story of meta-analysis. New York: Russell Sage Foundation.

    Google Scholar 

  • Koricheva, J., Gurevitch, J., & Mengersen, K. (Eds.) (2013). Handbook of meta-analysis in ecology and evolution. Princeton: Princeton University Press.

    Google Scholar 

  • Kupfersmid, J., & Fiala, M. (1991). A survey of attitudes and behaviors of authors who publish in psychology and education journals. American Psychologist, 46(3), 249–250. doi:10.1037/0003-066X.46.3.249.

    Article  Google Scholar 

  • Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ, 339(jul21 1), b2700–b2700. doi:10.1136/bmj.b2700.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lindsay, D. S. (2015). Replication in psychological science. Psychological Science, 26(12), 1827–1832. doi:10.1177/0956797615616374.

    Article  PubMed  Google Scholar 

  • Lipsey, M. W., & Wilson, D. (2000). Practical meta-analysis. Sage Publications, Inc ,Thousand Oaks.

  • Littell, J. H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis. Oxford University Press. Retrieved from http://www.oxfordscholarship.com/view/10.1093/acprof:oso/9780195326543.001.0001/acprof-9780195326543

  • Lucas, R. E., & Brent Donnellan, M. (2013). Improving the replicability and reproducibility of research published in the journal of research in personality. Journal of Research in Personality, 47(4), 453–454. doi:10.1016/j.jrp.2013.05.002.

    Article  Google Scholar 

  • Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498. doi:10.1037/a0039400.

    Article  PubMed  Google Scholar 

  • Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105–125. doi:10.1037/1082-989X.7.1.105.

    Article  PubMed  Google Scholar 

  • National Research Council (1992). Combining information: statistical issues and opportunities for research. Washington, DC: National Academies Press.

    Google Scholar 

  • O’Rourke, K. (2007). An historical perspective on meta-analysis: dealing quantitatively with varying study results. JRSM, 100(12), 579–582. doi:10.1258/jrsm.100.12.579.

    Article  Google Scholar 

  • Open Science Collaboration (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657–660. doi:10.1177/1745691612462588.

    Article  Google Scholar 

  • Open Science Collaboration (2015). Estimating the reproducibility of psychological. Science, 349(6251), aac4716. doi:10.1126/science.aac4716.

    Article  Google Scholar 

  • Palmer, T. M., & Sterne, J. A. C. (Eds.) (2016). Meta-analysis: an updated collection from the Stata journal (2nd ed.). College Station: Stata Press.

    Google Scholar 

  • Pearson, K. (1904). Report on certain enteric fever inoculation statistics. BMJ, 2(2288), 1243–1246. doi:10.1136/bmj.2.2288.1243.

    Article  Google Scholar 

  • Pigott, T. D. (2012). Advances in meta-analysis. New York: Springer. Retrieved from http://www.springer.com.libproxy1.nus.edu.sg/statistics/social+sciences+%26+law/book/978-1-4614-2277-8

  • R Development Core Team. (2016). R: a language and environment for statistical computing. Vienna, Austria. Retrieved from http://www.R-project.org/

  • Reed, J. G., & Baxter, P. M. (2009). Using reference databases. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 73–101). New York: Russell Sage Foundation.

    Google Scholar 

  • Rothstein, H. R., & Bushman, B. J. (2012). Publication bias in psychological science: comment on Ferguson and Brannick (2012). Psychological Methods, 17(1), 129–136. doi:10.1037/a0027128.

    Article  PubMed  Google Scholar 

  • Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester: John Wiley and Sons.

    Book  Google Scholar 

  • Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Shadish, W. R. (2015). Introduction to the special issue on the origins of modern meta-analysis. Research Synthesis Methods, 6(3), 219–220. doi:10.1002/jrsm.1148.

    Article  PubMed  Google Scholar 

  • Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752–760.

    Article  CAS  PubMed  Google Scholar 

  • Song, F., Parekh, S., Hooper, L., Loke, Y., Ryder, J., Sutton, A., et al. (2010). Dissemination and publication of research findings: an updated review of related biases. Health Technology Assessment, 14(8). doi:10.3310/hta14080.

  • Sterne, J. A. C., Egger, M., & Sutton, A. J. (2001). Meta-analysis software. In M. Egger, G. D. Smith, & D. G. Altman (Eds.), Systematic Reviews in Health Care: Meta-Analysis in Context (pp. 336–346). London: BMJ Publishing Group. Retrieved from doi:10.1002/9780470693926.ch17/summary

  • Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010). How many studies do you need? A primer on statistical power for meta-analysis. Journal of Educational and Behavioral Statistics, 35(2), 215–247. doi:10.3102/1076998609346961.

    Article  Google Scholar 

  • Viechtbauer, W. (2007). Hypothesis tests for population heterogeneity in meta-analysis. British Journal of Mathematical and Statistical Psychology, 60(1), 29–60. doi:10.1348/000711005X64042.

    Article  PubMed  Google Scholar 

  • Wallace, B. C., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2009). Meta-analyst: software for meta-analysis of binary, continuous and diagnostic data. BMC Medical Research Methodology, 9(1), 80. doi:10.1186/1471-2288-9-80.

    Article  PubMed  PubMed Central  Google Scholar 

  • Weare, K., & Nind, M. (2011). Mental health promotion and problem prevention in schools: what does the evidence say? Health Promotion International, 26(suppl 1), i29–i69. doi:10.1093/heapro/dar075.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Mike W.-L. Cheung was supported by the Academic Research Fund Tier 1 (FY2013-FRC5-002) from the Ministry of Education, Singapore. We would like to thank Maggie Chan for providing comments on an earlier version of this manuscript.

Author information

Authors and Affiliations

  1. Department of Psychology, Faculty of Arts and Social Sciences, National University of Singapore, Block AS4, Level 2, 9 Arts Link, Singapore, 117570, Singapore

    Mike W.-L. Cheung & Ranjith Vijayakumar

Authors
  1. Mike W.-L. Cheung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ranjith Vijayakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mike W.-L. Cheung.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheung, M.WL., Vijayakumar, R. A Guide to Conducting a Meta-Analysis. Neuropsychol Rev 26, 121–128 (2016). https://doi.org/10.1007/s11065-016-9319-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11065-016-9319-z

Keywords

Search

Navigation