Evolutionary Ecology

, Volume 26, Issue 5, pp 1253–1274 | Cite as

Methodological issues and advances in biological meta-analysis

  • Shinichi NakagawaEmail author
  • Eduardo S. A. Santos
Original Paper


Meta-analysis has changed the way researchers conduct literature reviews not only in medical and social sciences but also in biological sciences. Meta-analysis in biological sciences, especially in ecology and evolution (which we refer to as ‘biological’ meta-analysis) faces somewhat different methodological problems from its counterparts in medical and social sciences, where meta-analytic techniques were originally developed. The main reason for such differences is that biological meta-analysis often integrates complex data composed of multiple strata with, for example, different measurements and a variety of species. Here, we review methodological issues and advancements in biological meta-analysis, focusing on three topics: (1) non-independence arising from multiple effect sizes obtained in single studies and from phylogenetic relatedness, (2) detecting and accounting for heterogeneity, and (3) identifying publication bias and measuring its impact. We show how the marriage between mixed-effects (hierarchical/multilevel) models and phylogenetic comparative methods has resolved most of the issues under discussion. Furthermore, we introduce the concept of across-study and within-study meta-analysis, and propose how the use of within-study meta-analysis can improve many empirical studies typical of ecology and evolution.


Fixed-effect meta-analysis Random-effects meta-analysis Meta-regression Egger’s regression I2 Heterogeneity Multivariate meta-analysis Trim and fill method 



The authors thank J. Endler for conceiving this special issue and E. Schlicht, S. Chamberlain, M. Lagisz and two anonymous reviewers for the comments on an early version of this paper. S.N. acknowledges the Humboldt Fellowship and Marsden Fund for support and also is grateful to B. Kempenaers for hosting and providing an excellent working environment at the MPIO. E.S.A.S. acknowledges the support of a University of Otago PhD Scholarship.


  1. Adams DC (2008) Phylogenetic meta-analysis. Evolution 62:567–572PubMedCrossRefGoogle Scholar
  2. Arnqvist G, Wooster D (1995) Meta-analysis: synthesizing research findings in ecology and evolution. Trends Ecol Evol 10:236–240PubMedCrossRefGoogle Scholar
  3. Barto EK, Rillig MC (2012) Dissemination biases in ecology: effect sizes matter more than quality. Oikos 121:228–235Google Scholar
  4. Becker BJ (2005) Fail safe N or file-drawer number. In: Rothstein H, Sutton AJ, Borenstein M (eds) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, Chichester, pp 111–125Google Scholar
  5. Begg CB, Mazumdar M (1994) Operating characteristics of a rank correlation test for publication bias. Biometrics 50:1088–1101PubMedCrossRefGoogle Scholar
  6. Blomberg SP, Garland T, Ives AR (2003) Testing for phylogenetic signal in comparative data: behavioral traits are more labile. Evolution 57:717–745PubMedGoogle Scholar
  7. Bolker BM, Brooks ME, Clark CJ, Geange SW, Poulsen JR, Stevens MHH, White JSS (2009) Generalized linear mixed models: a practical guide for ecology and evolution. Trends Ecol Evol 24:127–135PubMedCrossRefGoogle Scholar
  8. Borenstein M (2005) Software for publication bias. In: Rothstein H, Sutton AJ, Borenstein M (eds) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, Chichester, pp 193–220Google Scholar
  9. Borenstein M, Hedges LV, Higgins JPI, Rothstein HR (2009) Introduction to meta-analysis. Wiley, ChichesterCrossRefGoogle Scholar
  10. Burnham KP, Anderson DR (2002) Model selection and multimodel inference: a practical information-theoretic approach, 2nd edn. Springer, BerlinGoogle Scholar
  11. Carmona D, Lajeunesse MJ, Johnson MTJ (2011) Plant traits that predict resistance to herbivores. Funct Ecol 25:358–367CrossRefGoogle Scholar
  12. Cheung MWL (2008) A model for integrating fixed-, random-, and mixed-effects meta-analyses into structural equation modeling. Psychol Methods 13:182–202PubMedCrossRefGoogle Scholar
  13. Cleasby IR, Nakagawa S (2012) The influence of male age on within-pair and extra-pair paternity in passerines. Ibis. doi: 10.1111/j.1474-919X.2011.01209.x
  14. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum, HillsdaleGoogle Scholar
  15. Congdon P (2003) Applied Bayesian modelling. Wiley, ChichesterCrossRefGoogle Scholar
  16. Cooper H, Hedges LV, Valentine JC (2009) The handbook of research synthesis and meta-analysis, 2nd edn. Russell Sage Foundation, New YorkGoogle Scholar
  17. Cornwallis CK, West SA, Davis KE, Griffin AS (2010) Promiscuity and the evolutionary transition to complex societies. Nature 466:969–972PubMedCrossRefGoogle Scholar
  18. Cumming G, Finch S (2001) A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educ Psychol Measurement 61:532–584Google Scholar
  19. Davidson AM, Jennions M, Nicotra AB (2011) Do invasive species show higher phenotypic plasticity than native species and, if so, is it adaptive? A meta-analysis. Ecol Lett 14:419–431PubMedCrossRefGoogle Scholar
  20. Dubois F, Cezilly F (2002) Breeding success and mate retention in birds: a meta-analysis. Behav Ecol Sociobiol 52:357–364CrossRefGoogle Scholar
  21. Duval S (2005) The trim and fill method. In: Rothstein H, Sutton AJ, Borenstein M (eds) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, Chichester, pp 127–144Google Scholar
  22. Duval S, Tweedie R (2000a) A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. J Am Stat Assoc 95:89–98Google Scholar
  23. Duval S, Tweedie R (2000b) Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56:455–463PubMedCrossRefGoogle Scholar
  24. Egger M, Smith GD, Schneider M, Minder C (1997) Bias in meta-analysis detected by a simple, graphical test. Br Med J 315:629–634CrossRefGoogle Scholar
  25. Egger M, Smith GD, Altman DG (2001) Systematic reviews in health care: meta-analysis in context, 2nd edn. BMJ, LondonCrossRefGoogle Scholar
  26. Evans SR, Hinks AE, Wilkin TA, Sheldon BC (2010) Age, sex and beauty: methodological dependence of age- and sex-dichromatism in the great tit Parus major. Biol J Linn Soc 101:777–796CrossRefGoogle Scholar
  27. Felsenstein J (1985) Phylogenies and the comparative method. Am Nat 125:1–15CrossRefGoogle Scholar
  28. Felsenstein J (2008) Comparative methods with sampling error and within-species variation: contrasts revisited and revised. Am Nat 171:713–725PubMedCrossRefGoogle Scholar
  29. Freckleton RP, Harvey PH, Pagel M (2002) Phylogenetic analysis and comparative data: a test and review of evidence. Am Nat 160:712–726PubMedCrossRefGoogle Scholar
  30. Garamszegi LZ (2006) Comparing effect sizes across variables: generalization without the need for Bonferroni correction. Behav Ecol 17:682–687CrossRefGoogle Scholar
  31. Gasparrini A (2011) Multivariate meta-analysis and meta-regression: package ‘mvmeta’. In: R package version 0.2.3 edn.
  32. Gates S (2002) Review of methodology of quantitative reviews using meta-analysis in ecology. J Anim Ecol 71:547–557CrossRefGoogle Scholar
  33. Gelman A, Hill J (2007) Data analysis using regression and multilevel/hierarchical models. Cambridge University Press, CambridgeGoogle Scholar
  34. Gilmour AR, Gogel BJ, Cullis BR, Welham SJ, Thompson R (2002) ASReml user guide release 1.0. VSN Internationl Ltd, Hemel Hempstead, UKGoogle Scholar
  35. Griffin AS, West SA (2003) Kin discrimination and the benefit of helping in cooperatively breeding vertebrates. Science 302:634–636PubMedCrossRefGoogle Scholar
  36. Griffin AS, Sheldon BC, West SA (2005) Cooperative breeders adjust offspring sex ratios to produce helpful helpers. Am Nat 166:628–632Google Scholar
  37. Grueber CE, Nakagawa S, Laws RJ, Jamieson IG (2011) Multimodel inference in ecology and evolution: challenges and solutions. J Evol Biol 24:699–711PubMedCrossRefGoogle Scholar
  38. Gurevitch J, Hedges LV (1999) Statistical issues in ecological meta-analyses. Ecology 80:1142–1149CrossRefGoogle Scholar
  39. Gurevitch J, Curtis PS, Jones MH (2001) Meta-analysis in ecology. Adv Ecol Res 32:199–247CrossRefGoogle Scholar
  40. Hadfield JD (2010) MCMC methods for multi-response generalised linear mixed models: the MCMCglmm R package. J Stat Softw 33:1–22Google Scholar
  41. Hadfield JD, Nakagawa S (2010) General quantitative genetic methods for comparative biology: phylogenies, taxonomies and multi-trait models for continuous and categorical characters. J Evol Biol 23:494–508PubMedCrossRefGoogle Scholar
  42. Hamilton WD (1964) The genetical evolution of social behaviour. I and II. J Theor Biol 7:1–52PubMedCrossRefGoogle Scholar
  43. Harrison F (2011) Getting started with meta-analysis. Methods Ecol Evol 2:1–10CrossRefGoogle Scholar
  44. Harvey PH, Pagel MD (1991) The comparative method in evolutionary biology. Oxford University Press, OxfordGoogle Scholar
  45. Hatchwell BJ (1999) Investment strategies of breeders in avian cooperative breeding systems. Am Nat 154:205–219CrossRefGoogle Scholar
  46. Hector KL, Nakagawa S (2012) Quantitative analysis of compensatory and catch-up growth in diverse taxa. J Anim Ecol. doi: 10.1111/j.1365-2656.2011.01942.x
  47. Hedges L, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, New YorkGoogle Scholar
  48. Hedges LV, Vevea JL (1998) Fixed- and random-effects models in meta-analysis. Psychol Methods 3:486–504CrossRefGoogle Scholar
  49. Hedges LV, Vevea JL (2005) Selection method approaches. In: Rothstein H, Sutton AJ, Borenstein M (eds) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, Chichester, pp 145–174Google Scholar
  50. Hedges LV, Gurevitch J, Curtis PS (1999) The meta-analysis of response ratios in experimental ecology. Ecology 80:1150–1156CrossRefGoogle Scholar
  51. Higgins JPT, Thompson SG (2002) Quantifying heterogeneity in a meta-analysis. Stat Med 21:1539–1558PubMedCrossRefGoogle Scholar
  52. Higgins JPT, Thompson SG, Deeks JJ, Altman DG (2003) Measuring inconsistency in meta-analyses. Br Med J 327:557–560CrossRefGoogle Scholar
  53. Higgins JPT, Thompson SG, Spiegelhalter DJ (2009) A re-evaluation of random-effects meta-analysis. J R Stat Soc A 172:137–159CrossRefGoogle Scholar
  54. Horváthová T, Nakagawa S, Uller T (2012) Strategic female reproductive investment in response to male attractiveness in birds. Proc R Soc B 279:163–170Google Scholar
  55. Hunt M (1997) How science takes stock: the story of meta-analysis. Russell Sage, New YorkGoogle Scholar
  56. Ives AR, Zhu J (2006) Statistics for correlated data: phylogenies, space, and time. Ecol Appl 16:20–32PubMedCrossRefGoogle Scholar
  57. Ives AR, Midford PE, Garland T (2007) Within-species variation and measurement error in phylogenetic comparative methods. Syst Biol 56:252–270PubMedCrossRefGoogle Scholar
  58. Jackson D, Riley R, White IR (2011) Multivariate meta-analysis: potential and promise. Stat Med 30:2481–2498CrossRefGoogle Scholar
  59. Jennions MD, Møller AP (2002) Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Proc R Soc B 269:43–48PubMedCrossRefGoogle Scholar
  60. Jennions MD, Lorite C, Rosenberg M, Rothstein H (2012) Publication and related biases. In: Koricheva J, Gurevitch J, Mengersen K (eds) The handbook of meta-analysis in ecology and evolution. Princeton University Press, Princeton (in press)Google Scholar
  61. Jones KS, Nakagawa S, Sheldon BC (2009) Environmental sensitivity in relation to size and sex in birds: meta-regression analysis. Am Nat 174:122–133PubMedCrossRefGoogle Scholar
  62. Kelly CD, Jennions MD (2011) Sexual selection and sperm quantity: meta-analyses of strategic ejaculation. Biol Rev 86:863–884PubMedCrossRefGoogle Scholar
  63. Knowles SCL, Nakagawa S, Sheldon BC (2009) Elevated reproductive effort increases blood parasitaemia and decreases immune function in birds: a meta-regression approach. Funct Ecol 23:405–415CrossRefGoogle Scholar
  64. Koricheva J, Gurevitch J, Mengersen K (2012) The handbook of meta-analysis in ecology and evolution. Princeton University Press, PrincetonGoogle Scholar
  65. Kueffer C, Niinemets U, Drenovsky RE, Kattge J, Milberg P, Poorter H, Reich PB, Werner C, Westoby M, Wright IJ (2011) Fame, glory and neglect in meta-analyses. Trends Ecol Evol 26:493–494PubMedCrossRefGoogle Scholar
  66. Lajeunesse MJ (2009) Meta-analysis and the comparative phylogenetic method. Am Nat 174:369–381PubMedCrossRefGoogle Scholar
  67. Lajeunesse MJ (2010) Achieving synthesis with meta-analysis by combining and comparing all available studies. Ecology 91:2561–2564PubMedCrossRefGoogle Scholar
  68. Lajeunesse MJ (2011a) On the meta-analysis of response ratios for studies with correlated and multi-group designs. Ecology 92:2049–2055CrossRefGoogle Scholar
  69. Lajeunesse MJ (2011b) phyloMeta: a program for phylogenetic comparative analyses with meta-analysis. Bioinformatics 27:2603–2604PubMedGoogle Scholar
  70. Lajeunesse MJ, Forbes MR (2003) Variable reporting and quantitative reviews: a comparison of three meta-analytical techniques. Ecol Lett 6:448–454CrossRefGoogle Scholar
  71. Leimu R, Koricheva J (2004) Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. Proc R Soc B 271:1961–1966PubMedCrossRefGoogle Scholar
  72. Leimu R, Koricheva J (2005) What determines the citation frequency of ecological papers? Trends Ecol Evol 20:28–32PubMedCrossRefGoogle Scholar
  73. Liermann M, Hilborn R (1997) Depensation in fish stocks: a hierarchic Bayesian meta-analysis. Can J Fish Aquat Sci 54:1976–1984Google Scholar
  74. Lipsey MW, Wilson DB (2001) Practical meta-analysis. Sage, Beverly HillsGoogle Scholar
  75. Lortie CJ, Aarssen LW, Budden AE, Koricheva JK, Leimu R, Tregenza T (2007) Publication bias and merit in ecology. Oikos 116:1247–1253CrossRefGoogle Scholar
  76. Lunn DJ, Thomas A, Best N, Spiegelhalter D (2000) WinBUGS—a Bayesian modelling framework: concepts, structure, and extensibility. Stat Computing 10:325–337CrossRefGoogle Scholar
  77. Lynch M (1991) Methods for the analysis of comparative data in evolutionary biology. Evolution 45:1065–1080CrossRefGoogle Scholar
  78. Martins EP, Hansen TF (1997) Phylogenies and the comparative method: a general approach to incorporating phylogenetic information into the analysis of interspecific data. Am Nat 149:646–667CrossRefGoogle Scholar
  79. Meunier J, Pinto SF, Burri R, Roulin A (2011) Eumelanin-based coloration and fitness parameters in birds: a meta-analysis. Behav Ecol Sociobiol 65:559–567CrossRefGoogle Scholar
  80. Milner RNC, Detto T, Jennions MD, Backwell PRY (2010) Experimental evidence for a seasonal shift in the strength of a female mating preference. Behav Ecol 21:311–316CrossRefGoogle Scholar
  81. Møller AP, Jennions MD (2001) Testing and adjusting for publication bias. Trends Ecol Evol 16:580–586CrossRefGoogle Scholar
  82. Nakagawa S, Cuthill IC (2007) Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev 82:591–605PubMedCrossRefGoogle Scholar
  83. Nakagawa S, Freckleton RP (2008) Missing inaction: the dangers of ignoring missing data. Trends Ecol Evol 23:592–596PubMedCrossRefGoogle Scholar
  84. Nakagawa S, Hauber ME (2010) Great challenges with few subjects: statistical strategies for neuroscientists. Neurosci Biobehav Rev 35:462–473PubMedCrossRefGoogle Scholar
  85. Osenberg CW, Sarnelle O, Cooper SD, Holt RD (1999) Resolving ecological questions through meta-analysis: goals, metrics, and models. Ecology 80:1105–1117CrossRefGoogle Scholar
  86. Pagel M (1999) Inferring the historical patterns of biological evolution. Nature 401:877–884PubMedCrossRefGoogle Scholar
  87. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L (2007) Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Stat Med 26:4544–4562PubMedCrossRefGoogle Scholar
  88. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L (2008) Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol 61:991–996PubMedCrossRefGoogle Scholar
  89. Pigott TD (2009) Handling missing data. In: Cooper H, Hedges LV, Valentine JC (eds) The handbook of research synthesis and meta-analysis. Russell Sage Foundation, New York, pp 399–416Google Scholar
  90. Pinheiro JC, Bates DM (2000) Mixed-effects models in S and S-Plus. Springer, New YorkCrossRefGoogle Scholar
  91. Poulin R (2000) Manipulation of host behaviour by parasites: a weakening paradigm? Proc R Soc B 267:787–792PubMedCrossRefGoogle Scholar
  92. Roberts CJ, Stanley TD (2005) Meta-regression analysis: issues of publication bias in economics. Blackwell, MaldenGoogle Scholar
  93. Rosenberg MS (2005) The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis. Evolution 59:464–468PubMedGoogle Scholar
  94. Rosenberg MS, Adams DC, Gurevitch J (2000) MetaWin: statistical software for meta-analysis, 2nd edn. Sinauer, SunderlandGoogle Scholar
  95. Rosenthal R (1979) The “file drawer problem” and tolerance for null results. Psychol Bull 86:638–641CrossRefGoogle Scholar
  96. Rothstein H, Sutton AJ, Borenstein M (2005) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, ChichesterCrossRefGoogle Scholar
  97. Santos ESA, Macedo RH (2011) Load lightening in southern lapwings: group-living mothers lay smaller eggs than pair-living mothers. Ethology 117:547–555CrossRefGoogle Scholar
  98. Santos ESA, Scheck D, Nakagawa S (2011) Dominance and plumage traits: meta-analysis and metaregression analysis. Anim Behav 82:3–19CrossRefGoogle Scholar
  99. Schielzeth H (2010) Simple means to improve the interpretability of regression coefficients. Methods Ecol and Evol 1:103–113CrossRefGoogle Scholar
  100. Schwarzer G (2010) Meta-analysis with R: package ‘meta’. In: R package version 1.6-1 edn.
  101. Schwarzer G, Carpenter J, Rucker G (2010) Empirical evaluation suggests Copas selection model preferable to trim and fill method for selection bias in meta-analysis. J Clin Epidemiol 63:282–288PubMedCrossRefGoogle Scholar
  102. Slatyer RA, Mautz BS, Backwell PR, Jennions MD (2011) Estimating genetic benefits of polyandry from experimental studies: a meta-analysis. Biol Rev 86:863–884CrossRefGoogle Scholar
  103. Smith BR, Blumstein DT (2008) Fitness consequences of personality: a meta-analysis. Behav Ecol 19:448–455CrossRefGoogle Scholar
  104. Sterne JAC, Becker BJ, Egger M (2005) The funnel plot. In: Rothstein H, Sutton AJ, Borenstein M (eds) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, Chichester, pp 75–98Google Scholar
  105. Sutton AJ (2009) Publication bias. In: Cooper H, Hedges LV, Valentine JC (eds) The handbook of research synthesis and meta-analysis. Russell Sage Foundation, New York, pp 435–452Google Scholar
  106. Sutton AJ, Higgins JPI (2008) Recent developments in meta-analysis. Stat Med 27:625–650PubMedCrossRefGoogle Scholar
  107. Sutton JT, Nakagawa S, Robertson BC, Jamieson IG (2011) Disentangling the roles of natural selection and genetic drift in shaping variation at MHC immunity genes. Mol Ecol 20:4408–4420PubMedCrossRefGoogle Scholar
  108. Terrin N, Schmid CH, Lau J (2005) In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 58:894–901PubMedCrossRefGoogle Scholar
  109. Thompson B (2002) What future quantitative social science research could look like: confidence intervals for effect sizes. Educ Researcher 31:25–32CrossRefGoogle Scholar
  110. Trikalinos TA, Ioannidis JP (2005) Assessing the evolution of effect sizes over time. In: Rothstein H, Sutton AJ, Borenstein M (eds) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, Chichester, pp 241–259Google Scholar
  111. Van den Bergh B, Dewitte S (2006) Digit ratio (2D: 4D) moderates the impact of sexual cues on men’s decisions in ultimatum games. Proc R Soc B 273:2091–2095PubMedCrossRefGoogle Scholar
  112. Verdu M, Traveset A (2004) Bridging meta-analysis and the comparative method: a test of seed size effect on germination after frugivores’ gut passage. Oecologia 138:414–418PubMedCrossRefGoogle Scholar
  113. Verdu M, Traveset A (2005) Early emergence enhances plant fitness: a phylogenetically controlled meta-analysis. Ecology 86:1385–1394CrossRefGoogle Scholar
  114. Viechtbauer W (2010) Conducting meta-analyses in R with the meta for package. J Stat Softw 36:1–48Google Scholar
  115. Wang MC, Bushman BJ (1998) Using the normal quantile plot to explore meta-analytic data sets. Psychol Methods 3:46–54CrossRefGoogle Scholar
  116. Wehi P, Nakagawa S, Trewick S, Morgan-Richards M (2011) Does predation result in adult sex ratio skew in a sexually dimorphic insect genus? J Evol Biol 24:2321–2328PubMedCrossRefGoogle Scholar
  117. Weir LK, Grant JWA, Hutchings JA (2011) The influence of operational sex ratio on the intensity of competition for mates. Am Nat 177:167–176PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  1. 1.Department of ZoologyUniversity of OtagoDunedinNew Zealand
  2. 2.Max Planck Institute for OrnithologySeewiesenGermany

Personalised recommendations