Statistical Methods and Applications

, Volume 19, Issue 2, pp 217–236 | Cite as

Consequences of effect size heterogeneity for meta-analysis: a Monte Carlo study

  • Mark J. Koetse
  • Raymond J. G. M. Florax
  • Henri L. F. de Groot
Open Access
Original Article


In this article we use Monte Carlo analysis to assess the small sample behaviour of the OLS, the weighted least squares (WLS) and the mixed effects meta-estimators under several types of effect size heterogeneity, using the bias, the mean squared error and the size and power of the statistical tests as performance indicators. Specifically, we analyse the consequences of heterogeneity in effect size precision (heteroskedasticity) and of two types of random effect size variation, one where the variation holds for the entire sample, and one where only a subset of the sample of studies is affected. Our results show that the mixed effects estimator is to be preferred to the other two estimators in the first two situations, but that WLS outperforms OLS and mixed effects in the third situation. Our findings therefore show that, under circumstances that are quite common in practice, using the mixed effects estimator may be suboptimal and that the use of WLS is preferable.


Effect size heterogeneity Meta-analysis Monte Carlo analysis OLS meta-estimator WLS meta-estimator Mixed effects meta-estimator Small sample performance 

JEL Classification

C12 C15 C40 



This research is supported through the program ‘Stimulating the Adoption of Energy-Efficient Technologies’, funded by the Netherlands Organization for Scientific Research (NWO) and the Dutch Ministry of Economic Affairs (SenterNovem). We are grateful to two anonymous referees for useful comments on an earlier version of this article. The usual disclaimer applies.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.


  1. Abreu M, De Groot HLF, Florax RJGM (2005) A meta-analysis of beta-convergence: the legendary 2%. J Econ Surv 19:389–420Google Scholar
  2. Bijmolt THA, Pieters RGM (2001) Meta-analysis in marketing when studies contain multiple measurements. Mark Lett 12: 157–169CrossRefGoogle Scholar
  3. Brockwell SE, Gordon IR (2001) A comparison of statistical methods for meta-analysis. Stat Med 20: 825–840CrossRefGoogle Scholar
  4. Brons MRA, Nijkamp P, Pels E, Rietveld P (2008) A meta-analysis of the price elasticity of gasoline demand: a SUR approach. Energy Econ 30: 2105–2122CrossRefGoogle Scholar
  5. De Dominicis L, Florax RJGM, De Groot HLF (2008) Meta-analysis of the relationship between income inequality and economic growth. Scott J Political Econ 55: 654–682CrossRefGoogle Scholar
  6. Field AP (2001) Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects methods. Psychol Methods 6: 161–180CrossRefGoogle Scholar
  7. Greene WH (2000) Econometric analysis, 4th edn. Prentice-Hall, Upper Saddle Rivern, New JerseyGoogle Scholar
  8. Hedges LV (1994) Fixed effects models. In: Cooper H, Hedges LV (eds) The handbook of research synthesis. Russell Sage Foundation, New YorkGoogle Scholar
  9. Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, Orlando, FloridazbMATHGoogle Scholar
  10. Higgins JPT, Thompson SG (2004) Controlling the risk of spurious findings from meta-regression. Stat Med 23: 1663–1682CrossRefGoogle Scholar
  11. Hunt M (1997) How science takes stock: the story of meta-analysis. Russell Sage Foundation, New YorkGoogle Scholar
  12. Koetse MJ, De Groot HLF, Florax RJGM (2008) Capital-energy substitution and shifts in factor demand: a meta-analysis. Energy Econ 30: 2236–2251CrossRefGoogle Scholar
  13. Koetse MJ, De Groot HLF, Florax RJGM (2009) A meta-analysis of the investment-uncertainty relationship. South Econ J 76: 283–306Google Scholar
  14. Koetse MJ, Florax RJGM, De Groot HLF (2005) Correcting for primary study misspecifications in meta-analysis. Tinbergen Institute Discussion Paper 05-029/3, Tinbergen Institute, AmsterdamGoogle Scholar
  15. Kuhnert R, Böhning D (2007) A comparison of three different models for estimating relative risk in meta-analysis of 3 clinical trials under unobserved heterogeneity. Stat Med 26: 2277–2296CrossRefMathSciNetGoogle Scholar
  16. Nijkamp P, Poot J (2004) Meta-analysis of the effect of fiscal policies on long-run growth. Eur J Political Econ 20: 91–124CrossRefGoogle Scholar
  17. Nijkamp P, Poot J (2005) The last word on the wage curve? A meta-analytic assessment. J Econ Surv 19: 421–450Google Scholar
  18. Oswald FL, Johnson JW (1998) On the robustness, bias, and stability of statistics from meta-analysis of correlation coefficients: some initial Monte Carlo findings. J Appl Psychol 83: 164–178CrossRefGoogle Scholar
  19. Roberts CJ, Stanley TD (2005) Meta-regression analysis: issues of publication bias in economics. Blackwell, OxfordGoogle Scholar
  20. Sanchez-Meca J, Marin-Martinez F (1997) Homogeneity tests in meta-analysis: a Monte Carlo comparison of statistical power and Type I error. Qual Quant 31: 385–399CrossRefGoogle Scholar
  21. Sanchez-Meca J, Marin-Martinez F (1998) Weighting by inverse variance or by sample size in meta-analysis: a simulation study. Educ Psychol Meas 58: 211–220CrossRefGoogle Scholar
  22. Stanley TD (2001) Wheat from chaff: meta-analysis as quantitative literature review. J Econ Perspect 15: 131–150CrossRefGoogle Scholar
  23. Stanley TD (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf Bull Econ Stat 70: 103–127Google Scholar
  24. Stanley TD, Jarrell SB (1989) Meta-regression analysis: a quantitative method of literature surveys. J Econ Surv 3: 54–67Google Scholar
  25. Sutton AJ, Abrams KR, Sheldon TA, Song F (2000) Methods for meta-analysis in medical research. Wiley, New YorkGoogle Scholar
  26. Weichselbaumer D, Winter-Ebmer R (2005) A meta-analysis of the international gender wage gap. J Econ Surv 19: 479–511Google Scholar

Copyright information

© The Author(s) 2009

Authors and Affiliations

  • Mark J. Koetse
    • 1
  • Raymond J. G. M. Florax
    • 1
    • 2
  • Henri L. F. de Groot
    • 1
    • 3
  1. 1.Department of Spatial EconomicsVU University AmsterdamAmsterdamThe Netherlands
  2. 2.Department of Agricultural EconomicsPurdue UniversityWest LafayetteUSA
  3. 3.Tinbergen InstituteAmsterdamThe Netherlands

Personalised recommendations