Consequences of effect size heterogeneity for meta-analysis: a Monte Carlo study
- 575 Downloads
In this article we use Monte Carlo analysis to assess the small sample behaviour of the OLS, the weighted least squares (WLS) and the mixed effects meta-estimators under several types of effect size heterogeneity, using the bias, the mean squared error and the size and power of the statistical tests as performance indicators. Specifically, we analyse the consequences of heterogeneity in effect size precision (heteroskedasticity) and of two types of random effect size variation, one where the variation holds for the entire sample, and one where only a subset of the sample of studies is affected. Our results show that the mixed effects estimator is to be preferred to the other two estimators in the first two situations, but that WLS outperforms OLS and mixed effects in the third situation. Our findings therefore show that, under circumstances that are quite common in practice, using the mixed effects estimator may be suboptimal and that the use of WLS is preferable.
KeywordsEffect size heterogeneity Meta-analysis Monte Carlo analysis OLS meta-estimator WLS meta-estimator Mixed effects meta-estimator Small sample performance
JEL ClassificationC12 C15 C40
This research is supported through the program ‘Stimulating the Adoption of Energy-Efficient Technologies’, funded by the Netherlands Organization for Scientific Research (NWO) and the Dutch Ministry of Economic Affairs (SenterNovem). We are grateful to two anonymous referees for useful comments on an earlier version of this article. The usual disclaimer applies.
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
- Abreu M, De Groot HLF, Florax RJGM (2005) A meta-analysis of beta-convergence: the legendary 2%. J Econ Surv 19:389–420Google Scholar
- Greene WH (2000) Econometric analysis, 4th edn. Prentice-Hall, Upper Saddle Rivern, New JerseyGoogle Scholar
- Hedges LV (1994) Fixed effects models. In: Cooper H, Hedges LV (eds) The handbook of research synthesis. Russell Sage Foundation, New YorkGoogle Scholar
- Hunt M (1997) How science takes stock: the story of meta-analysis. Russell Sage Foundation, New YorkGoogle Scholar
- Koetse MJ, De Groot HLF, Florax RJGM (2009) A meta-analysis of the investment-uncertainty relationship. South Econ J 76: 283–306Google Scholar
- Koetse MJ, Florax RJGM, De Groot HLF (2005) Correcting for primary study misspecifications in meta-analysis. Tinbergen Institute Discussion Paper 05-029/3, Tinbergen Institute, AmsterdamGoogle Scholar
- Nijkamp P, Poot J (2005) The last word on the wage curve? A meta-analytic assessment. J Econ Surv 19: 421–450Google Scholar
- Roberts CJ, Stanley TD (2005) Meta-regression analysis: issues of publication bias in economics. Blackwell, OxfordGoogle Scholar
- Stanley TD (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf Bull Econ Stat 70: 103–127Google Scholar
- Stanley TD, Jarrell SB (1989) Meta-regression analysis: a quantitative method of literature surveys. J Econ Surv 3: 54–67Google Scholar
- Sutton AJ, Abrams KR, Sheldon TA, Song F (2000) Methods for meta-analysis in medical research. Wiley, New YorkGoogle Scholar
- Weichselbaumer D, Winter-Ebmer R (2005) A meta-analysis of the international gender wage gap. J Econ Surv 19: 479–511Google Scholar