Abstract
This chapter provides a basic methodological guide to the conduct of meta-analyses, and it reviews the tools used for the purpose. It considers the specific features of empirical econometric analysis, and it cites as examples its application in policy/program evaluation. A salient feature of empirical studies in economics is that estimated effects mainly derive from econometric techniques based on multiple regressions on large samples, using different model specifications, datasets, identification strategies, and definitions of the variables of primary interest. A second salient feature is that, in economics, the analyst typically runs many regressions and the published results are severely selected from those of a larger set of econometric models. This may give rise to marked data dredging and publication bias. Accordingly, this chapter describes modern meta-analytic methods with which to derive generalized conclusions from heterogeneous results and to detect and correct for publication bias.
This is a preview of subscription content, log in via an institution to check access.
References
Altman DG, Bland JM (2011) How to obtain the P value from a confidence interval. Br Med J 343:d2304. https://doi.org/10.1136/bmj.d2304
Andrews I, Kasy M (2019) Identification of and correction for publication bias. Am Econ Rev 109(8):2766–2794. https://doi.org/10.1257/aer.20180310
Ashenfelter O, Harmon C, Oosterbeek H (1999) A review of estimates of the schooling/earnings relationship, with tests for publication bias. Labour Econ 6(4):453–470. https://doi.org/10.1016/S0927-5371(99)00041-X
Bassanini A, Caroli E (2015) Is work bad for health? The role of constraint versus choice. Ann Econ Statistic 119(120):13–37. https://doi.org/10.15609/annaeconstat2009.119-120.13
Begg CB, Berlin JA (1988) Publication bias: a problem in interpreting medical data. J Royal Statistic Soc Series A 151(3):419–463. https://doi.org/10.2307/2982993
Blanco-Perez C, Brodeur A (2020) Publication bias and editorial statement on negative findings. Econ J 130(629):1126–1247. https://doi.org/10.1093/ej/ueaa011
Bom PRD, Rachinger H (2019) A kinked meta-regression model for publication bias correction. Res Synth Methods 10(4):497–514. https://doi.org/10.1002/jrsm.1352
Bornmann L, Rüdiger M (2015) Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J Assoc Inf Sci Technol 66(11):2215–2222. https://doi.org/10.1002/asi.23329
Brodeur A, Cook N, Heyes A (2016) Star wars: the empirics strike back. Am Econ J Appl Econ 8(1):1–32. https://doi.org/10.1257/app.20150044
Brodeur A, Cook N, Heyes A (2020) Methods matter: p-hacking and publication bias in causal analysis in economics. Am Econ Rev 110(11):3634–3660. https://doi.org/10.1257/aer.20190687
Calonico S, Cattaneo MD, Farrell MH (2018) On the effect of bias estimation on coverage accuracy in nonparametric inference. J Am Stat Assoc 113(522):767–779. https://doi.org/10.1080/01621459.2017.1285776
Cameron AC, Miller DL (2015) A practitioner’s guide to cluster-robust inference. J Hum Resour 50(2):371–372. https://doi.org/10.3368/jhr.50.2.317
Cameron AC, Gelbach JB, Miller DL (2008) Bootstrap-based improvements for inference with clustered errors. Rev Econ Stat 90(3):414–427. https://doi.org/10.1162/rest.90.3.414
Cameron AC, Gelbach JB, Miller DL (2011) Robust inference with multiway clustering. J Bus Econ Stat 29(2):238–249. https://doi.org/10.1198/jbes.2010.07136
Card NA (2012) Applied meta-analysis for social science research. The Guilford Press, London
Card D, Krueger AB (1995) Time-series minimum-wage studies: a meta-analysis. Am Econ Rev 85(2):238–243
Card D, Kluve J, Weber A (2010) Active labour market policy evaluations: a meta-analysis*. Econ J 120(548):F452–F477. https://doi.org/10.1111/j.1468-0297.2010.02387.x
Card D, Kluve J, Weber A (2017) What works? A meta analysis of recent active labor market program evaluations. J Eur Econ Assoc 16(3):894–931. https://doi.org/10.1093/jeea/jvx028
Cattaneo MD, Jansson M, Ma X (2018) Manipulation testing based on density discontinuity. Stata J 18(1):234–261. https://doi.org/10.1177/1536867X1801800115
Cattaneo MD, Jansson M, Ma X (2020) Simple local polynomial density estimators. J Am Stat Assoc 115(531):1449–1455. https://doi.org/10.1080/01621459.2019.1635480
Cazachevici A, Havranek T, Horvath R (2020) Remittances and economic growth: a meta-analysis. World Dev 134:105021. https://doi.org/10.1016/j.worlddev.2020.105021
Cohen J (1988) Statistical power analysis in the behavioral sciences. Lawrence Erlbaum Associates, Hillsdale
Cohn LD, Becker BJ (2003) How meta-analysis increases statistical power. Psychol Methods 8(3):243–253. https://doi.org/10.1037/1082-989X.8.3.243
Correia S (2014) Reghdfe: Stata module to perform linear or instrumental-variable regression absorbing any number of high-dimensional fixed effects, URL https://ideas.repec.org/c/boc/bocode/s457874.html, statistical Software Components S457874, Boston College Department of Economics
De Long JB, Lang K (1992) Are all economic hypotheses false? J Polit Econ 100(6):1257–1272. https://doi.org/10.1086/261860
De Luca G, Magnus JR (2011) Bayesian model averaging and weighted-average least squares: equivariance, stability, and numerical issues. Stata J 11(4):518–544. https://doi.org/10.1177/1536867X1201100402
Doucouliagos C (1995) Worker participation and productivity in labor-managed and participatory capitalist firms: a meta-analysis. ILR Rev 49(1):58–77. https://doi.org/10.1177/001979399504900104
Doucouliagos H (2011) How large is large? Preliminary and relative guidelines for interpreting partial correlations in economics, working papers SWP 2011/5. Deakin University, Department of Economics
Doucouliagos H, Laroche P (2003) What do unions do to productivity? a meta-analysis. Industrial Relations. J Econ Soc 42(4):650–691. https://doi.org/10.1111/1468-232X.00310
Doucouliagos H, Laroche P (2009) Unions and profits: a meta-regression analysis. Industrial Relations. J Econ Soc 48(1):146–184. https://doi.org/10.1111/j.1468-232X.2008.00549.x
Doucouliagos H, Stanley TD (2012) Are all economic facts greatly exaggerated? Theory competition and selectivity. J Econ Stud 27(2):314–339. https://doi.org/10.1111/j.1467-6419.2011.00706.x
Drydakis N (2021) Sexual orientation and earnings: a meta-analysis 2012–2020. J Popul Econ 35:409. https://doi.org/10.1007/s00148-021-00862-1
Egger M, Smith GD, Schneider M, Minder C (1997) Bias in meta-analysis detected by a simple, graphical test. Br Med J 315(7109):629–634. https://doi.org/10.1136/bmj.315.7109.629
Eicher TS, Papageorgiou C, Raftery AE (2011) Default priors and predictive performance in Bayesian model averaging, with application to growth determinants. J Appl Econ 26(1):30–55. https://doi.org/10.1002/jae.1112
Eysenck HJ (1978) An exercise on mega-silliness. Am Psychol 33(5):517
Feinstein AR (1995) Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol 48(1):71–79. https://doi.org/10.1016/0895-4356(94)00110-c
Filomena M, Picchio M (2022) Retirement and health outcomes in a meta-analytical framework. J Econ Surv. (Early View) https://doi.org/10.1111/joes.12527
Fisher RA (1934) Statistical methods for research workers, 5th edn. Oliver and Boyd LTD, Edinburgh
Franco A, Malhotra N, Simonovits G (2014) Publication bias in the social sciences: unlocking the file drawer. Science 345(6203):1502–1505. https://doi.org/10.1126/science.1255484
Gerber A, Malhotra N (2008) Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quart J Politic Sci 3(3):313–326. https://doi.org/10.1561/100.00008024
Glass G (1976) Primary, secondary, and meta-analysis of research. Educ Res 5(10):3–8. https://doi.org/10.2307/1174772
Gu A, Yoo HI (2019) Vcemway: a one-stop solution for robust inference with multiway clustering. Stata J 19(4):900–912. https://doi.org/10.1177/1536867X19893637
Haelermans C, Borghans L (2012) Wage effects of on-the-job training: a meta-analysis. Br J Ind Relat 50(2):502–528. https://doi.org/10.1111/j.1467-8543.2012.00890.x
Hansen BE (2007) Least squares model averaging. Econometrica 75(4):1175–1189. https://doi.org/10.1111/j.1468-0262.2007.00785.x
Havranek T, Sokolova A (2020) Do consumers really follow a rule of thumb? Three thousand estimates from 144 studies say “probably not”. Rev Econ Dyn 35:97–122. https://doi.org/10.1016/j.red.2019.05.004
Havranek T, Horvath R, Irsova Z, Rusnak M (2015) Cross-country heterogeneity in intertemporal substitution. J Int Econ 96(1):100–118. https://doi.org/10.1016/j.jinteco.2015.01.012
Havránek T, Stanley TD, Doucouliagos H, Bom P, Geyer-Klingeberg J, Iwasaki I, Reed WR, Rost K, van Aert RCM (2020) Reporting guidelines for meta-analysis in economics. J Econ Surv 34(3):469–475. https://doi.org/10.1111/joes.12363
Hedges LV, Olkin I (1980) Vote-counting methods in research synthesis. Psychol Bull 88(2):359–369. https://doi.org/10.1037/0033-2909.88.2.359
Heimberger P (2021) Does employment protection affect unemployment? A meta-analysis. Oxford Econ Paper 73(3):982–1007. https://doi.org/10.1093/oep/gpaa037
Hendry DF, Hans-Martin K (2004) We ran one regression. Oxf Bull Econ Stat 66(5):799–810. https://doi.org/10.1111/j.1468-0084.2004.102_1.x
Ioannidis JPA (2016) The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The MilkBank Quarterly 94(3):485–514. https://doi.org/10.1111/1468-0009.12210
Kennedy PE (2002) Sinning in the basement: what are the rules? The ten commandments of applied econometrics. J Econ Surv 16(4):569–589. https://doi.org/10.1111/1467-6419.00179
Kluve J (2010) The effectiveness of European active labor market programs. Labour Econ 17(6):904–918. https://doi.org/10.1016/j.labeco.2010.02.004
Kuhn A (2018) The complex effects of retirement on health. IZA World of Labor 430. https://doi.org/10.15185/izawol.430
Langan D, Higgins JP, Jackson D, Bowden J, Veroniki AA, Kontopantelis E, Viechtbauer W, Simmonds M (2019) A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Res Synth Methods 10(1):83–98. https://doi.org/10.1002/jrsm.1316
Laroche P (2016) A meta-analysis of the union-job satisfaction relationship. Br J Ind Relat 55(4):709–741. https://doi.org/10.1111/bjir.12193
Liang KY, Zeger SL (1986) Longitudinal data analysis using generalized linear models. Biometrika 73(1):13–22. https://doi.org/10.1093/biomet/73.1.13
Magnus JR, De Luca G (2016) Weighted-average least squares (WALS): a survey. J Econ Surv 30(1):117–148. https://doi.org/10.1111/joes.12094
Magnus JR, Powell O, Prüfer P (2010) A comparison of two model averaging techniques with an application to growth empirics. J Econ 154(2):139–153. https://doi.org/10.1016/j.jeconom.2009.07.004
Mallows CL (1973) Some comments on Cp. Technometrics 15(4):661–675. https://doi.org/10.2307/1267380
McCrary J (2008) Manipulation of the running variable in the regression discontinuity design: a density test. J Econ 142:698–714. https://doi.org/10.1016/j.jeconom.2007.05.005
Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Br Med J 339:b2535. https://doi.org/10.1136/bmj.b2535
Nishimura Y, Oikawa M, Motegi H (2018) What explains the difference in the effect of retirement on health? Evidence from global aging data. J Econ Surv 32(3):792–847. https://doi.org/10.1111/joes.12215
Pearson K (1904) Report on certain enteric fever inoculation statistics. Br Med J 2(2288):1243–1246
Purgato M, Adams CE (2012) Heterogeneity: the issue of apples, oranges and fruit pie. Epidemiol Psychiatr Sci 21:27–29. https://doi.org/10.1017/S2045796011000643
Roodman D, Nielsen M, MacKinnon JG, Webb MD (2019) Fast and wild: bootstrap inference in stata using boottest. Stata J 19(1):4–60. https://doi.org/10.1177/1536867X19830877
Rosenthal R (1979) The ‘file drawer problem’ and tolerance for null results. Psychol Bull 86(3):638–641. https://doi.org/10.1037/0033-2909.86.3.638
Rosenthal R (1991) Meta-analytic procedures for social research, revised edn. Sage Publications, Newbury Park
Rosenthal R, DiMatteo MR (2001) Meta-analysis: recent developments in quantitative methods for literature reviews. Annu Rev Psychol 52:59–82. https://doi.org/10.1146/annurev.psych.52.1.59
Sala-i-Martin XX (1997) I just ran two million regressions. Am Econ Rev 87(2):178–183
Sharpe D (1997) Of apples and oranges, file drawers and garbage: why validity issue in meta-analysis will not go away. Clin Psychol Rev 17(8):881–901. https://doi.org/10.1016/S0272-7358(97)00056-1
Simonsohn A, Nelson LD (2014) P-curve: a key to the file-drawer. J Exp Psychol Gen 143(2):534–547. https://doi.org/10.1037/a0033242
Simonsohn A, Nelson LD, Simmons JP (2014) P-curve and effect size: correcting for publication bias using only significant results. Perspect Psychol Sci 9(6):666–681. https://doi.org/10.1177/1745691614553988
Stanley TD (2005) Beyond publication bias. J Econ Surv 19(3):309–345. https://doi.org/10.1111/j.0950-0804.2005.00250.x
Stanley TD (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf Bull Econ Stat 70(1):103–127. https://doi.org/10.1111/j.1468-0084.2007.00487.x
Stanley TD, Doucouliagos H (2010) Picture this: a simple graph that reveals much ado about research. J Econ Surv 24(1):170–191. https://doi.org/10.1111/j.1467-6419.2009.00593.x
Stanley TD, Doucouliagos H (2012) Meta-regression analysis in economics and business. Rouledge, Abingdon
Stanley TD, Doucouliagos H (2014) Meta-regression approximations to reduce publication selection bias. Res Synth Methods 5(1):60–78. https://doi.org/10.1002/jrsm.1095
Stanley TD, Doucouliagos H (2015) Neither fixed nor random: weighted least squares meta-analysis. Stat Med 34(13):2116–2127. https://doi.org/10.1002/sim.6481
Stanley TD, Doucouliagos H, Giles M, Heckemeyer JH, Johnston RJ, Laroche P, Nelson JP, Paldam M, Poot J, Pugh G et al (2013) Meta-analysis of economics research reporting guidelines. J Econ Surv 27(2):390–394. https://doi.org/10.1111/joes.12008
Steel MFJ (2020) Model averaging and its use in economics. J Econ Lit 58(3):644–719. https://doi.org/10.1257/jel.20191385
Sutton AJ, Abrams KR, Jones DR, Jones DR, Sheldon TA, Song F (2000) Methods for meta-analysis in medical research. Wiley, Chichester
Turner RM, Bird SM, Higgins JPT (2013) The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS One 8(3):e59202. https://doi.org/10.1371/journal.pone.0059202
Veroniki AA, Jackson D, Viechtbauer W, Bender R, Bowden J, Knapp G, Kuss O, Higgins JP, Langan D, Salanti G (2016) Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods 7(1):55–79. https://doi.org/10.1002/jrsm.1164
Vooren M, Haelermans C, Groot W, van den Brink HM (2019) The effectiveness of active labor market policies: a meta-analysis. J Econ Surv 33(1):125–149. https://doi.org/10.1111/joes.12269
Weichselbaumer D, Winter-Ebmer R (2005) A meta-analysis of the international gender wage gap. J Econ Surv 19(3):479–511. https://doi.org/10.1111/j.0950-0804.2005.00256.x
Wilson N, McDaid S (2021) The mental health effects of a universal basic income: a synthesis of the evidence from previous pilots. Soc Sci Med 287:114374. https://doi.org/10.1016/j.socscimed.2021.114374
Xue X, Reed RR, Menclova A (2020) Social capital and health: a meta-analysis. J Health Econ 72:102317. https://doi.org/10.1016/j.jhealeco.2020.102317
Xue X, Cheng M, Zhang W (2021) Does education really improve health? A meta-analysis. J Econ Surv 35(1):71–105. https://doi.org/10.1111/joes.12399
Acknowledgments
Responsible Section Editor: E. Rettore. The chapter has benefitted from valuable comments by the editors, anonymous referees, and Mattia Filomena and Michele Ubaldi. There is no conflict of interest.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Section Editor information
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this entry
Cite this entry
Picchio, M. (2022). Meta-Analysis. In: Zimmermann, K.F. (eds) Handbook of Labor, Human Resources and Population Economics. Springer, Cham. https://doi.org/10.1007/978-3-319-57365-6_350-1
Download citation
DOI: https://doi.org/10.1007/978-3-319-57365-6_350-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-57365-6
Online ISBN: 978-3-319-57365-6
eBook Packages: Springer Reference Economics and FinanceReference Module Humanities and Social SciencesReference Module Business, Economics and Social Sciences