Skip to main content
SpringerLink
Log in

Meta-analysis: Statistical Methods

  • Chapter
  • First Online:
Benefit Transfer of Environmental and Resource Values

Part of the book series: The Economics of Non-Market Goods and Resources ((ENGO,volume 14))

Abstract

Meta-analysis is the quantitative synthesis of multiple primary studies containing estimates of similar empirical magnitudes or effect sizes . Meta-analysis allows generalizations about the underlying population of effects and increases the power of statistical tests. Meta-regression analysis can control statistically for factual heterogeneity , methodological diversity, and possible biases among the primary studies. In the context of benefit transfers, meta-analysis can produce reduced-form functions that identify and test systematic influences of study, economic, and resource attributes on willingness to pay and other environmental valuations. This chapter provides an introduction to basic statistical methods employed in meta-analysis , including weighted-averages and meta-regressions . The chapter identifies and discusses solutions to several econometric problems commonly associated with metadata , including heterogeneity, heteroskedasticity , correlated effects , and publication bias . Basic statistical concepts and methods are illustrated using a sample of estimates for the value of a statistical life , including within-sample and out-of-sample forecasts . Benefit-transfer errors are assessed using several alternative statistical measures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Previous articles that discuss methodological issues concerning benefit transfer and meta-analysis include Bergstrom and Taylor (2006), Johnston and Rosenberger (2010), Johnston et al. (2006), Lindhjem and Navrud (2008), Rosenberger and Loomis (2000a), Rosenberger and Phipps (2007), Rosenberger and Stanley (2006), Shrestha and Loomis (2001, 2003), Shrestha et al. (2007), Smith and Pattanayak (2002), Stapler and Johnston (2009), and Van Houtven et al. (2007).

  2. 2.

    Borenstein et al. (2009) is an excellent introduction to basic statistical models employed in meta-analysis; see also Cooper (2010) and Cooper et al. (2009). Specialized software available for meta-analysis includes CMA (Biostat 2005), SAS, and Stata (Steme 2009). Many basic calculations can be implemented using Excel or other statistical software, although standard errors are not always correctly computed in non-specialized software packages (Konstantopoulos and Hedges 2009; Rhodes 2012). Monte Carlo comparisons of alternative models presented below are found in Rhodes (2012).

  3. 3.

    As noted by White (2009, p. 61), “one does hear of [and encounter] innocents who think that database or Web searches retrieve everything that exists on a topic.”

  4. 4.

    For the RES model, the prediction interval describes the possible distribution of true effect sizes, given estimates of the between-study variance and the RES variance; see Borenstein et al. (2009). For these data, the 95 % prediction interval is $9.77 ± 1.96 (21.4 + 1.06)1/2 or $0.48–19.06 million per life.

  5. 5.

    An alternative measure of heterogeneity is I 2 = ((Q − df)/ Q) × 100, which is 92.5 % for the VSL data. This statistic describes the proportion of observed variance due to real differences in the estimates rather than chance, i.e., the excess dispersion divided by the total dispersion. Values above 75 % are considered “high.”

  6. 6.

    Smith and Kaoru (1990, p. 425) express doubts about the use of variance weights for a meta-regression. They argue that the “weighting implicitly assumes that the estimates based on incorrect modeling assumptions remain unbiased but simply have less informational content.” Alternatives include use of robust standard errors (e.g., Huber-White); weighting by the sample size; inclusion of the sample size as a regressor; and inclusion of regressors that describe the possible biases. See Nelson and Kennedy (2009) and Rhodes (2012) for additional discussion.

  7. 7.

    As discussed by Kennedy (2008, p. 339), misspecification is not always a disaster. Although the estimated coefficients are biased, a parsimonious model can still provide better forecasts as the biased parameters incorporate some of the information in the unobserved or omitted variables. Many existing meta-analyses in environmental economics focus on “taking stock of the literature” through parameter estimation for a host of explanatory variables, but this does not guarantee that the models can generate good forecasts for a benefit-function transfer.

  8. 8.

    Point estimates for the VSL are usually preferred for ethical reasons, although different life-saving benefits may be given different values (Kenkel 2003). Meta-regressions can be used to correct for methodological dispersion, obtain a summary value for the income elasticity, correct for publication bias, examine the influence of labor market imperfections, or examine situational differences in VSL. A range of estimates also is valuable for sensitivity analysis in benefit-cost studies and other project evaluations.

  9. 9.

    I also experimented with a dummy variable for four studies where the standard errors were obtained by an indirect regression on sample size. The dummy coefficient for these studies was insignificant. Although the precision variable corrects for publication bias, its interpretation is somewhat different in the random-effect regressions.

  10. 10.

    Using their metadata for 32 VSL estimates, I also estimated the MM model in Bellavance et al. (2009, p. 455). Using metareg, I could (approximately) reproduce their coefficient and between-study variance estimates, but most of my t-statistics were smaller. The REML model for these data failed to converge. Their reported income elasticity estimates ranged from 0.72–0.86 for a restricted sample and 0.84–1.08 for the full sample. A VSL income elasticity of 0.7–0.9 is reported in Lindhjem et al. (2011), which is reduced to 0.3–0.4 for restricted samples.

  11. 11.

    Rhodes (2012) argues that the seemingly-unrelated regressions (SUR) model can be used when studies report multiple outcomes from the same data set.

  12. 12.

    Florax (2002) proposes use of Moran’s I and Moran’s scatter-plot as methods for visualizing within-study and between-study dependence. Stata also contains a number of tests and procedures for clustered data.

  13. 13.

    Nelson (2011) illustrates the use of a truncated regression model for dealing with metadata containing selection bias; see Wooldridge (2006) for an introduction to this model.

References

  • Bellavance, F., Dionne, G., & Lebeau, M. (2009). The value of a statistical life: A meta-analysis with a mixed effects regression model. Journal of Health Economics, 28, 444–464.

    Article  Google Scholar 

  • Bergstrom, J. C., & Taylor, L. O. (2006). Using meta-analysis for benefits transfer: Theory and practice. Ecological Economics, 60, 351–360.

    Article  Google Scholar 

  • Biostat (2005). Comprehensive meta-analysis (Version 2.0). Englewood, NJ: Biostat, Inc.

    Google Scholar 

  • Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley.

    Book  Google Scholar 

  • Cameron, A. C., & Trivedi, P. K. (2005). Microeconometrics: Methods and applications. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Cavlovic, T. A., Baker, K. H., Berrens, R. P., & Gawande, L. (2000). A meta-analysis of environmental Kuznets curve studies. Agricultural and Resource Economics Review, 29, 32–42.

    Google Scholar 

  • Cooper, H. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles: Sage.

    Google Scholar 

  • Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

    Google Scholar 

  • Costa, D. L., & Kahn, M. E. (2004). Changes in the value of life, 1940–1980. Journal of Risk and Uncertainty, 29, 159–180.

    Article  Google Scholar 

  • Doucouliagos, H., & Stanley, T. D. (2009). Publication selection bias in minimum-wage research? A meta-regression analysis. British Journal of Industrial Relations, 47, 406–428.

    Article  Google Scholar 

  • Doucouliagos, H., & Stanley, T. D. (2010). Picture this: A simple graph that reveals much ado about research. Journal of Economic Surveys, 24, 170–191.

    Article  Google Scholar 

  • Doucouliagos, H., & Stanley, T. D. (2012). Meta-regression analysis in economics and business. New York: Routledge.

    Google Scholar 

  • Doucouliagos, H., Stanley, T. D., & Giles, M. (2012). Are estimates of the value of a statistical life exaggerated? Journal of Health Economics, 31, 197–208.

    Article  Google Scholar 

  • Fisher, R. A. (1932). Statistical methods for research workers. London: Oliver & Boyd.

    Google Scholar 

  • Florax, R. J. G. M. (2002). Accounting for dependence among study results in meta-analysis: Methodology applications to the valuation and use of natural resources. Retrieved March 23, 2012, from Free University of Amsterdam. Web site: http://dare.ubvu.vu.nl/handle/1871/1643

  • Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351–379.

    Google Scholar 

  • Hedges, L., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando: Academic Press.

    Google Scholar 

  • Hoehn, J. P. (2006). Methods to address selection effects in the meta regression and transfer of ecosystem values. Ecological Economics, 60, 389–398.

    Article  Google Scholar 

  • Hox, J. J., & de Leeuw, E. D. (2003). Multilevel models for meta-analysis. In S. P. Reise & N. Duan (Eds.), Multilevel modeling: Methodological advances, issues, and applications (pp. 90–111). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Jeppesen, T., List, J. A., & Folmer, H. (2002). Environmental regulations and new plant decisions: Evidence from a meta-analysis. Journal of Regional Science, 42, 19–49.

    Article  Google Scholar 

  • Johnston, R. J., Besedin, E. Y., & Ranson, M. H. (2006). Characterizing the effects of valuation methodology in function-based benefits transfer. Ecological Economics, 60, 407–419.

    Article  Google Scholar 

  • Johnston, R. J., Besedin, E. Y., & Wardwell, R. F. (2003). Modeling relationships between use and nonuse value for surface water quality: A meta-analysis. Water Resources Research, 39, 1363–1371.

    Article  Google Scholar 

  • Johnston, R. J., & Rosenberger, R. S. (2010). Methods, trends and controversies in contemporary benefit transfer. Journal of Economic Surveys, 24, 479–510.

    Google Scholar 

  • Kenkel, D. (2003). Using estimates of the value of a statistical life in evaluating consumer policy regulations. Journal of Consumer Policy, 26, 1–21.

    Article  Google Scholar 

  • Kennedy, P. (2008). A guide to econometrics (6th ed.). Malden, MA: Blackwell Publishing.

    Google Scholar 

  • Konstantopoulos, S., & Hedges, L. V. (2009). Analyzing effect sizes: Fixed-effects models. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 279–293). New York: Russell Sage Foundation.

    Google Scholar 

  • Lindhjem, H., & Navrud, S. (2008). How reliable are meta-analyses for international benefit transfer? Ecological Economics, 66, 425–435.

    Article  Google Scholar 

  • Lindhjem, H., Navrud, S., Braathen, N. A., & Biausque, V. (2011). Valuing mortality risk reductions from environmental transport and health policies: A global meta-analysis of stated preferences. Risk Analysis, 31, 1381–1407.

    Article  Google Scholar 

  • Moeltner, K., & Rosenberger, R. S. (2008). Predicting resource policy outcomes via meta-regression: Data space, model space, and the quest for ‘optimal scope’. The BE Journal of Economic Analysis & Policy. DOI:10.2202/1935-1682.2028

  • Nelson, J. P. (2004). Meta-analysis of airport noise and hedonic property values: Problems and prospects. Journal of Transport Economics and Policy, 38, 1–28.

    Google Scholar 

  • Nelson, J. P. (2011). Alcohol marketing, adolescent drinking and publication bias in longitudinal studies: A critical survey using meta-analysis. Journal of Economic Surveys, 25, 191–232.

    Article  Google Scholar 

  • Nelson, J. P., & Kennedy, P. E. (2009). The use (and abuse) of meta-analysis in environmental and resource economics: An assessment. Environmental and Resource Economics, 42, 345–377.

    Article  Google Scholar 

  • Pearson, K. (1904). Report on certain enteric fever inoculation statistics. British Medical Journal, 3, 1243–1246.

    Google Scholar 

  • Randall, A., Kidder, A., & Chen, D.-R. (2008). Meta-analysis for benefits transfer—Toward value estimates for some outputs of multifunctional agriculture. Paper presented at the 12th Congress of the European Association of Agricultural Economics, Ghent. Web site: http://ageconsearch.umn.edu/bitstream/43648/2/013.pdf

  • Rhodes, W. (2012). Meta-analysis: An introduction using regression models. Evaluation Review, 36, 24–71.

    Google Scholar 

  • Roberts, C. J., & Stanley, T. D. (2005). Meta-regression analysis: Issues of publication bias in economics. Oxford: Blackwell.

    Google Scholar 

  • Rosenberger, R. S., & Loomis, J. B. (2000a). Using meta-analysis for benefit transfer: In-sample convergent validity tests of an outdoor recreation database. Water Resources Research, 36, 1097–1107.

    Article  Google Scholar 

  • Rosenberger, R. S., & Loomis, J. B. (2000b). Panel stratification in meta-analysis of economic studies: An investigation of its effects in the recreation valuation literature. Journal of Agricultural and Applied Economics, 32, 459–470.

    Google Scholar 

  • Rosenberger, R. S., & Phipps, T. (2007). Correspondence and convergence in benefit transfer accuracy: Meta-analytic review of the literature. In S. Navrud & R. Ready (Eds.), Environmental value transfer: Issues and methods (pp. 23–43). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Rosenberger, R. S., & Stanley, T. D. (2006). Measurement, generalization, and publication: Sources of error in benefit transfer and their management. Ecological Economics, 60, 372–378.

    Article  Google Scholar 

  • Shrestha, R. K., & Loomis, J. B. (2001). Testing a meta-analysis model for benefit transfer in international outdoor recreation. Ecological Economics, 39, 67–83.

    Article  Google Scholar 

  • Shrestha, R. K., & Loomis, J. B. (2003). Meta-analytic benefit transfer of outdoor recreation economic values: Testing out-of-sample convergent validity. Environmental and Resource Economics, 25, 79–100.

    Article  Google Scholar 

  • Shrestha, R., Rosenberger, R., & Loomis, J. (2007). Benefit transfer using meta-analysis in recreation economic valuation. In S. Navrud & R. Ready (Eds.), Environmental value transfer: Issues and methods (pp. 161–177). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Smith, V. K., & Kaoru, Y. (1990). Signals or noise? Explaining the variation in recreation benefit estimates. American Journal of Agricultural Economics, 72, 419–433.

    Article  Google Scholar 

  • Smith, V. K., & Pattanayak, S. K. (2002). Is meta-analysis a Noah’s arc for non-market valuation? Environmental and Resource Economics, 22, 271–296.

    Article  Google Scholar 

  • Stanley, T. D. (2001). Wheat from chaff: Meta-analysis as quantitative literature review. Journal of Economic Perspectives, 15, 131–150.

    Article  Google Scholar 

  • Stanley, T. D. (2008). Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxford Bulletin of Economics and Statistics, 70, 103–127.

    Google Scholar 

  • Stanley, T. D., & Jarrell, S. B. (1989). Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic Surveys, 3, 161–170.

    Article  Google Scholar 

  • Stanley, T. D., Jarrell, S. B., & Doucouliagos, H. (2010). Could it be better to discard 90 % of the data? A statistical paradox. American Statistician, 64, 70–77.

    Article  Google Scholar 

  • Stanley, T. D. & Rosenberger, R. S. (2009). Are nonmarket values systematically underestimated? Reducing publication bias for benefit transfer. Presented at the 2009 Meta-Analysis of Economic Research Workshop. Retrieved April 28, 2012, from Hendrix College. Web site: http://www.hendrix.edu/uploadedFiles/Departments_and_Programs/Business_and_Economics/AMAES/RootnMRA.pdf

  • Stapler, R. W., & Johnston, R. J. (2009). Meta-analysis, benefit transfer, and methodological covariates: Implications for transfer error. Environmental and Resource Economics, 42, 227–246.

    Article  Google Scholar 

  • Sterne, J. A. C. (2009). Meta-analysis in Stata: An updated collection from the Stata Journal. College Station, TX: Stata Press.

    Google Scholar 

  • U.S. Environmental Protection Agency (2006). SAB advisory on EPA’s issues in valuing mortality risk reductions, EPA-SAB-08-001. Retrieved March 10, 2012, from http://yosemite.epa.gov/ee/epa/eerm.nsf/vwRepNumLookup/EE-0494?OpenDocument

  • U.S. Environmental Protection Agency. (2010). Valuing mortality risk reductions for environmental policy: A white paper. Washington, DC: EPA. Retrieved February 15, 2012, from http://www.epa.gov

  • Van Houtven, G. (2008). Methods for the meta-analysis of willingness-to-pay data. PharmacoEconomics, 26, 901–910.

    Article  Google Scholar 

  • Van Houtven, G., Powers, J., & Pattanayak, S. K. (2007). Valuing water quality improvements in the United States using meta-analysis: Is the glass half-full or half-empty for national policy analysis? Resource and Energy Economics, 29, 206–228.

    Article  Google Scholar 

  • Viscusi, W. K., & Aldy, J. E. (2003). The value of a statistical life: A critical review of market estimates throughout the world. Journal of Risk and Uncertainty, 27, 5–76.

    Article  Google Scholar 

  • Walsh, R. G., Johnson, D. M., & McKean, J. R. (1989). Issues in nonmarket valuation and policy applications: A retrospective glance. Western Journal of Agricultural Economics, 14, 178–188.

    Google Scholar 

  • White, H. D. (2009). Scientific communications and literature retrieval. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 51–71). New York: Russell Sage Foundation.

    Google Scholar 

  • Wooldridge, J. M. (2006). Introductory econometrics: A modern approach (3rd ed.). Mason, OH: Thompson South-Western.

    Google Scholar 

Download references

Acknowledgments

This paper is dedicated to the memory of my colleague and long-time friend, Peter Kennedy, who passed away suddenly in 2010. He was a great source of wisdom about econometric methods and about life. I would also like to acknowledge comments and suggestions received from Ed Coulson, Mark Roberts, and Jim Tybout.

Author information

Authors and Affiliations

  1. Department of Economics, Pennsylvania State University, University Park, PA, USA

    Jon P. Nelson

Authors
  1. Jon P. Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jon P. Nelson .

Editor information

Editors and Affiliations

  1. Department of Economics and George Perkins Marsh Institute, Clark University, Worcester, Massachusetts, USA

    Robert J. Johnston

  2. School of Business and Law, CQUniversity, Rockhampton, Queensland, Australia

    John Rolfe

  3. Dept. of Forest Ecosystems & Society, Oregon State University, Corvallis, Oregon, USA

    Randall S. Rosenberger

  4. Department of Environmental Economics, Vrije Universiteit, Amsterdam, The Netherlands

    Roy Brouwer

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Nelson, J.P. (2015). Meta-analysis: Statistical Methods. In: Johnston, R., Rolfe, J., Rosenberger, R., Brouwer, R. (eds) Benefit Transfer of Environmental and Resource Values. The Economics of Non-Market Goods and Resources, vol 14. Springer, Dordrecht. https://doi.org/10.1007/978-94-017-9930-0_15

Download citation

Publish with us

Policies and ethics

Search

Navigation