Environmental and Resource Economics

, Volume 55, Issue 3, pp 357–386 | Cite as

Due Diligence in Meta-analyses to Support Benefit Transfers

  • Kevin J. Boyle
  • Christopher F. ParmeterEmail author
  • Brent B. Boehlert
  • Robert W. Paterson


Meta-analyses are becoming a popular tool for supporting benefit transfers, but the availability of studies is a direct consequence of policy issues, research funding, and investigator interest. We investigate fragility versus robustness of the meta-equation by considering sample selection, removing one observation or study at a time with replacement, and removing/adding regressors. Several key variables are found to be robust, strengthening the argument for their use in policy prescriptions. The key insights are that these methods can be used to parse meta-data to identify the most appropriate set(s) of observations and regressors to support literature evaluations, benefit transfers and other practical applications using statical summaries of empirical data.


Horizontal robustness Vertical robustness Leave-one-out Regression diagnostics Sample selection Meta-equation 


  1. Bateman I, Jones A (2003) Contrasting conventional with multi-level modeling approaches to meta-analysis: expectation consistency in U.K. woodland recreation values. Land Econ 79:235–258CrossRefGoogle Scholar
  2. Bell F, Sorensen PE, Leeworthy VR (1982) The economic impact and valuation of saltwater recreational fisheries in Florida, technical report, report number 47. Florida Sea Grant CollegeGoogle Scholar
  3. Belsley D, Kuh E, Welsch RE (1980) Regression diagnostics. Wiley, New YorkCrossRefGoogle Scholar
  4. Bergstrom JC, Taylor LO (2006) Using meta-analysis for benefits transfer: theory and practice. Ecol Econ 60(2):351–360CrossRefGoogle Scholar
  5. Bishop RC, Boyle KJ, Welsh MP, Baumgartner RM, Rathbun PR (1987) Glen canyon dam releases and downstream recreation: an analysis of user preferences and economic values, technical report, Glen Canyon Environmental Studies. U.S., Bureau of ReclamationGoogle Scholar
  6. Bockstael N, Graefe A, Strand I, Caldwell L (1986) Economic analysis of artificial reefs: a pilot study of selected valuation methodologies, technical report, technical report 6. Artificial Reef Development CenterGoogle Scholar
  7. Boyle KJ, Bishop R, Caudill J, Charbonneau J, Larson D, Markowski MA, Unsowrth RE, Paterson RW (1998) A database of sport fishing values, technical report. U.S., Department of the Interior, Rish and Wildlife Servie Economics DivisionGoogle Scholar
  8. Boyle KJ, Kuminoff NV, Parmeter CF, Pope JC (2009) Necessary conditions for valid benefit transfers. Am J Agric Econ 91(5):1328–1334CrossRefGoogle Scholar
  9. Brooks R (1990) Montana bioeconomics study: a contingent valuation assessment of angler attitudes and economic benefits. Technical report, Montana Department of Fish, Wildlife and ParksGoogle Scholar
  10. Brooks R (1991) Montana bioeconomics study: warm water fishing in Montana: a contingent valuation assessment of angler attitudes and economic benefits for selected waters statewide. Technical report, Montana Department of Fish, Wildlife and ParksGoogle Scholar
  11. Brouwer R, Spaninks FA (1999) The validity of environmental benefits transfer: further empirical testing. Environ Resour Econ 14(1):95–117CrossRefGoogle Scholar
  12. Carson RT, Mitchell RC, Hanemann WM, Kopp RJ, Presser S, Ruud PA (1992) A contingent valuation study of lost passive use values resulting from the exxon valdez oil spill. Technical report, Report to the Attorney General, State of AlaskaGoogle Scholar
  13. Connelly NA, Brown TL, Knuth BA (1988) New york statewide angler survey. Technical report, NY State Department of Environmental Conservation, Division of Fish and WildlifeGoogle Scholar
  14. Cook RD, Weisberg S (1982) Residuals and influence in regression. Chapman and Hall, New YorkGoogle Scholar
  15. Crutchfield JA, Schelle K (1978) An economic analysis of washington ocean recreational salmon fishing with particular emphasis on the role played by the charter vessel industry. Technical report, Pacific Fishery Management Council, National Marine Fisheries CouncilGoogle Scholar
  16. Davidson R, MacKinnon JG (2004) Econometric theory and methods. OxfordGoogle Scholar
  17. Donnelly DM, Loomis JB, Sorg CF, Nelson LJ (1985) Net economic value of recreational steelhead fishing in idaho, technical report, resource bulletin RM-9. U.S., Forest Service Rocky Mountain Forest and Range Experiment StationGoogle Scholar
  18. Duffield J, Neher CJ, Brown TC (1992) Recreation benefits of instream flow: application to Montana’s Big Hole and Bbitterroot rivers. Water Resour Res 28:2169–2181CrossRefGoogle Scholar
  19. EPA US (2006) Report of the EPA work group on VSL meta-analysis. NCEE-0494. National Center for Environmental Economics, EPA, WashingtonGoogle Scholar
  20. Harris CC (1983), Assessing the validity of economic methods for evaluating sport fishery benefits: a behavioral approach, PhD thesis, University of MichiganGoogle Scholar
  21. Hoehn JP (2006) Methods to address selection effects in the meta-regression transfer of ecosystem values. Ecol Econ 60(2):389–398CrossRefGoogle Scholar
  22. Johnson DM (1989) Economic benefits of alternative fishery management programs, PhD thesis, Colorado State UniversityGoogle Scholar
  23. Johnston RJ, Besedin EY, Iovanna R, Miller CJ, Wardwell RF, Ranson MH (2005) Systematic variation in willingness to pay for aquatic resource improvements and implications for benefit transfer: a meta-analysis. Can J Agric Econ 53:221–248CrossRefGoogle Scholar
  24. Johnston RJ, Rosenberger RS (2010) Methods, trends and controversies in contemporary benefit transfer. J Econ Surv 24:479–510Google Scholar
  25. Kaul S, Boyle KJ, Kuminoff NV, Parmeter CF, Pope JC (2013) What can we learn from benefit transfer errors? Evidence from 20 years of research on convergent validity. J Environ Econ Manag (Forthcoming)Google Scholar
  26. Kirchoff S, Colby BG, LaFrance JT (1997) Evaluating the performance of benefit transfer: an empirical inquiry. J Environ Econ Manag 33:75–93CrossRefGoogle Scholar
  27. Leamer EE (1983) Let’s take the con out of econometrics. Am Econ Rev 73(1):31–43Google Scholar
  28. Levine R, Renelt D (1992) A sensitivity analysis of cross-country growth regressions. Am Econ Rev 82(4):942–963Google Scholar
  29. Lindhjem H, Navrud S (2008) How reliable are meta-analyses for international benefit transfers? Ecol Econ 66(2–3):425–435CrossRefGoogle Scholar
  30. Moeltner K, Boyle KJ, Paterson RW (2007) Meta-analysis and benefit transfer for resource valuation: addressing classical challenges with Bayesian modeling. J Environ Econ Manag 53(2):250–269CrossRefGoogle Scholar
  31. Moeltner K, Woodward R (2009) Meta-functional benefit transfer for wetland valuation: making the most of small samples. Environ Resour Econ 42(1):89–108CrossRefGoogle Scholar
  32. Nelson JP, Kennedy PE (2009) The use (and abuse) of meta-analysis in environmental and natural resource economics: an assessment. Environ Resour Econ 42(3):345–377CrossRefGoogle Scholar
  33. Proietti T (2003) Leave-K-Out diagnostics in state-space models. J Time Ser Anal 214(2):221–236CrossRefGoogle Scholar
  34. Racine JS (1997) Feasible cross-validatory model selection for general stationary processes. J Appl Econ 12:169–179CrossRefGoogle Scholar
  35. Richardson L, Loomis J (2009) The total economic value of threatened, endangered and rare species: an updated meta-analysis. Ecol Econ 68:1535–1548CrossRefGoogle Scholar
  36. Rosenberger RS, Johnston RJ (2009) Selection effects in meta-analysis and benefit transfer: avoiding unintended consequences. Land Econ 85:410–428Google Scholar
  37. Rosenberger RS, Loomis JB (2000) Using meta-analysis for benefit transfer: in-sample convergent validity test of an outdoor recreation database. Water Resour Res 36(4):1097–1107CrossRefGoogle Scholar
  38. Rosenberger RS, Stanley TD (2006) Measurement, generalization and publication: sources of error in benefit transfers and their management. Ecol Econ 60(2):372–378CrossRefGoogle Scholar
  39. Rowe RD, Michelsen M, Morey ER (1989) Fishing for atlantic salmon in maine: an investigation into angler activity and management options, technical report, RCG/Hagler. Bailly Inc., BoulderGoogle Scholar
  40. Semykina A, Wooldridge JM (2010) Estimating panel data models in the presence of endogeneity and selection. J Econ 159:375–380Google Scholar
  41. Shrestha RK, Loomis JB (2001) Testing a meta-analysis model for benefit transfer in international outdoor recreation. Ecol Econ 39(1):67–83CrossRefGoogle Scholar
  42. Shrestha RK, Loomis JB (2003) Meta-analytic benefit transfer of outdoor recreation economic values: testing out-of-sample convergent validity. Environ Resour Econ 25(1):79–100CrossRefGoogle Scholar
  43. Smith VK, Huang J (1995) Can markets value air quality? A meta-analysis of hedonic property value models. J Political Econ 103:209–227CrossRefGoogle Scholar
  44. Smith VK, Kaoru Y (1990) Signals or noise? Explaining the variation in recreation benefit estimates. Am J Agric Econ 72:419–433CrossRefGoogle Scholar
  45. Smith VK, Osborne LL (1996) Do contingent valuation estimates pass a “scope” test? A meta-analysis. J Environ Econ Manag 31:287–301CrossRefGoogle Scholar
  46. Smith VK, Pattanayak SK (2002) Is meta-analysis a Noah’s Ark for non-market valuation? Environ Resour Econ 22:271–296CrossRefGoogle Scholar
  47. Sorg CF, Loomis JB, Donnelly DM, Peterson GL, Nelson LJ (1985) New economic value of cold and warm water fishing in idaho, technical report, USDA Forest Service Resource BulletinGoogle Scholar
  48. Stanley TD (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf Bull Econ Stat 70:103–127Google Scholar
  49. Stapler RW, Johnston RJ (2009) Meta-analysis, benefit transfer and methodological covariates: implications for transfer error. Environ Resour Econ 42(2):227–246CrossRefGoogle Scholar
  50. van den Bergh JC, Button KJ, Nijkamp P, Pepping GC (1997) Meta-analysis in environmental economics. Kluwer, DordrechtCrossRefGoogle Scholar
  51. VandenBerg TP, Poe GL, Powell JR (2001) Assessing the accuracy of benefits transfers: evidence from a grouped, multi-site contingent valuation study of ground water quality. In: Bergstrom JC, Boyle KJ, Poe GL (eds) The economic value of water quality, chap 6. Edward Elgar Publishers, CheltenhamGoogle Scholar
  52. vanHoutven G, Powers J, Pattanayak S (2007) Valuing water quality improvements in the United States using meta-analysis: is the glass half-full or half-empty for national policy analysis? Resour Energy Econ 29:206–228CrossRefGoogle Scholar
  53. Wegge TC, Hanemann WM, Strand IE (1986) An economic assessment of marine recreational fishing in southern california, technical report, NOAA-TM-NMFS-SWR-015. NOAA National Marine Fisheries Service Southwest RegionGoogle Scholar
  54. Woodward RT, Wui Y-S (2001) The economic value of wetland services: a meta-analysis. Ecol Econ 37:257–270Google Scholar
  55. Wooldridge JM (1995) Selection corrections for panel data models under conditional mean independence assumptions. J Econom 68(1):115–132CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Kevin J. Boyle
    • 1
  • Christopher F. Parmeter
    • 2
    Email author
  • Brent B. Boehlert
    • 3
  • Robert W. Paterson
    • 3
  1. 1.Virginia Polytechnic Institute and State UniversityBlacksburgUSA
  2. 2.University of MiamiCoral GablesUSA
  3. 3.Industrial Economics, IncorporatedCambridgeUK

Personalised recommendations