Skip to main content
SpringerLink
Log in

Due Diligence in Meta-analyses to Support Benefit Transfers

  • Published:
Environmental and Resource Economics Aims and scope Submit manuscript

Abstract

Meta-analyses are becoming a popular tool for supporting benefit transfers, but the availability of studies is a direct consequence of policy issues, research funding, and investigator interest. We investigate fragility versus robustness of the meta-equation by considering sample selection, removing one observation or study at a time with replacement, and removing/adding regressors. Several key variables are found to be robust, strengthening the argument for their use in policy prescriptions. The key insights are that these methods can be used to parse meta-data to identify the most appropriate set(s) of observations and regressors to support literature evaluations, benefit transfers and other practical applications using statical summaries of empirical data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. The new U.S. EPA Guidelines for Preparing Economic Analyses use the terminology study-case and policy-case, which is a change in the standard terminology that has traditionally used the terms study site and policy site. We follow this new terminology to try to avoid confusion between the literature and the practical applications of benefit transfers. In addition, the use of the term “case” rather than “site” has an important advantage for the benefit-transfer literature. Benefit transfers need not always be transfers to a new physical location (a new site). A transfer can occur at the same location as the original study if values are predicted backward or forward through time, or are simply used to estimate values for a policy question that is different from the objective of the original valuation study. The term case, therefore, recognizes that transfers can be to a new policy-case at the original study location or to a policy case at a different location.

  2. http://yosemite.epa.gov/EE/Epa/eerm.nsf/vwSER/DEC917DAEB820A25852569C40078105B?OpenDocument, accessed January 1, 2010.

  3. Rosenberger and Loomis (2000) and Shrestha and Loomis (2003) are not unique in making this mistake in their meta-analyses. Note, Stapler and Johnston (2009) also mix welfare measures in their meta-analysis. Hoehn (2006) went even further including producer-surplus estimates, hedonic-price estimates, replacement-cost estimates and contingent-valuation estimates in his meta-equation. This is one type of what Rosenberger and Stanley (2006) refer to as generalization error.

  4. Rosenberger and Johnston (2009) identify a number of types of sample selection that include research priority (i.e., what applications are chosen for funding), studies that are published and the value estimates reported, and the selection criteria analysts apply to collect their meta-data (see also Stanley 2008). Thus, sorting data by state is only one method that could potentially be applied to investigate sample selection. However, it is unlikely that sufficient data exists to assess selection along these different criterion, such as grant proposals.

  5. We acknowledge that there are a number of published meta-analyses that pool the valuation data, which we suggest an analyst should decided to include or exclude from the meta-data. Any potential differences in these pooled categories of data controlled by the inclusion of binary variables in meta-regressions to identify their respective effects. Such simple controls are not always appropriate or possible. Consider the Marshallian/Hicksian surplus decision. For a price increase, Hicksian surplus exceeds Marshallian surplus and the converse holds for a price decrease. This dynamic cannot be controlled by a single binary variable. As the number of binary variables is increased to account for interaction effects the dimensionality curse can reduce the efficiency of the meta-regression parameter estimates (fixed \(N\) with increasing \(k\)). In addition, some interaction cells may have very small or be empty, which further complicates estimation. While we do not address this issue here, the reader is directed to Kaul et al. (2013) who discuss this topic in the context of nonparametric estimation of a meta-equation.

  6. This issue is still likely to arise if we used a different criterion to identify potential selection.

  7. Note that we do not use the between correction employed in Wooldridge’s actual Sect. 3.2. This is due to the fact that he suggests estimating a probit for each time period. In our model we are in essence estimating one probit so any time or individual demeaning would remove the inverse Mill’s ratio from the regression.

  8. To our knowledge, bootstrapping in panel data sample selection models with an unbalanced dataset has not been discussed in the econometrics literature; see Semykina and Wooldridge (2010) for a general treatment of sample selection in panel data models. This is an interesting avenue for future research.

  9. It is also relevant to consider the errors, but we focus on the parameter estimates because they provide a clearer picture of leverage and influence and are the relevant statistics for policy analysis.

  10. One could also consider the median absolute deviation as well.

  11. In the empirical results to follow, sample selection is not found to be significant and so we dispense with incorporating \(\gamma \) in our analysis of robustness in this and the following sections.

  12. An alternative measure of overall influence of an observation is the change in the OLS residuals from leaving a single observation out. This index can be calculated as follows,

    $$\begin{aligned} DF_i=\hat{\omega }_i-\hat{\omega }_{(i)}=\frac{h_{ii}\hat{\omega }_i}{1-h_{ii}}=D_i\frac{s^2(1-h_{ii})k}{\hat{\omega }_i}, \end{aligned}$$

    where \(\hat{\omega }_{(i)}\) is the \(i{\mathrm{th}}\) residual, constructed leaving the \(i{\mathrm{th}}\) observation out of the analysis. We elect to use \(D_i\) since in general \(DF_i\) and \(D_i\) yield similar results but \(D_i\) has a more intuitive feel.

  13. See Davidson and MacKinnon (2004, Section 2.6) for more on influence and leverage.

  14. This approach is common in the time-series literature where \(n_k\) consecutive observations would be left out (see Proietti 2003).

  15. The last equality here follows from the fact that

    $$\begin{aligned} I+P^{(t)}+P^{(t)}\left(I-P^{(t)}\right)^{-1}P^{(t)}&= \left(I-P^{(t)}\right)^{-1}\\ P^{(t)}\left(I-P^{(t)}\right)^{-1}+P^{(t)}+P^{(t)} \left(I-P^{(t)}\right)^{-1}P^{(t)}&= 0, \end{aligned}$$

    both of which can be discerned from the matrix equality \(A-A(A+B)^{-1}A= B-B(A+B)^{-1}B\), when \((A+B)^{-1}\) exists (let \(A=I\) and \(B=-P^{(t)}\)).

  16. A heteroscedasticity robust version of this statistic could also be constructed consistent with the concerns of Nelson and Kennedy (2009).

  17. Furthermore, we can extend this analysis to the hypothesis testing domain to determine the largest range of coefficient estimates for which standard hypothesis tests of statistical significance (on \(\beta _A\)) do not reject.

  18. If a study did not explicitly state the year of the estimate, we used either the year the study was published or the year the study was submitted for publication to convert the estimates.

  19. Our sample size is comparable to that of many meta-analyses in the resource economics literature. For example, across eight studies that address a variety of commodities the average sample size was 128 with a median of 84 (Bateman and Jones 2003; Johnston et al. 2005; Richardson and Loomis 2009; Smith and Huang 1995; Smith and Kaoru 1990; Smith and Osborne 1996; vanHoutven et al. 2007; Woodward and Wui 2001.)

  20. The states where a CV study was conducted are California, Colorado, Florida, Idaho, Maine, Montana, New York, South Carolina and Washington.

  21. See https://www.census.gov/prod/www/abs/fishing.html, accessed on January 29, 2010.

  22. In a majority of our leave-one-out analyses, the IMR was not statistically significant at the 10 % level, further drawing into question the statistical robustness of selection in general to the appearance of a given observation (study). In Hoehn’s (2006) work, the IMR is strongly significant (5 % level) with fewer observations than we use.

  23. The Bell study produced identical results to the benchmark estimates when we dropped it for all variables whereas the Crutchfield study produced the same results for all variables except the salt variable, which deviated by 106 %, and then when averaged across the total number of variables, produced the 8 % reported.

References

  • Bateman I, Jones A (2003) Contrasting conventional with multi-level modeling approaches to meta-analysis: expectation consistency in U.K. woodland recreation values. Land Econ 79:235–258

    Article  Google Scholar 

  • Bell F, Sorensen PE, Leeworthy VR (1982) The economic impact and valuation of saltwater recreational fisheries in Florida, technical report, report number 47. Florida Sea Grant College

  • Belsley D, Kuh E, Welsch RE (1980) Regression diagnostics. Wiley, New York

    Book  Google Scholar 

  • Bergstrom JC, Taylor LO (2006) Using meta-analysis for benefits transfer: theory and practice. Ecol Econ 60(2):351–360

    Article  Google Scholar 

  • Bishop RC, Boyle KJ, Welsh MP, Baumgartner RM, Rathbun PR (1987) Glen canyon dam releases and downstream recreation: an analysis of user preferences and economic values, technical report, Glen Canyon Environmental Studies. U.S., Bureau of Reclamation

  • Bockstael N, Graefe A, Strand I, Caldwell L (1986) Economic analysis of artificial reefs: a pilot study of selected valuation methodologies, technical report, technical report 6. Artificial Reef Development Center

  • Boyle KJ, Bishop R, Caudill J, Charbonneau J, Larson D, Markowski MA, Unsowrth RE, Paterson RW (1998) A database of sport fishing values, technical report. U.S., Department of the Interior, Rish and Wildlife Servie Economics Division

  • Boyle KJ, Kuminoff NV, Parmeter CF, Pope JC (2009) Necessary conditions for valid benefit transfers. Am J Agric Econ 91(5):1328–1334

    Article  Google Scholar 

  • Brooks R (1990) Montana bioeconomics study: a contingent valuation assessment of angler attitudes and economic benefits. Technical report, Montana Department of Fish, Wildlife and Parks

  • Brooks R (1991) Montana bioeconomics study: warm water fishing in Montana: a contingent valuation assessment of angler attitudes and economic benefits for selected waters statewide. Technical report, Montana Department of Fish, Wildlife and Parks

  • Brouwer R, Spaninks FA (1999) The validity of environmental benefits transfer: further empirical testing. Environ Resour Econ 14(1):95–117

    Article  Google Scholar 

  • Carson RT, Mitchell RC, Hanemann WM, Kopp RJ, Presser S, Ruud PA (1992) A contingent valuation study of lost passive use values resulting from the exxon valdez oil spill. Technical report, Report to the Attorney General, State of Alaska

  • Connelly NA, Brown TL, Knuth BA (1988) New york statewide angler survey. Technical report, NY State Department of Environmental Conservation, Division of Fish and Wildlife

  • Cook RD, Weisberg S (1982) Residuals and influence in regression. Chapman and Hall, New York

    Google Scholar 

  • Crutchfield JA, Schelle K (1978) An economic analysis of washington ocean recreational salmon fishing with particular emphasis on the role played by the charter vessel industry. Technical report, Pacific Fishery Management Council, National Marine Fisheries Council

  • Davidson R, MacKinnon JG (2004) Econometric theory and methods. Oxford

  • Donnelly DM, Loomis JB, Sorg CF, Nelson LJ (1985) Net economic value of recreational steelhead fishing in idaho, technical report, resource bulletin RM-9. U.S., Forest Service Rocky Mountain Forest and Range Experiment Station

  • Duffield J, Neher CJ, Brown TC (1992) Recreation benefits of instream flow: application to Montana’s Big Hole and Bbitterroot rivers. Water Resour Res 28:2169–2181

    Article  Google Scholar 

  • EPA US (2006) Report of the EPA work group on VSL meta-analysis. NCEE-0494. National Center for Environmental Economics, EPA, Washington

  • Harris CC (1983), Assessing the validity of economic methods for evaluating sport fishery benefits: a behavioral approach, PhD thesis, University of Michigan

  • Hoehn JP (2006) Methods to address selection effects in the meta-regression transfer of ecosystem values. Ecol Econ 60(2):389–398

    Article  Google Scholar 

  • Johnson DM (1989) Economic benefits of alternative fishery management programs, PhD thesis, Colorado State University

  • Johnston RJ, Besedin EY, Iovanna R, Miller CJ, Wardwell RF, Ranson MH (2005) Systematic variation in willingness to pay for aquatic resource improvements and implications for benefit transfer: a meta-analysis. Can J Agric Econ 53:221–248

    Article  Google Scholar 

  • Johnston RJ, Rosenberger RS (2010) Methods, trends and controversies in contemporary benefit transfer. J Econ Surv 24:479–510

    Google Scholar 

  • Kaul S, Boyle KJ, Kuminoff NV, Parmeter CF, Pope JC (2013) What can we learn from benefit transfer errors? Evidence from 20 years of research on convergent validity. J Environ Econ Manag (Forthcoming)

  • Kirchoff S, Colby BG, LaFrance JT (1997) Evaluating the performance of benefit transfer: an empirical inquiry. J Environ Econ Manag 33:75–93

    Article  Google Scholar 

  • Leamer EE (1983) Let’s take the con out of econometrics. Am Econ Rev 73(1):31–43

    Google Scholar 

  • Levine R, Renelt D (1992) A sensitivity analysis of cross-country growth regressions. Am Econ Rev 82(4):942–963

    Google Scholar 

  • Lindhjem H, Navrud S (2008) How reliable are meta-analyses for international benefit transfers? Ecol Econ 66(2–3):425–435

    Article  Google Scholar 

  • Moeltner K, Boyle KJ, Paterson RW (2007) Meta-analysis and benefit transfer for resource valuation: addressing classical challenges with Bayesian modeling. J Environ Econ Manag 53(2):250–269

    Article  Google Scholar 

  • Moeltner K, Woodward R (2009) Meta-functional benefit transfer for wetland valuation: making the most of small samples. Environ Resour Econ 42(1):89–108

    Article  Google Scholar 

  • Nelson JP, Kennedy PE (2009) The use (and abuse) of meta-analysis in environmental and natural resource economics: an assessment. Environ Resour Econ 42(3):345–377

    Article  Google Scholar 

  • Proietti T (2003) Leave-K-Out diagnostics in state-space models. J Time Ser Anal 214(2):221–236

    Article  Google Scholar 

  • Racine JS (1997) Feasible cross-validatory model selection for general stationary processes. J Appl Econ 12:169–179

    Article  Google Scholar 

  • Richardson L, Loomis J (2009) The total economic value of threatened, endangered and rare species: an updated meta-analysis. Ecol Econ 68:1535–1548

    Article  Google Scholar 

  • Rosenberger RS, Johnston RJ (2009) Selection effects in meta-analysis and benefit transfer: avoiding unintended consequences. Land Econ 85:410–428

    Google Scholar 

  • Rosenberger RS, Loomis JB (2000) Using meta-analysis for benefit transfer: in-sample convergent validity test of an outdoor recreation database. Water Resour Res 36(4):1097–1107

    Article  Google Scholar 

  • Rosenberger RS, Stanley TD (2006) Measurement, generalization and publication: sources of error in benefit transfers and their management. Ecol Econ 60(2):372–378

    Article  Google Scholar 

  • Rowe RD, Michelsen M, Morey ER (1989) Fishing for atlantic salmon in maine: an investigation into angler activity and management options, technical report, RCG/Hagler. Bailly Inc., Boulder

    Google Scholar 

  • Semykina A, Wooldridge JM (2010) Estimating panel data models in the presence of endogeneity and selection. J Econ 159:375–380

    Google Scholar 

  • Shrestha RK, Loomis JB (2001) Testing a meta-analysis model for benefit transfer in international outdoor recreation. Ecol Econ 39(1):67–83

    Article  Google Scholar 

  • Shrestha RK, Loomis JB (2003) Meta-analytic benefit transfer of outdoor recreation economic values: testing out-of-sample convergent validity. Environ Resour Econ 25(1):79–100

    Article  Google Scholar 

  • Smith VK, Huang J (1995) Can markets value air quality? A meta-analysis of hedonic property value models. J Political Econ 103:209–227

    Article  Google Scholar 

  • Smith VK, Kaoru Y (1990) Signals or noise? Explaining the variation in recreation benefit estimates. Am J Agric Econ 72:419–433

    Article  Google Scholar 

  • Smith VK, Osborne LL (1996) Do contingent valuation estimates pass a “scope” test? A meta-analysis. J Environ Econ Manag 31:287–301

    Article  Google Scholar 

  • Smith VK, Pattanayak SK (2002) Is meta-analysis a Noah’s Ark for non-market valuation? Environ Resour Econ 22:271–296

    Article  Google Scholar 

  • Sorg CF, Loomis JB, Donnelly DM, Peterson GL, Nelson LJ (1985) New economic value of cold and warm water fishing in idaho, technical report, USDA Forest Service Resource Bulletin

  • Stanley TD (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf Bull Econ Stat 70:103–127

    Google Scholar 

  • Stapler RW, Johnston RJ (2009) Meta-analysis, benefit transfer and methodological covariates: implications for transfer error. Environ Resour Econ 42(2):227–246

    Article  Google Scholar 

  • van den Bergh JC, Button KJ, Nijkamp P, Pepping GC (1997) Meta-analysis in environmental economics. Kluwer, Dordrecht

    Book  Google Scholar 

  • VandenBerg TP, Poe GL, Powell JR (2001) Assessing the accuracy of benefits transfers: evidence from a grouped, multi-site contingent valuation study of ground water quality. In: Bergstrom JC, Boyle KJ, Poe GL (eds) The economic value of water quality, chap 6. Edward Elgar Publishers, Cheltenham

    Google Scholar 

  • vanHoutven G, Powers J, Pattanayak S (2007) Valuing water quality improvements in the United States using meta-analysis: is the glass half-full or half-empty for national policy analysis? Resour Energy Econ 29:206–228

    Article  Google Scholar 

  • Wegge TC, Hanemann WM, Strand IE (1986) An economic assessment of marine recreational fishing in southern california, technical report, NOAA-TM-NMFS-SWR-015. NOAA National Marine Fisheries Service Southwest Region

  • Woodward RT, Wui Y-S (2001) The economic value of wetland services: a meta-analysis. Ecol Econ 37:257–270

    Google Scholar 

  • Wooldridge JM (1995) Selection corrections for panel data models under conditional mean independence assumptions. J Econom 68(1):115–132

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

    Kevin J. Boyle

  2. University of Miami, Coral Gables, FL, USA

    Christopher F. Parmeter

  3. Industrial Economics, Incorporated, Cambridge, MA, UK

    Brent B. Boehlert & Robert W. Paterson

Authors
  1. Kevin J. Boyle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christopher F. Parmeter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Brent B. Boehlert
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robert W. Paterson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christopher F. Parmeter.

Additional information

This research was sponsored by the U.S. EPA Grant # RD-83345901-0, and the data were gathered under a grant from the U.S. Fish and Wildlife Service. Senior authorship is shared by Boyle and Parmeter.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Boyle, K.J., Parmeter, C.F., Boehlert, B.B. et al. Due Diligence in Meta-analyses to Support Benefit Transfers. Environ Resource Econ 55, 357–386 (2013). https://doi.org/10.1007/s10640-012-9630-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10640-012-9630-y

Keywords

Search

Navigation