Skip to main content
SpringerLink
Log in

Reliability and Validity in Nonmarket Valuation

  • Published:
Environmental and Resource Economics Aims and scope Submit manuscript

Abstract

We propose a framework for assessing the accuracy of nonmarket values. This involves adapting two widely-used concepts. Reliability addresses variance and validity addresses potential biases. These concepts are formally defined and adapted to assess the accuracy of individual nonmarket valuation studies and the potential accuracy of valuation methods. We illustrate the framework by considering, in a preliminary way, the reliability and validity of the contingent-valuation and travel-cost methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. There are some additional issues that must be addressed when estimating WTA. For a discussion, see Kim et al. (2015).

  2. Here we take an implied assumption that a public good is being valued and recognize that for private goods the incentives for truthful preference revelation are different.

  3. By consequence, this also provides evidence that the chosen valuation method and implementation procedures in a nonmarket valuation study are reliable, but reliability of a method, as discussed below, requires multiple reliability studies before general conclusions can be reached about the method and implementation procedures. Similar considerations apply to validity assessment.

  4. Another form of construct validity testing used to evaluate the performance of econometric models is cross validation. Here, a hold-out sample of observations is used to evaluate model performance. The estimated model is used to predict responses of the hold-out sample. Predicted responses are graphed as a function of observed responses, and valid responses should fall on or near a 45-degree line from the origin. Here we do not discuss all approaches used to investigate construct validity, but use convergent validity as an example because it is the most commonly used assessment procedure in the literature.

  5. Another alternative is the “adding up test”, which can be viewed as a special case of a scope test.

  6. These numbers should be interpreted with caution. One suspects that several of the studies included in the meta-analyses would not fare well in a comprehensive validity assessment. Still the consistency of overvaluation by CV cannot be ignored.

  7. We are grateful to an anonymous reviewer for pointing this out.

  8. The use of the term “reliable” by the panel follows the U.S. legal system’s use of this term and can be considered a synonym to our use of the term “accuracy,” which includes reliability, as defined here, and validity.

  9. This possibility was pointed out to us by an anonymous reviewer. This is an interesting point that we do not recall seeing it in the literature.

  10. Other variables, such as fish catch rates reported in surveys, may also be endogenous.

  11. Interestingly, if by “hypothetical,” Scott meant “inconsequential,” he has a point.

References

  • Arrow K, Solow R, Portnoy PR, Leamer EE, Radner R, Schuman H (1993) Report of the NOAA panel on contingent valuation. Fed Reg 58:4601–4614

    Google Scholar 

  • Bhattacharjee S, Kling CL, Herriges JA (2009). Kuhn-Tucker estimation of recreation demand—a study of temporal stability. Agricultural and Applied Economics Association annual meeting

  • Berrens RP (2000) Reluctant respondents and contingent valuation surveys. Appl Econ Lett 7:263–266

    Article  Google Scholar 

  • Bockstael NE, Kling CL (1988) Valuing environmental quality: weak complementarity with sets of goods. Am J Agric Econ 70(3):654–662

    Article  Google Scholar 

  • Bockstael NE, Hanemann WM, Kling CL (1987) Modelling recreational demand in a multiple site framework. Water Resour Res 23(5):951–960

    Article  Google Scholar 

  • Boxall PC, Adamowicz WL (2002) Understanding heterogeneous preferences in random utility models: a latent class approach. Environ Resour Econ 23(4):421–446

    Article  Google Scholar 

  • Boxall PC, Rollins K, Englin J (2003) Heterogeneous preferences for congestion during a wilderness experience. Resour Energy Econ 25:177–195

    Article  Google Scholar 

  • Brander LM, Van Beukering P, Cesar HSJ (2007) The recreational value of coral reefs: a meta-analysis. Ecol Econ 63(1):209–218

    Article  Google Scholar 

  • Brower R, Bateman IJ (2005) Temporal stability and transferability of willingness to pay for flood control and wetland conservation. Water Resour Res 46(2):353–361

  • Brown WG, Nawas F (1973) Impact of aggregation on the estimation of outdoor recreation demand functions. Am J Agric Econ 55(2):246–249

    Article  Google Scholar 

  • Carmines EG, Zeller RA (1979) Reliability and validity assessment. In: Sage university paper series on quantitative applications in the social sciences. Sage, Woburn, MA

  • Carson R (2011) Contingent valuation: a comprehensive bibliography and history. Edward Elgar, Cheltenham

    Book  Google Scholar 

  • Carson R, Flores NE, Martin K, Wright JL (1996) Contingent valuation and revealed preference methodologies: comparing the estimates for quasi-public goods. Land Econ 72:80–99

    Article  Google Scholar 

  • Carson R, Flores NE, Meade NF (2001) Contingent valuation: controversies and evidence. Environ Resour Econ 19:173–210

    Article  Google Scholar 

  • Carson R, Groves RM (2007) Incentive and informational properties of preference questions. Environ Resour Econ 37:181–210

    Article  Google Scholar 

  • Carson R, Groves T, List JA (2014) Consequentiality: a theoretical and experimental exploration of a single binary choice. J Assoc Environ Resour Econ 1:171–207

    Google Scholar 

  • Carson R, Hanemann M, Kopp RJ, Krosnick JA, Mitchell RC, Presser S, Rudd PA, Smith VK, Conaway M, Martin K (1997) Temporal reliability of estimates from contingent valuation. Land Econ 73(2):151–163

    Article  Google Scholar 

  • Carson R, Mitchell RC, Hanemann M, Kopp RJ, Presser S, Ruud PA (2003) Contingent valuation and lost passive use: damages from the Exxon Valdez oil spill. Environ Resour Econ 1:171–207

    Google Scholar 

  • Cesario FJ, Knetsch JL (1970) Time bias in recreation benefit estimates. Water Resour Res 6(3):700–704

    Article  Google Scholar 

  • Champ PA, Brown TC (1997) A comparison of contingent and actual voting behavior. In: Proceedings from W-133 benefits and cost transfer in natural resource planning, 10th Interim Report, pp 77–98

  • Champ PA, Boyle KJ, Brown TC (eds) (2017) A primer on nonmarket valuation, Springer, New York

  • Chu A, Eisenhower D, Hay M, Morganstein D, Neter J, Waksberg J (1992) Measuring the recall error in self-reported fishing and hunting activities. J Off Stat 8(1):19–39

    Google Scholar 

  • Desvousges WH, Matthews K, Train K (2012) Adequate responsiveness to scope in contingent valuation. Ecol Econ 84:121–128

    Article  Google Scholar 

  • Diamond P (1996) Testing the internal consistency of contingent valuation surveys. J Environ Econ Manag 30(3):337–347

    Article  Google Scholar 

  • Diamond PA, Hausman JA (1994) Contingent valuation: Is some number better than no number? J Econ Perspect 8:45–64

    Article  Google Scholar 

  • Egan KJ, Herriges JA, Kling CL, Downing JA (2009) Valuing water quality as a function of water quality measures. Am J Agric Econ 91(1):106–123

    Article  Google Scholar 

  • Gems B, Ghosh D, Hitlin R (1982) A recall experiment: impact of time on recall on recreational fishing trips. In: Proceedings of the section on survey research methods. American Statistical Association, Washington, DC

  • Gen S (2004) Meta-analysis of environmental valuation studies. Public Policy, Georgia Institute of Technology, Atlanta

    Google Scholar 

  • Grijalva TR, Berrens RP, Bohara A, Shaw W (2002) Testing the validity of contingent behavior trip responses. Am J Agric Econ 84(2):401–414

    Article  Google Scholar 

  • Haab TC, McConnell KE (2002) Valuing environmental and natural resources: the econometrics of non-market valuation. Edward Elgar, Cheltenham

    Book  Google Scholar 

  • Haener MK, Boxall PC, Adamowicz WL, Kuhnke DH (2004) Aggregation bias in recreation site choice models: resolving the resolution problem. Land Econ 80(4):561–574

    Article  Google Scholar 

  • Hanley N (1989) Valuing non-market goods using contingent valuation. J Econ Surv 3(3):235–252

    Article  Google Scholar 

  • Hanley N, Schlapfer F, Spurgeon J (2003) Aggregating the benefits of environmental improvements: distance-decay functions for use and non-use values. J Environ Manag 68:297–304

    Article  Google Scholar 

  • Hausman JA (1993) Contingent valuation: a critical assessment. Emerald Group Publishing Limited, Bingley

    Book  Google Scholar 

  • Hausman JA (2012) Contingent valuation: from dubious to hopeless. J Econ Perspect 26:43–56

    Article  Google Scholar 

  • Herriges JA, Kling CL (2008) Revealed preference approaches to environmental valuation, volumes I and II. Routledge, New York

    Google Scholar 

  • Holmes T, Adamowicz W, Carlsson F (2017) Choice experiments. In: Champ PA, Boyle KJ, Brown TC (eds) A primer on nonmarket valuation, Ch. 5. Springer, New York

  • Hynes S, Hanley N, Garvey E (2007) Up the proverbial creek without a paddle: accounting for variable participant skill levels in recreational demand modelling. Environ Resour Econ 36(4):413–426

    Article  Google Scholar 

  • Jacobsen JB, Hanley N (2009) Are there income effects on global willingness to pay for biodiversity conservation? Env Resour Econ 43(2):137–160

    Article  Google Scholar 

  • Joen Y, Herriges JA, Kling CL, Downing JA (2011) The role of water quality perceptions in modelling lake recreation demand. In: Bennett JW (ed) The international handbook on non-market environmental valuation. Edward Elgar, Cheltenham, pp 74–101

    Google Scholar 

  • Johnston RJ (2006) Is hypothetical bias universal? Validating contingent valuation response using a binding public referendum. J Environ Econ Manag 52:469–481

    Article  Google Scholar 

  • Johnston R, Boyle KJ, Adamowicz W, Bennett J, Brouwer R, Cameron TA, Hanemann WM, Hanley N, Ryan M, Scarpa R, Tourangeau R, Vossler CA (2017) Contemporary guidance for stated preference studies. J Assoc Environ Resour Econ 4(2):319–405

    Google Scholar 

  • Jones-Lee MW, Hammerton M, Philips PR (1985) The value of safety: results of a national sample survey. Econ J 95:49–72

    Article  Google Scholar 

  • Kaoru Y (1995) Measuring marine recreation benefits of water quality improvements by the nested random utility model. Resour Energy Econ 17(2):119–136

    Article  Google Scholar 

  • Kealy MJ, Dovidio JF, Rockel ML (1988) Accuracy in valuation is a matter of degree. Land Econ 64(2):158–171

    Article  Google Scholar 

  • Kealy MJ, Montgomery M, Dovidio JF (1990) Reliability and predictive validity of contingent valuation: Does the nature of the good matter? J Environ Econ Manag 19:244–263

    Article  Google Scholar 

  • Kim Y, Kling CL, Zhao J (2015) Understanding behavioral explanations for the WTP-WTA divergence through a neoclassical lens: implications for environmental policy. Ann Rev Resour Econ 7(1):169–187

    Article  Google Scholar 

  • Kinnel JC, Bingham MF, Mohamed AF, Desvousges WH, Kiler TB, Hastings EK, Kuhns KT (2006) Estimating site choice decisions for urban recreators. Land Econ 82(2):257–272

    Article  Google Scholar 

  • Landry CE, Hindsley P (2011) Valuing beach quality with hedonic property models. Land Econ 87(1):92–108

    Article  Google Scholar 

  • Landry CE, Liu H (2009) A semi-parametric estimator for revealed and stated preference data—an application to recreational beach visitation. J Environ Econ Manag 57(2):205–218

    Article  Google Scholar 

  • Lindhjem H, Navrud S (2009) Asking for individual or household willingness to pay for environmental goods? Environ Resour Econ 43(1):11–29

    Article  Google Scholar 

  • List JA, Gallet CA (2001) What experimental protocol influence disparities between actual and hypothetical stated values? Evidence from a meta-analysis. Environ Resour Econ 20:241–254

    Article  Google Scholar 

  • Little J, Berrens RP (2004) Explaining disparities between actual and hypothetical stated values: further investigation using meta-analysis. Econ Bull 3:1–13

    Google Scholar 

  • List JA, Gallet CA (2001) What experimental protocol influence disparities between actual and hypothetical stated values? Environ Resour Econ 20(3):241–254

    Article  Google Scholar 

  • Loehman E, De VH (1982) Application of stochastic choice modeling to policy analysis of public goods: a case study of air quality improvements. Rev Econ Stat 54:474–480

    Article  Google Scholar 

  • Loomis JB (1989) Test–retest reliability of the contingent valuation method: a comparison of general population and visitor responses. Am J Agric Econ 71:76–84

    Article  Google Scholar 

  • Loomis JB (1990) Comparative reliability of the dichotomous choice and open-ended contingent valuation techniques. J Environ Econ Manag 18(1):78–85

    Article  Google Scholar 

  • Loureiro ML, Loomis JB (2017) How sensitive are environmental valuations to economic downturns? Ecol Econ 140:235–240

    Article  Google Scholar 

  • Lupi F, Hoehn JP, Christie GC (2003) Using an economic model of recreational fishing to evaluate the benefits of sea lamprey (Petromyzon marinus) control on the St Marys River. J Great Lakes Res 29:742–754

    Article  Google Scholar 

  • Massey DM, Newbold SC, Gentner B (2006) Valuing water quality changes using a bioeconomic model of a coastal recreational fishery. J Environ Econ Manag 52(1):482–500

    Article  Google Scholar 

  • Mazurkiewicz SM, Boyle KJ, Teisl MF, Morris KI, Clark AG (1996) Recall bias and reliability of survey data: moose hunting in Maine. Wildl Soc Bull 24(1):140–148

    Google Scholar 

  • McCollum DW (1986) The travel cost method: time, specification and validity. University of Wisconsin-Madison, Madison

    Google Scholar 

  • McConnell KE (1977) Congestion and willingness to pay: a study of beach use. Land Econ 53(2):185–195

    Article  Google Scholar 

  • McConnell KE, Strand IE, Valdes S (1998) Testing temporal reliability and carry-over effect: the role of correlated responses in test–retest reliability studies. Environ Resour Econ 12:357–374

    Article  Google Scholar 

  • Mitchell RC, Carson RT (1989) Using surveys to value public goods: the contingent valuation method. Resources for the Future, Washington, DC

    Google Scholar 

  • Mkwara L, Marsh D, Scarpa R (2015). Testing the stability of welfare estimates in travel cost random utility models of recreation: an application to the Rotorua Lakes, New Zealand. 59th Australian Agricultural and Resource Economics Society Conference

  • Moeltner K, Englin J (2004) Choice behavior under time-variant quality: state dependence versus Play-It-By-Ear in selecting ski resorts. J Bus Econ Stat 22(2):214–224

    Article  Google Scholar 

  • Morrison MD, Blamey RK, Bennett JW (2000) Minimising payment vehicle bias in contingent valuation studies. Environ Resour Econ 16:407–422

    Article  Google Scholar 

  • Murdock J (2006) Handling unobserved site characteristics in random utility models of recreation demand. J Environ Econ Manag 51(1):1–25

    Article  Google Scholar 

  • Murphy JJ, Allen PG, Stevens TH, Weatherhead D (2005) A meta-analysis of hypothetial bias in stated preference valuation. Environ Resour Econ 30:313–325

    Article  Google Scholar 

  • Parsons GR (2017) Travel cost. In: Chap PA, Boyle KJ, Brown TC (eds) A primer on nonmarket valuation, Ch. 6, Springer, New York

    Google Scholar 

  • Parsons GR, Stefanova S (2011) Gauging the value of the short-term site closures in a travel-cost RUM model of recreation demand with a little help from stated preference data. In: Whitehead J, Haab TC, Huang J (eds) Preference data for environmental valuation: combining revealed and stated preference approaches. Routledge, Taylor & Francis Group, New York

    Google Scholar 

  • Phaneuf DJ (2002) A random utility model for total maximum daily loads: estimating the benefits of watershed-based ambient water quality improvements. Water Resourc Res 38(11):1254

    Article  Google Scholar 

  • Poe GL, Vossler CA (2011) Consequentiality and contingent values: an emerging paradigm. Edward Elgar, Cheltenham

    Google Scholar 

  • Randall A (1994) A difficulty with the travel cost method. Land Econ 70(1):88–96

    Article  Google Scholar 

  • Reiling SD, Boyle KJ, Phillips ML, Anderson MW (1990) Temporal reliability of contingent values. Land Econ 66(2):128–134

    Article  Google Scholar 

  • Roe BE, Just DR (2009) Internal and external validity in economics research: tradeoffs between experiments, field experiments, natural experiments, and field data. Am J Agric Econ 91(5):1266–1271

    Article  Google Scholar 

  • Scarpa R, Thiene M, Train K (2008) Utility in willingness to pay space: a tool to address confounding random scale effects in destination choice to the Alps. Am J Agric Econ 90(4):994–1010

    Article  Google Scholar 

  • Scott A (1965) The valuation of game resources: some theoretical aspects. In: Fisheries Canadian, Report, iv. Department of Fisheries of Canada, Ottawa, Canada

  • Shrestha RK, Loomis JB (2001) Testing a meta-analysis model for benefit transfer in international outdoor recreation. Ecol Econ 39(1):67–83

    Article  Google Scholar 

  • Shrestha RK, Loomis JB (2003) Meta-analytic benefit transfer of outdoor recreation economic values: testing Out-of-Sample convergent validity. Environ Resour Econ 25(1):79–100

    Article  Google Scholar 

  • Smith VK, Kaoru Y (1990) Signals or noise? Explaining the variation in recreation benefit estimates. Am J Agric Econ 72(2):419–433

    Article  Google Scholar 

  • Stevens JB (1969) Effects of nonprice variables upon participation in water-oriented outdoor recreation: comment. Am J Agric Econ 51(1):192–193

    Article  Google Scholar 

  • Stevens TH, DeCoteau NE, Willis CE (1997) Sensitivity of contingent valuation to alternative payment schedules. Land Econ 73(1):140–148

    Article  Google Scholar 

  • Stumborg BE, Baerenklau KA, Bishop RC (2001) Nonpoint source pollution and present values: a contingent valuation study of Lake Mendota. Appl Econ Perspect Policy 23(1):120–132

    Google Scholar 

  • Tarrant MA, Manfredo MJ, Bayley PB, Hess R (1993) Effects of recall bias and nonresponse bias on self-report estimates of angling participation. North Am J Fish Manag 13(2):217–222

    Article  Google Scholar 

  • Teisl MF, Boyle KJ, McCollum DW, Reiling SD (1995) Test-retest reliability of contingent valuation with independent sample pretest and posttest control groups. Am J Agric Econ 77:613–619

    Article  Google Scholar 

  • Timmins C, Murdock J (2007) A revealed preference approach to the measurement of congestion in travel cost models. J Environ Econ Manag 53(2):230–249

    Article  Google Scholar 

  • Trochim, William M (2002). The research methods knowledge base. http://www.anatomyfacts.com/research/researchmethodsknowledgebase.pdf. Accessed 30 July 2017

  • Trice AH, Wood SE (1958) Measurement of recreation benefits. Land Econ 34(3):195–207

    Article  Google Scholar 

  • Vossler CA, Doyon M, Rondeau D (2012) Truth in consequentiality: theory and field evidence on discrete choice experiments. Am Econ J Microecon 4:145–171

    Article  Google Scholar 

  • Vossler CA, Kerkvliet J (2003) A criterion validity test of the contingent valuation method: comparing hypothetical and actual voting behavior for a public referendum. J Environ Econ Manag 45:631–649

    Article  Google Scholar 

  • Vossler CA, Kerkvliet J, Polasky S, Gainutdinova O (2003) Externally validating contingent valuation: an open space survey and referendum in Corvallis, Oregon. J Econ Behav Org 51:261–277

    Article  Google Scholar 

  • Ward FA, Beal D (2000) Valuing nature with travel cost models. Edward Elgar, Cheltenham

    Book  Google Scholar 

  • Whitehead J, Noonan DS, Marquardt E (2014) Criterion and predictive validity of revealed and stated preference data: the case of Mountain Home Music concert demand. Econ Bus Lett 3(2):87–95

    Article  Google Scholar 

  • Zeller RA, Carmines EG (1980) Measurement in the social sciences: the link between theory and data. Cambridge University Press, Cambridge

    Google Scholar 

Download references

Acknowledgements

This article used Chapter 12 in Champ et al (2017) as a starting point. We thank Springer, the book’s publisher, for permission to do so. We also thank Ian Bateman for encouraging us to adapt and further develop the ideas in the chapter. Two anonymous reviewers provided many helpful comments and suggestions. Coral Bishop provided invaluable help in preparing the manuscript.

Author information

Authors and Affiliations

  1. Department of Agricultural and Applied Economics, University of Wisconsin-Madison, Madison, WI, USA

    Richard C. Bishop

  2. Virginia Tech Program in Real Estate, 430A Bishop-Favrao Hall (0715), 1345 Perry Street, Blacksburg, VA, 24061, USA

    Kevin J. Boyle

Authors
  1. Richard C. Bishop
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kevin J. Boyle
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kevin J. Boyle.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bishop, R.C., Boyle, K.J. Reliability and Validity in Nonmarket Valuation. Environ Resource Econ 72, 559–582 (2019). https://doi.org/10.1007/s10640-017-0215-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10640-017-0215-7

Keywords

Search

Navigation