Environmental and Resource Economics

, Volume 69, Issue 2, pp 365–393 | Cite as

Single-Choice, Repeated-Choice, and Best-Worst Scaling Elicitation Formats: Do Results Differ and by How Much?

  • Daniel R. Petrolia
  • Matthew G. Interis
  • Joonghyun Hwang


This paper presents what we believe to be the most comprehensive suite of comparison criteria regarding multinomial discrete-choice experiment elicitation formats to date. We administer a choice experiment focused on ecosystem-service valuation to three independent samples: single-choice, repeated-choice, and best-worst scaling elicitation. We test whether results differ by parameter estimates, scale factors, preference heterogeneity, status-quo effects, attribute non-attendance, and magnitude and precision of welfare measures. Overall, we find limited evidence of differences in attribute parameter estimates, scale factors, and attribute increment values across elicitation treatments. However, we find significant differences in status-quo effects across elicitation treatments, with repeated-choice resulting in greater proportions of “action” votes, and consequently, higher program-level welfare estimates. Also, we find that single-choice yields drastically less-precise welfare estimates. Finally, we find some differences in attribute non-attendance behavior across elicitation formats, although there appears to be little consistency in class shares even within a given elicitation treatment.


Best-worst scaling Choice experiment Contingent valuation Ecosystem-service valuation Stated preference Survey Willingness to pay 



The authors thank A.A.J. Marley and two anonymous referees for comments that greatly improved the manuscript. This research was conducted under award NA10OAR4170078 to the Mississippi-Alabama Sea Grant Consortium by the NOAA Office of Ocean and Atmospheric Research, U.S. Department of Commerce, and was supported by the USDA Cooperative State Research, Education & Extension Service, Multistate Project W-3133 “Benefits and Costs of Natural Resources Policies Affecting Ecosystem Services on Public and Private Lands” (Hatch # MIS-033130).

Supplementary material

10640_2016_83_MOESM1_ESM.docx (68 kb)
Supplementary material 1 (docx 67 KB)


  1. Alemu MH, Mørkbak MR, Olsen SB, Jensen CL (2013) Attending to the reasons for attribute non-attendance in choice experiments. Environ Resour Econ 54:333–359CrossRefGoogle Scholar
  2. Arrow K, Solow R, Portney PR, Leamer EE, Radner R, Schuman H (1993) Report of the NOAA panel on contingent valuation. Fed Regist 58:4601–4614Google Scholar
  3. Bateman IJ, Cole M, Cooper P, Georgiou S, Hadley D, Poe GL (2004) On visible choice sets and scope sensitivity. J Environ Econ Manag 47:71–93CrossRefGoogle Scholar
  4. Beaumais O, Prunetti D, Casacianca A, Pieri X (2015) Improving solid waste management in the Island of Beauty (Corsica): a latent-class rank-ordered logit approach with observed heterogeneous ranking capabilities. Revue d’economie politique 125(2):209–231CrossRefGoogle Scholar
  5. Blamey RK, Bennett JW, Louviere JJ, Morrison MD, Rolfe JC (2002) Attribute causality in environmental choice modelling. Environ Resour Econ 23:167–186CrossRefGoogle Scholar
  6. Bliemer MCJ, Rose JM (2009) Efficiency and sample size requirements for stated choice experiments. Transportation Research Board Annual Meeting, Washington, DCGoogle Scholar
  7. Bliemer MCJ, Rose JM (2005) Efficiency and sample size requirements for stated choice studies. Report ITLS-WP-05-08, Institute of Transport and Logistics Studies, University of SydneyGoogle Scholar
  8. Campbell D, Hensher DA, Scarpa R (2011) Non-attendance to attributes in environmental choice analysis: a latent class specification. J Environ Plan Manag 54(8):061–76CrossRefGoogle Scholar
  9. Campbell D, Hutchinson WG, Scarpa R (2008) Incorporating discontinuous preferences into the analysis of discrete choice experiments. Environ Resour Econ 41:401–417CrossRefGoogle Scholar
  10. Carlsson F, Mørkbak MR, Olsen SB (2012) The first time is the hardest: a test of ordering effects in choice experiments. J Choice Model 5(2):19–37CrossRefGoogle Scholar
  11. Carson RT (2012) Contingent valuation: a practical alternative when prices aren’t available. J Econ Perspect 26(4):27–42CrossRefGoogle Scholar
  12. Carson RT (1985) Three essays on contingent valuation. PhD thesis, University of California, BerkeleyGoogle Scholar
  13. Carson RT, Czajkowski M (2013) A new baseline model for estimating willingness to pay from discrete choice models. Presented at the 2013 international choice modelling conference, July. Cited 9 Dec 2014
  14. Carson RT, Groves T (2007) Incentive and informational properties of preference questions. Environ Resour Econ 37:181–210CrossRefGoogle Scholar
  15. Carson RT, Louviere JJ (2011) A common nomenclature for stated preference elicitation approaches. Environ Resour Econ 49:539–559CrossRefGoogle Scholar
  16. Chapman RG, Staelin R (1982) Exploiting rank ordered choice set data within the stochastic utility model. J Mark Res XIX:288–301Google Scholar
  17. ChoiceMetrics (2011) Ngene 1.1 user manual and reference guideGoogle Scholar
  18. Collins AT, Rose JM, Hensher DA (2013) Specification issues in a generalized random parameters attribute nonattendance model. Transp Res Part B 56:234–253CrossRefGoogle Scholar
  19. Day B, Bateman IJ, Carson RT, Dupont D, Louviere JJ, Morimoto S, Scarpa R, Wang P (2012) Ordering effects and choice set awareness in repeat-response stated preference studies. J Environ Econ Manag 63:73–91CrossRefGoogle Scholar
  20. Day B, Prades JLP (2010) Ordering anomalies in choice experiments. J Environ Econ Manag 59:271–285CrossRefGoogle Scholar
  21. Flynn TN, Louviere JJ, Peters TJ, Coast J (2007) Best-worst scaling: what it can do for health care research and how to do it. J Health Econ 26:71–89CrossRefGoogle Scholar
  22. Flynn T, Marley AJ (2014) Best worst scaling: theory and methods. In: Hess S, Daly A (eds) Handbook of choice modelling. Edward Elgar Publishing, Camberley, pp 178–201Google Scholar
  23. Greene WH (2012) Reference Guide, NLOGIT Version 5.0, Econometric Software, Inc., Plainview, NYGoogle Scholar
  24. Haab TC, McConnell KE (2002) Valuing environmental and natural resources: the econometrics of non-market valuation. Edward Elgar, NorthamptonCrossRefGoogle Scholar
  25. Hair JF Jr, Black WC, Babin BJ, Anderson RE (2010) Multivariate data analysis, 7th edn. Pearson, Upper Saddle RiverGoogle Scholar
  26. Hanemann W (1985) Some issues in continuous- and discrete-response contingent valuation studies. Northeast J Agric Econ 14:5–13Google Scholar
  27. Hensher DA, Collins AT, Greene WH (2013) Accounting for attribute non-attendance and common-metric aggregation in a probabilistic decision process mixed multinomial logit model: a warning on potential confounding. Transportation 40:1003–1020CrossRefGoogle Scholar
  28. Hensher DA, Greene WH (2003) The mixed logit model: the state of the practice. Transportation 30:133–176CrossRefGoogle Scholar
  29. Hensher DA, Greene WH (2010) Non-attendance and dual processing of common-metric attribute in choice analysis: a latent class specification. Empir Econ 39:413–426CrossRefGoogle Scholar
  30. Hensher DA, Rose JM, Greene WH (2012) Inferring attribute non-attendance from stated choice data: implications for willingness to pay estimates and a warning for stated choice experiment design. Transportation 39:235–245CrossRefGoogle Scholar
  31. Hess S, Stathopoulos A, Campbell D, O’Neill V, Caussade S (2013) It’s not that I don’t care, I just don’t care very much: confounding between attribute non-attendance and taste heterogeneity. Transportation 40:583–607CrossRefGoogle Scholar
  32. Holmes TP, Boyle KJ (2005) Dynamic learning and context-dependence in sequential, attribute-based, stated-preference valuation questions. Land Econ 81:114–126CrossRefGoogle Scholar
  33. Interis MG, Petrolia DR (2016) Location, location, habitat: how the value of ecosystem services varies across location and by habitat. Land Econ 92(2):292–307CrossRefGoogle Scholar
  34. Ladenburg J, Olsen SB (2008) Gender-specific starting point bias in choice experiments: evidence from an empirical study. J Environ Econ Manag 56:275–285CrossRefGoogle Scholar
  35. List JA, Sinha P, Taylor MH (2006) Using choice experiments to value non-market goods and services: evidence from field experiments. B.E. J Econ Anal Policy 5(2):1–37Google Scholar
  36. Louviere JJ, Flynn TN, Carson RT (2010) Discrete choice experiments are not conjoint analysis. J Choice Model 3(3):57–72CrossRefGoogle Scholar
  37. Louviere JJ, Flynn TN, Marley AAJ (2015) Best-worst scaling: theory, methods and applications. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  38. Marden JI (1995) Analyzing and modeling rank data. Chapman and Hall, LondonGoogle Scholar
  39. Marley AAJ, Louviere JJ (2005) Some probabilistic models of best, worst, and best-worst choices. J Math Psychol 49:464–480CrossRefGoogle Scholar
  40. McNair B, Bennett J, Hensher D (2011) A comparison of responses to single and repeated discrete choice questions. Resour Energy Econ 33:554–571CrossRefGoogle Scholar
  41. McNair B, Hensher D, Bennett B (2012) Modelling heterogeneity in response behavior towards a sequence of discrete choice questions: a probabilistic decision process model. Environ Resour Econ 51:599–616CrossRefGoogle Scholar
  42. Meyerhoff J, Liebe U (2009) Status quo effect in choice experiments: empirical evidence on attitudes and choice task complexity. Land Econ 85(3):515–528CrossRefGoogle Scholar
  43. Newell LW, Swallow SK (2013) Real-payment choice experiments: valuing forested wetlands and spatial attributes within a landscape context. Ecol Econ 92:37–47CrossRefGoogle Scholar
  44. Pattison J, Boxall PC, Adamowicz WL (2011) The economic benefits of wetland retention and restoration in Manitoba. Can J Agric Econ 59:223–244CrossRefGoogle Scholar
  45. Petrolia DR, Interis MG (2013) Should we be using repeated-choice surveys to value public goods? Assoc Environ Resour Econ Newsl 33(2):19–25Google Scholar
  46. Petrolia DR, Interis MG, Hwang J (2014) America’s wetland? A national survey of willingness to pay for restoration of Louisiana’s coastal wetlands. Mar Resour Econ 29(1):17–37CrossRefGoogle Scholar
  47. Poe G, Giraud K, Loomis J (2005) Computational methods for measuring the difference of empirical distributions. Am J Agric Econ 87(2):353–365CrossRefGoogle Scholar
  48. Potoglou D, Burge P, Flynn T, Netten A, Malley J, Forder J, Brazier JE (2011) Best-worst scaling vs. discrete choice experiments: an empirical comparison using social care data. Soc Sci Med 72:1717–1727CrossRefGoogle Scholar
  49. Rigby D, Burton M, Lusk JL (2015) Journals, preferences, and publishing in Agricultural and Environmental Economics. Am J Agric Econ 97(2):490–509CrossRefGoogle Scholar
  50. Samuelson PA (1954) The pure theory of public expenditure. Rev Econ Stat 36(4):387–389CrossRefGoogle Scholar
  51. Scarpa R, Notaro S, Louviere JJ, Raffaelli R (2011) Exploring scale effects of best/worst rank ordered choice data to estimate benefits of tourism in Alpine Grazing Commons. Am J Agric Econ 93(3):813–828CrossRefGoogle Scholar
  52. Scarpa R, Thiene M, Hensher DA (2010) Monitoring choice task attribute attendance in nonmarket valuation of multiple park management services: does it matter? Land Econ 86(4):817–839CrossRefGoogle Scholar
  53. Scheufele G, Bennett J (2012) Response strategies and learning in discrete choice experiments. Environ Resour Econ 52:435–453CrossRefGoogle Scholar
  54. Silz-Carson K, Chilton SM, Hutchinson WG (2010) Bias in choice experiments for public goods. Newcastle discussion papers in Economics, no. 2010/05, Newcastle University Business SchoolGoogle Scholar
  55. StataCorp (2013) Stata release 13.0 statistical software. StataCorp LP, College StationGoogle Scholar
  56. Swait J, Louviere J (1993) The role of the scale parameter in the estimation and comparison of multinomial logit models. J Mark Res XXX:305–314Google Scholar
  57. Taylor LO, Morrison MD, Boyle KJ (2010) Exchange rules and the incentive compatibility of choice experiments. Environ Resour Econ 47:197–220CrossRefGoogle Scholar
  58. Train KE (2009) Discrete choice methods with simulation, 2nd edn. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  59. Vossler CA, Doyon M, Rondeau D (2012) Truth in consequentiality: theory and field evidence on discrete choice experiments. Am Econ J Microecon 4:145–171CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  1. 1.Deparment of Agricultural EconomicsMississippi State UniversityMississippi StateUSA
  2. 2.Fish and Wildlife Research InstituteFlorida Fish and Wildlife Conservation CommissionGainesvilleUSA

Personalised recommendations