Environmental and Resource Economics

, Volume 62, Issue 3, pp 433–455 | Cite as

Experimental Design Criteria and Their Behavioural Efficiency: An Evaluation in the Field

  • Richard T. Yao
  • Riccardo Scarpa
  • John M. Rose
  • James A. Turner


Comparative results from an evaluation of inferred attribute non-attendance are provided for experimental designs optimised for three commonly employed statistical criteria, namely: orthogonality, Bayesian D-efficiency and optimal orthogonality in the difference. Survey data are from a choice experiment used to value the conservation of threatened native species in New Zealand’s production forests. In line with recent literature, we argue that attribute non-attendance can be taken as one of the important measures of behavioural efficiency. We focus on how this varies when alternative design criteria are used. Attribute non-attendance is inferred using an approach based on constrained latent classes. Given our proposed criterion to evaluate behavioural efficiency, our data indicate that the Bayesian D-efficiency criterion provides behaviourally more efficient choice tasks compared to the other two criteria.


Attribute non-attendance Choice experiment Experimental design Latent class logit model Production forests Threatened native species 


  1. Balcombe K, Burton M, Rigby D (2011) Skew and attribute non-attendance within the bayesian mixed logit model. J Environ Econ Manag 62(3):446–461CrossRefGoogle Scholar
  2. Balcombe K, Fraser I (2011) A general treatment of ‘don’t know’ responses from choice experiments. Eur Rev Agric Econ 38(2):171–191CrossRefGoogle Scholar
  3. Bliemer MCJ, Rose JM (2009) Designing stated choice experiments: the state of the art. In: Kitamura R, Yoshi T, Yamamoto T (eds) The expanding sphere of travel behavior research. Selected papers from the 11th International Conference on Travel Behavior Research, Kyoto, 16–20 August 2009Google Scholar
  4. Bliemer MCJ, Rose JM (2010) Construction of experimental designs for mixed logit models allowing for correlation across choice observations. Transp Res B Methodol 46(3):720–734CrossRefGoogle Scholar
  5. Bliemer MCJ, Rose JM (2011) Experimental design influences on stated choice outputs: an empirical study in air travel choice. Transp Res A Policy 45:63–79CrossRefGoogle Scholar
  6. Boxall PC, Adamowicz WL (2002) Understanding heterogeneous preferences in random utility models: a latent class approach. Environ Resour Econ 23(4):421–446CrossRefGoogle Scholar
  7. Campbell D, Hutchinson WG, Scarpa R (2008) Incorporating discontinuous preferences into the analysis of discrete choice experiments. Environ Resour Econ 34(3):396–411Google Scholar
  8. Campbell D, Hensher DA, Scarpa R (2012a) Cost thresholds, cut-offs and sensitivities in stated choice analysis: identification and implications. Resour Energy Econ 41:401–417CrossRefGoogle Scholar
  9. Campbell D, Mørkbak MR, Olsen SB (2012b) Response latency in stated choice experiments: impact on preference, variance and processing heterogeneity. Paper presented at the European Association of Environmental and Resource Economists 19th Annual Conference in Prague, Czech Republic, 27–30 June 2012Google Scholar
  10. Cantillo V, Ortúzar JD (2006) Implications of thresholds in discrete choice modelling. Transp Rev 26:667–691CrossRefGoogle Scholar
  11. Cantillo V, Heydecker B, Ortúzar JD (2006) A discrete choice model incorporating thresholds for perception in attribute values. Transp Res B Methodol 40:807–825CrossRefGoogle Scholar
  12. Carlsson F, Kataria M, Lampi E (2010) Dealing with ignored attributes in choice experiments on valuation of sweden’s environmental quality objectives. Environ Resour Econ 47:65–89CrossRefGoogle Scholar
  13. ChoiceMetrics (2012) Ngene 1.1.1 user manual and reference guide.
  14. Chou HY, Lu JL, Fu C (2008) The study of price accept threshold for the “blue highway” tour of the North-East Region in Taiwan. J MarSci Technol 16:255–264Google Scholar
  15. Econometric Software, Inc. (2012) NLOGIT 5. Plainview, New YorkGoogle Scholar
  16. Fasolo B, McClelland GH, Todd PM (2007) Escaping the tyranny of choice: when fewer attributes make choice easier. Mark Theory Decis 7:13–26CrossRefGoogle Scholar
  17. Ferrini S, Scarpa R (2007) Designs with a priori information for nonmarket valuation with choice experiments: a Monte Carlo study. J Environ Econ Manag 53:342–363zbMATHCrossRefGoogle Scholar
  18. Fiske ST, Taylor SE (1984) Social cognition. Addison-Wesley, MassachusettsGoogle Scholar
  19. Greene WH, Hensher DA (2003) A latent class model for discrete choice analysis: contrasts with mixed logit. Transp Res B Methodol 37:681–698CrossRefGoogle Scholar
  20. Greene WH, Hensher DA (2010) Ordered choices and heterogeneity in attribute processing. J Transp Econ Policy 44:331–364Google Scholar
  21. Han S, Gupta S, Lehmann DR (2001) Consumer price sensitivity and price thresholds. J Retail 77:435–456CrossRefGoogle Scholar
  22. Hensher DA (2006) How do respondents process stated choice experiments? Attribute consideration under varying information load. J Appl Econom 21:861–878MathSciNetCrossRefGoogle Scholar
  23. Hensher DA (2008) Joint estimation of process and outcome in choice experiments and implications for willingness to pay. J Transp Econ Policy 42:297–322Google Scholar
  24. Hensher DA (2010) Hypothetical bias, stated choice studies and willingness to pay. Transp Res B Methodol 44:735–752CrossRefADSGoogle Scholar
  25. Hensher DA, Layton D (2008) Attribute referencing, cognitive rationalisation and implications for willingness to pay. Working paper, Institute of Transport and Logistics Studies, The University of SydneyGoogle Scholar
  26. Hensher DA, Greene WH (2010) Non-attendance and dual processing of common-metric attributes in choice analysis: a latent class specification. Empir Econ 39:413–426CrossRefGoogle Scholar
  27. Hensher DA, Rose JM, Greene WH (2005a) Applied choice analysis: a primer. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  28. Hensher DA, Rose JM, Greene WH (2005b) The implications on willingness to pay of respondents ignoring specific attributes. Transportation 32:203–222CrossRefGoogle Scholar
  29. Hensher DA, Rose JM, Greene WH (2012) Inferring attribute non-attendance from stated choice data: implications for willingness to pay estimates and a warning for stated choice experiment design. Transportation 39:235–245CrossRefGoogle Scholar
  30. Hess S, Smith C, Falzarano S, Stubits J (2008) Measuring the effects of different experimental designs and survey administration methods using an Atlanta managed lanes stated preference survey. Transp Res Rec 2049:144–152CrossRefGoogle Scholar
  31. Hole A (2011) A discrete choice model with endogenous attribute attendance. Econ Lett 110(3):203–205MathSciNetCrossRefGoogle Scholar
  32. Islam T, Louviere JJ, Burke PF (2007) Modeling the effects of including/excluding attributes in choice experiments on systematic and random components. Int J Mark Res 24:289–300CrossRefGoogle Scholar
  33. Kennedy P (2008) A guide to econometrics, 6th edn. Wiley-Blackwell, LondonGoogle Scholar
  34. Kerr GN, Sharp BMH (2010) Choice experiment adaptive design benefits: a case study. Aust J Agric Resour Econ 54:407–420CrossRefGoogle Scholar
  35. Kessels R, Goos P, Vandebroek M (2006) A comparison of criteria to design efficient choice experiments. J Mark Res 43:409–419CrossRefGoogle Scholar
  36. Kessels R, Goos P, Vandebroek M (2008) Optimal designs for conjoint experiments. Comput Stat Data Anal 52(5):2369–2387zbMATHMathSciNetCrossRefGoogle Scholar
  37. Kessels R, Jones B, Goos P, Vandebroek M (2011) The usefulness of Bayesian optimal designs for discrete choice experiments. Appl Stoch Model Bus 27(3):173–188MathSciNetCrossRefGoogle Scholar
  38. Kinter ET, Prior TJ, Carswell CI, Bridges JFP (2012) A comparison of two experimental design approaches in applying conjoint analysis in patient-centered outcomes research: a randomized trial. Patient 5(4):279–294CrossRefPubMedGoogle Scholar
  39. Lancsar E, Louviere JJ (2006) Deleting ‘irrational’ responses from discrete choice experiments: a case of investigating or imposing preferences? Health Econ 15:797–811CrossRefPubMedGoogle Scholar
  40. Louviere JJ, Hensher DA (1983) Using discrete choice models with experimental design data to forecast consumer demand for a unique cultural event. J Consum Res 10:348–361CrossRefGoogle Scholar
  41. Louviere JJ, Woodworth G (1983) Design and analysis of simulated consumer choice or allocation experiments: an approach based on aggregate data. J Mark Res 20:350–367CrossRefGoogle Scholar
  42. Louviere JJ, Hensher DA, Swait JD (2000) Stated choice methods analysis and application. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  43. Louviere JJ, Islam T, Wasi N, Street D, Burgess L (2008) Designing discrete choice experiments: Do optimal designs come at a price? J Consum Res 35:360–375CrossRefGoogle Scholar
  44. Manski C (1977) The structure of random utility models. Theor Decis 8:229–254zbMATHMathSciNetCrossRefGoogle Scholar
  45. McFadden D (1974) Conditional logit analysis of qualitative choice behavior. In: Zarembka P (ed) Frontiers in econometrics. Academic Press, New YorkGoogle Scholar
  46. McIntosh E, Ryan M (2002) Using discrete choice experiments to derive welfare estimates for the provision of elective surgery: implications of discontinuous preferences. J Econ Psychol 23:367–382CrossRefGoogle Scholar
  47. Meyerhoff J, Liebe U (2009) Discontinuous preferences in choice experiments: evidence at the choice task level. Paper presented at the 17th EAERE conference, Amsterdam, 24–27 June 2009Google Scholar
  48. Mørkbak MR, Christensen T, Gyrd-Hansen D (2010) Choke price bias in choice experiments. Environ Resour Econ 45:537–551CrossRefGoogle Scholar
  49. Puckett SM, Hensher DA (2009) Revealing the extent of process heterogeneity in choice analysis: an empirical assessment. Transp Res A Policy 431:117–126CrossRefGoogle Scholar
  50. Rose JM, Black I (2006) means matter, but variance matter too: decomposing response latency influences on variance heterogeneity in stated preference experiments. Mark Lett 17(4):295–310CrossRefGoogle Scholar
  51. Rose JM, Bliemer MCJ (2008) Stated preference experimental design strategies. In: Hensher DA, Button KJ (eds) Handbook of transport modelling. Elsevier, OxfordGoogle Scholar
  52. Rose JM, Bliemer MCJ (2013) Sample size requirements for stated choice experiments. Transportation 40:1021–1041Google Scholar
  53. Sandor Z, Wedel M (2001) Designing conjoint choice experiments using managers’ prior beliefs. J Mark Res 38:430–444CrossRefGoogle Scholar
  54. Sandor Z, Wedel M (2002) Profile construction in experimental choice designs for mixed logit models. Mark Sci 21:455–475CrossRefGoogle Scholar
  55. Sandor Z, Wedel M (2005) Heterogeneous conjoint choice designs. J Mark Res 42:210–218CrossRefGoogle Scholar
  56. Scarpa R, Rose J (2008) Design efficiency for non-market valuation with choice modelling: how to measure it, what to report and why. Aust J Agric Resour Econ 52:253–282CrossRefGoogle Scholar
  57. Scarpa R, Campbell D, Hutchinson GW (2007) Benefit estimates for landscape improvements: sequential Bayesian design and respondents’ rationality in a choice experiment. Land Econ 83:617–34CrossRefGoogle Scholar
  58. Scarpa R, Ferrini S, Willis K (2005) Performance of error component models for status quo-effects in choice experiments. In: Scarpa R, Alberini A (eds) Applications of simulation methods in environmental and resource economics. Springer, DordrechtCrossRefGoogle Scholar
  59. Scarpa R, Gilbride TJ, Campbell D, Hensher DA (2009) Modelling attribute non-attendance in choice experiments for rural landscape valuation. Eur J Agric Econ 36:151–74CrossRefGoogle Scholar
  60. Scarpa R, Thiene M, Hensher DA (2010) Monitoring choice task attribute attendance in non-market valuation of multiple park management services: Does it matter? Land Econ 86(4):817–839CrossRefGoogle Scholar
  61. Scarpa R, Zanoli R, Bruschi V, Naspetti S (2013) Inferred and stated attribute non-attendance in food choice experiments. Am J Agric Econ 95(1):165–180CrossRefGoogle Scholar
  62. Severin V (2001) Comparing statistical and respondent efficiency in choice experiments. Ph.D. Dissertation, Discipline of Marketing, Faculty of Economics and Business, University of SydneyGoogle Scholar
  63. Street DJ, Burgess L (2004) Optimal and near-optimal pairs for the estimation of effects in 2-level choice experiments. J Stat Plan Inference 118:185–199zbMATHMathSciNetCrossRefGoogle Scholar
  64. Street DJ, Burgess L (2007) The construction of optimal stated choice experiments: theory and methods. Wiley, New JerseyCrossRefGoogle Scholar
  65. Street DJ, Burgess L, Louviere JJ (2005) Quick and easy choice tasks: constructing optimal and nearly optimal stated choice experiments. Int J Res Mark 22:459–470CrossRefGoogle Scholar
  66. Swait J (1994) A structural equation model of latent segmentation and product choice for cross-sectional revealed reference choice data. J Retail Consum Serv 1:77–89CrossRefGoogle Scholar
  67. Swait J (2001) A non-compensatory choice model incorporating attribute cutoffs. Transp Res B Methodol 35:903–928CrossRefGoogle Scholar
  68. Thurstone L (1931) The indifference function. J Soc Psychol 2:139–167CrossRefGoogle Scholar
  69. Train KE (2009) Discrete choice methods with simulation, 2nd edn. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  70. Vermeulen B, Goos P, Scarpa R, Vandebroek M (2011) Bayesian conjoint choice designs for measuring willingness to pay. Environ Resour Econ 48:129–149CrossRefGoogle Scholar
  71. Vermunt JK, Magidson J (2005) Technical guide for Latent GOLD Choice 4.0: basic and advanced. Statistical Innovations Inc., MassachusettsGoogle Scholar
  72. Viney R, Savage E, Louviere JJ (2005) Empirical investigation of experimental design properties of discrete choice experiments in health care. Health Econ 14:349–362CrossRefPubMedGoogle Scholar
  73. Weber B, Aholt A, Neuhaus C, Trautner P, Elger CE, Teichert T (2007) Neural evidence for reference-dependence in real-market-transactions. NeurImage 35:441–447CrossRefGoogle Scholar
  74. Yao RT, Scarpa R, Turner JA, Barnard TD, Rose JM, Palma JHN, Harrison DR (2014) Valuing biodiversity enhancement in New Zealand’s planted forests: socioeconomic and spatial determinants of willingness-to-pay. Ecol Econ 98:90–101CrossRefGoogle Scholar
  75. Yu J, Goos P, Vandebroek M (2012) A comparison of different Bayesian design criteria for setting up stated preference studies. Transp Res B Methodol 46:789–807CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • Richard T. Yao
    • 1
  • Riccardo Scarpa
    • 2
    • 3
  • John M. Rose
    • 4
  • James A. Turner
    • 5
  1. 1.Economics, Ecosystems and Climate TeamScion (NZ Forest Research Institute Ltd.)RotoruaNew Zealand
  2. 2.The University of WaikatoHamiltonNew Zealand
  3. 3.Gibson InstituteQueens UniversityBelfastUK
  4. 4.Institute for ChoiceUniversity of South Australia Business SchoolSydneyAustralia
  5. 5.AgResearch LtdHamiltonNew Zealand

Personalised recommendations