Theory and Decision

, Volume 84, Issue 2, pp 215–238 | Cite as

D-efficient or deficient? A robustness analysis of stated choice experimental designs

  • Joan L. WalkerEmail author
  • Yanqiao Wang
  • Mikkel Thorhauge
  • Moshe Ben-Akiva


This paper is motivated by the increasing popularity of efficient designs for stated choice experiments. The objective in efficient designs is to create a stated choice experiment that minimizes the standard errors of the estimated parameters. In order to do so, such designs require specifying prior values for the parameters to be estimated. While there is significant literature demonstrating the efficiency improvements (and cost savings) of employing efficient designs, the bulk of the literature tests conditions where the priors used to generate the efficient design are assumed to be accurate. However, there is substantially less literature that compares how different design types perform under varying degree of error of the prior. The literature that does exist assumes small fractions are used (e.g., under 20 unique choice tasks generated), which is in contrast to computer-aided surveys that readily allow for large fractions. Further, the results in the literature are abstract in that there is no reference point (i.e., meaningful units) to provide clear insight on the magnitude of any issue. Our objective is to analyze the robustness of different designs within a typical stated choice experiment context of a trade-off between price and quality. We use as an example transportation mode choice, where the key parameter to estimate is the value of time (VOT). Within this context, we test many designs to examine how robust efficient designs are against a misspecification of the prior parameters. The simple mode choice setting allows for insightful visualizations of the designs themselves and also an interpretable reference point (VOT) for the range in which each design is robust. Not surprisingly, the D-efficient design is most efficient in the region where the true population VOT is near the prior used to generate the design: the prior is $20/h and the efficient range is $10–$30/h. However, the D-efficient design quickly becomes the most inefficient outside of this range (under $5/h and above $40/h), and the estimation significantly degrades above $50/h. The orthogonal and random designs are robust for a much larger range of VOT. The robustness of Bayesian efficient designs varies depending on the variance that the prior assumes. Implementing two-stage designs that first use a small sample to estimate priors are also not robust relative to uninformative designs. Arguably, the random design (which is the easiest to generate) performs as well as any design, and it (as well as any design) will perform even better if data cleaning is done to remove choice tasks where one alternative dominates the other.


Stated choice experiments Robustness Mode choice model Value-of-time Experimental design D-efficient 



An earlier draft of this work was initially presented at the Transportation Research Board annual meeting in January of 2015 (Walker et al. 2015), and we thank the reviewers from that process as well as the discussion that followed from the presentation and circulation of the working paper. We thank Andre de Palma and Nathalie Picard for organizing the symposium in honor of Daniel McFadden and for following it up with this special issue. We thank two anonymous reviewers assigned by this journal. We also thank Michael Galczynski for the idea for the title.


  1. Adamowicz, W., Louviere, J., & Williams, M. (1994). Combining revealed and stated preference methods for valuing environmental amenities. San diego: Academic Press Inc JNL-Comp Subscriptions.Google Scholar
  2. Belenky, P. (2011). Revised departmental guidance on valuation of travel time in economic analysis (Revision 2). Washington DC: Department of Transportation.Google Scholar
  3. Bliemer, M. C. J., & Collins, A. T. (2016). On determining priors for the generation of efficient stated choice experimental designs. The Journal of Choice Modelling, 21, 10–14.CrossRefGoogle Scholar
  4. Bliemer, M. C. J., & Rose, J. M. (2010a). Serial choice conjoint analysis for estimating discrete choice models. In S. Hess & A. Daly (Eds.), Choice modelling: The State-of-the-art and the State-of-practice (Proceedings from the inaugural International Choice Modelling Conference) (pp. 139–161). Bingley: Emerald Group Publishing.Google Scholar
  5. Bliemer, M. C. J., & Rose, J. M. (2010b). Construction of experimental designs for mixed logit models allowing for correlation across choice observations. Transportation Research Part B: Methodological, 6, 720–734.CrossRefGoogle Scholar
  6. Bliemer, M. C. J., & Rose, J. M. (2011). Experimental design influences on stated choice outputs: An empirical study in air travel choice. Transportation Research Part A: Policy and Practice, 1, 63–79.Google Scholar
  7. Bliemer, M. C. J., Rose, J. M., & Chorus, C. G. (2017). Detecting dominance in stated choice data and accounting for dominance-based scale differences in logit models. Transportation Research Part B: Methodological, 201, 83–104.CrossRefGoogle Scholar
  8. Bliemer, M. C. J., Rose, J. M., & Hensher, D. A. (2009). Efficient stated choice experiments for estimating nested logit models. Transportation Research Part B: Methodological, 1, 19–35.CrossRefGoogle Scholar
  9. Bliemer, M. C. J., Rose, J. M., & Hess, S. (2008). Approximation of Bayesian efficiency in experimental choice designs. Journal of Choice Modelling, 1, 98–126.CrossRefGoogle Scholar
  10. Bradley, M., & Bowy, P. H. L. (1984). A stated preference analysis of bicyclist route choice (pp. 39–53). Summer Annual Meeting, Sussex: PTRC.Google Scholar
  11. Burgess, L., & Street, D. J. (2005). Optimal designs for choice experiments with asymmetric attributes. Journal of Statistical Planning and Inference, 1, 288–301.CrossRefGoogle Scholar
  12. ChoiceMetrics, (2012a). Ngene 1.0.2 User Manual & Reference Guide, version: 4/02/2010, ChoiceMetrics Pty Ltd.Google Scholar
  13. ChoiceMetrics, (2012b) Ngene software, version: 1.1.1, Build: 305, developed by Rose, John M.; Collins, Andrew T.; Bliemer, Michiel C.J.; Hensher, David A., ChoiceMetrics Pty LtdGoogle Scholar
  14. Cummings, R. G., Brookshire, D. S., & Schulze, W. D. (Eds.). (1986). Valuing environmental goods: A state of the arts assessment of the contingent method. Totowa: Rowman and Allanheld.Google Scholar
  15. Davidson, J. D. (1973). Forecasting Traffic on STOL. Operational Research Quarterly, 4, 561–569.CrossRefGoogle Scholar
  16. Fosgerau, M., & Börjesson, M. (2015). Manipulating a stated choice experiment. Journal of Choice Modelling, 16, 43–49.CrossRefGoogle Scholar
  17. Fowkes, A. S. (2000). Recent developments in state preference techniques in transport research. In J. D. D. Ortuzar (Ed.), Stated preference modelling techniques (pp. 37–52). London: PTRC Education and Research Services.Google Scholar
  18. Hensher, D. A., & Louviere, J. J. (1983). Identifying individual preferences for international air fares-an application of functional-measurement theory. Bath: Univ Bath.Google Scholar
  19. Hensher, D. A. (1982). Functional-measurement, individual preference and discrete-choice modeling-theory and application. Journal of Economic Psychology, 4, 323–335.CrossRefGoogle Scholar
  20. Hensher, D. A. (1994). Stated preference analysis of travel choices-the state of practice. Dordrecht: Kluwer Academic Publ.Google Scholar
  21. Johnson, F. R., Kanninen, B., Bingham, M., & Özdemir, S. (2007). Experimental design for stated choice studies. In B. J. Kanninen (Ed.), Valuing environmental amenities using stated choice studies (pp. 159–202). Dordrecht: Springer.CrossRefGoogle Scholar
  22. Johnson, F. R., Lancsar, E., Marshall, D., Kilambi, V., Muehlbacher, A., Regier, D. A., et al. (2013). Constructing experimental designs for discrete-choice experiments: Report of the ISPOR conjoint analysis experimental design good research practices task force. Value in Health, 1, 3–13.CrossRefGoogle Scholar
  23. Kanninen, B. J. (2002). Optimal design for multinomial choice experiments. Journal of Marketing Research, 2, 214–227.CrossRefGoogle Scholar
  24. Kocur, G., Adler, T., Hyman, W., & Audet, E. (1982). Guide to forecasting travel demand with direct utility measurement. USA Department of Transportation, UMTA: Washington D. C.Google Scholar
  25. Kuhfeld, W. F., Tobias, R. D., & Garratt, M. (1994). Efficient Experimental Design with Marketing Research Applications. Journal of Marketing Research (JMR), 31(4), 545–557.CrossRefGoogle Scholar
  26. Louviere, J. J., & Hensher, D. (1982). Design and analysis of simulated choice or allocation experiments in travel choice modeling. Transportation Research Record, 890, 11–17.Google Scholar
  27. Louviere, J. J., & Hensher, D. A. (1983). Using discrete choice models with experimental-design data to forecast consumer demand for a unique cultural event. Journal of Consumer Research, 3, 348–361.CrossRefGoogle Scholar
  28. Louviere, J. J., & Kocur, G. (1983). The magnitude of individual-level variations in demand coefficients: A Xenia Ohio case example. Transportation Research, Part A, 5, 363–373.CrossRefGoogle Scholar
  29. Louviere, J. J., & Woodworth, G. (1983). Design and analysis of simulated consumer choice or allocation experiments - an approach based on aggregate data. Journal of Marketing Research, 4, 350–367.CrossRefGoogle Scholar
  30. Louviere, J. J., Meyer, R., Stetzer, F., & Beavers, L. L. (1973). Theory, methodology and findings in mode choice behavior Working Paper No., The Institute of Urban and Regional Research. Iowa City: The University of Iowa.Google Scholar
  31. Mitchell, R. C., & Carson, R. T. (Eds.). (1989). Using Surveys to Value Public-Goods - The Contingent Valuation Method. Press for Resources for the Future. Baltimore: John Hopkins Univ.Google Scholar
  32. Ojeda-Cabral, M., Hess, S. & Batley, R. (2016). Understanding valuation of travel time changes: are preferences different under different stated choice design settings? Transportation, 1-21.Google Scholar
  33. Rose, J. M., & Bliemer, M. C. J. (2008). Stated preference experimental design strategies. In D. A. Hensher & K. Button (Eds.), Handbook in transport (2nd ed.). Oxford: Elsevier Science.Google Scholar
  34. Rose, J. M., & Bliemer, M. C. J. (2009). Constructing efficient stated choice experimental designs. Transport Reviews, 5, 587–617.CrossRefGoogle Scholar
  35. Rose, J. M., & Bliemer, M. C. J. (2013). Sample size requirements for stated choice experiments. Transportation, 40(5), 1021–1041.CrossRefGoogle Scholar
  36. Rose, J. M., & Bliemer, M. C. J. (2014). Stated choice experimental design theory: The who, the what and the why ch. 7. In S. Hess & A. Daly (Eds.), Handbook of Choice Modelling (pp. 152–177). Cheltenham: Edward Elgar Publishing.Google Scholar
  37. Rose, J. M., Bliemer, M. C. J., Hensher, D. A., & Collins, A. T. (2008). Designing efficient stated choice experiments in the presence of reference alternatives. Transportation Research Part B, 4, 395–406.CrossRefGoogle Scholar
  38. Sándor, Z., & Wedel, M. (2001). Designing Conjoint Choice Experiments Using Managers’ Prior Beliefs. Journal of Marketing Research, 38(4), 430–444.CrossRefGoogle Scholar
  39. Sándor, Z., & Wedel, M. (2002). Profile construction in experimental choice designs for mixed logit models. Marketing Science, 21(4), 455–475.CrossRefGoogle Scholar
  40. Small, K. A. (1982). The scheduling of consumer activities: Work trips. American Economic Review, 3, 467–479.Google Scholar
  41. Street, D. J., & Burgess, L. (2004). Optimal and near-optimal pairs for the estimation of effects in 2-level choice experiments. Journal of Statistical Planning and Inference, 1–2, 185–199.CrossRefGoogle Scholar
  42. Street, D. J., Bunch, D. S., & Moore, B. J. (2001). Optimal designs for 2k paired comparison experiments. Communications in Statistics - Theory and Methods, 10, 2149–2171.CrossRefGoogle Scholar
  43. Street, D. J., Burgess, L., & Louviere, J. J. (2005). Quick and easy choice sets: Constructing optimal and nearly optimal stated choice experiments. International Journal of Research in Marketing, 4, 459–470.CrossRefGoogle Scholar
  44. Thorhauge, M., Cherchi, E. & Rich, J. (2014). Building efficient stated choice design for departure time choices using the scheduling model: Theoretical considerations and practical implications, Selected Proceedings from the Annual Transport Conference, Aalborg University, Denmark.Google Scholar
  45. Thorhauge, M., Cherchi, E., & Rich, J. (2016a). How flexible is flexible? Accounting for the effect of rescheduling possibilities in choice of departure time for work trips. Transportation Research Part A: Practice and Policy, 86, 177–193.Google Scholar
  46. Thorhauge, M., Haustein, S., & Cherchi, E. (2016b). Social psychology meets microeconometrics: Accounting for the theory of planned behaviour in departure time choice. Transportation Research Part F: Traffic Psychology and Behaviour, 38, 94–105.CrossRefGoogle Scholar
  47. Toner, J.P., Clark, S.D., Grant-Muller, S. & Fowkes, A.S. (1999). Anything you can do, we can do better: A provocative introduction to a new approach to stated preference design, World Transport Research, Vol 1-4.Google Scholar
  48. Walker, J. L., Wang, Y., Mikkel, T., & Ben-Akiva, E. (2015). D-EFFICIENT OR DEFICIENT? A robustness analysis of stated choice experimental designs, presented at the 94th annual meeting of the Transportation Research Board. Washington: D. C.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Department of Civil and Environmental Engineering, Center for Global Metropolitan StudiesUniversity of California, BerkeleyBerkeleyUSA
  2. 2.Department of Civil and Environmental EngineeringUniversity of California, BerkeleyBerkeleyUSA
  3. 3.Department of Management EngineeringTechnical University of DenmarkKongens LyngbyDenmark
  4. 4.Department of Civil and Environmental EngineeringMassachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations