Skip to main content

Statistical Analysis of Choice Experiments and Surveys

Abstract

Measures of households' past behavior, their expectations with respect to future events and contingencies, and their intentions with respect to future behavior are frequently collected using household surveys. These questions are conceptually difficult. Answering them requires elaborate cognitive and social processes, and often respondents report only their “best” guesses and/or estimates, using more or less sophisticated heuristics. A large body of literature in psychology and survey research shows that as a result, responses to such questions may be severely biased. In this paper, (1) we describe some of the problems that are typically encountered, (2) provide some empirical illustrations of these biases, and (3) develop a framework for conceptualizing survey response behavior and for integrating structural models of response behavior into the statistical analysis of the underlying economic behavior.

This is a preview of subscription content, access via your institution.

References

  1. Adams, P., M. D. Hurd, D. McFadden, A. Merrill, and T. Ribeiro. (2003). “Healthy, Wealthy, and Wise? Tests for Direct Causal Paths Between Health and Socioeconomic Status,” Journal of Econometrics 112(1), 3–56.

    Google Scholar 

  2. Alexander, C. S. and H. J. Becker. (1978). “The Use of Vignettes in Survey Research,” Public Opinion Quarterly 42(1), 93–104.

    Article  Google Scholar 

  3. Allais, M. (1953). “Le Comportement De L'homme Rationnel Devant Le Risque: Critique Des Postulats Et Axiomes De L'école Américaine,” Econometrica 21(4), 503–546.

    Google Scholar 

  4. Banks, J., A. Kapteyn, J. P. Smith, and A. van Soest. (2004). “International Comparisons of Work Disability,” Working Paper No. 155, Labor and Population Group, RAND, Santa Monica, CA.

  5. Bemmaor, A. C. (1995). “Predicting Behavior from Intention-to-buy Measures: The Parametric Case,” Journal of Marketing Research 32(May), 176–191.

    Google Scholar 

  6. Ben-Akiva, M., D. McFadden, M. Abe, U. BÖckenholt, D. Bolduc, D. Gopinath, T. Morikawa, V. Ramaswamy, V. Rao, D. Revelt, and D. Steinberg. (1997). “Modeling Methods for Discrete Choice Analysis,” Marketing Letters 8(3), 273–286.

    Article  Google Scholar 

  7. Ben-Akiva, M., D. McFadden, T. Gärling, D. Gopinath, D. Bolduc, A. BÖrsch-Supan, P. Delquié, O. Larichev, T. Morikawa, A. Polydoropoulou, and V. Rao. (1999). “Extended Framework for Modeling Choice Behavior,” Marketing Letters 10(3), 187–203.

    Article  Google Scholar 

  8. Ben-Akiva, M., D. McFadden, K. Train, J. Walker, C. Bhat, M. Bierlaire, D. Bolduc, A. BÖrsch-Supan, D. Brownstone, D. S. Bunch, A. Daly, A. de Palma, D. Gopinath, A. Karlstrom, and M. Munizaga. (2002). “Hybrid Choice Models: Progress and Challenges,” Marketing Letters 13(3), 163–175.

    Article  Google Scholar 

  9. Bergadaá, M. M. (1990). “The role of Time in the Action of the Consumer,” Journal of Consumer Research 17(3), 289–302.

    Google Scholar 

  10. Bound, J., C. Brown, and N. Mathiowetz. (2001). “Measurement Error in Survey Data.” In J. J. Heckman and E. Leamer (eds.), Handbook of Econometrics, Volume 5, Amsterdam: Elsevier, pp. 3705–3843.

    Google Scholar 

  11. Bruine de Bruin, W., P. S. Fischbeck, N. A. Stiber, and B. Fischhoff. (2002). “What Number is Fifty-Fifty? Redistributing Excessive 50% Responses in Elicited Probabilities,” Risk Analysis 22(4), 713–723.

    Google Scholar 

  12. Buchanan, B. and G. Morrison. (1987). “Sampling Properties of Rate Questions with Implications for Survey Research,” Marketing Science 16(3), 286–298.

    Google Scholar 

  13. Chandon, P., V. G. Morwitz, and W. J. Reinartz. (2005). “Do intentions Really Predict Behavior? Self-generated Validity Effects in Survey Research,” Journal of Marketing 69(2), 1–14.

    Article  Google Scholar 

  14. Cowing, T. and D. McFadden. (1984). Microeconomic Modeling and Policy Analysis: Studies in Residential Energy Demand. New York: Academic Press.

    Google Scholar 

  15. Crossley, T. F. and S. Kennedy. (2002). “The Reliability of Self-assessed Health Status,” Journal of Health Economics 21(4), 643–658.

    Article  Google Scholar 

  16. Davidson, D., P. Suppes, and S. Siegel. (1956). Decision Making: An Experimental Approach. Stanford, CA: Stanford University Press.

    Google Scholar 

  17. Dex, S. (1995). “The Reliability of Recall Data: A literature Review,” Bulletin de Méthodologie Sociologique, 49, December, 58–89.

    Google Scholar 

  18. Dominitz, J., and C. F. Manski. (1997). “Using Expectations Data to Study Subjective Income Expectations,” Journal of the American Statistical Association 92(439), 855–867.

    Google Scholar 

  19. Dominitz, J. and C. F. Manski. (2004). “How Should We Measure Consumer Confidence?” Journal of Economic Perspectives 18(2), 51–66.

    Article  Google Scholar 

  20. Feldman, J. M. and J. G. Lynch Jr. (1988). “Self-generated Validity and Other Effects of Measurement on Belief, Attitude, Intention, and Behavior,” Journal of Applied Psychology 73(3), 421–435.

    Article  Google Scholar 

  21. Gan, L., M. D. Hurd, and D. McFadden. (2003). “Individual Subjective Survival Curves,” Working Paper No. 9480, National Bureau of Economic Research (NBER), Cambridge, MA.

    Google Scholar 

  22. Gardiner, J. M., and A. Richardson-Klavehn. (2000). “Remembering and Knowing,” in E. Tulving and F. I. M. Craik (eds.), Handbook of Memory, New York: Oxford University Press, pp. 229–244.

    Google Scholar 

  23. Granbois, D. H. and J. O. Summers. (1975). “Primary and Secondary Validity of Consumer Purchase Probabilities,” Journal of Consumer Research 1(4), 31–38.

    Article  Google Scholar 

  24. Harris, K. M. and M. P. Keane. (1999). “A Model of Health Plan Choice: Inferring Preferences and Perceptions from a Combination of Revealed Preference and Attitudinal Data,” Journal of Econometrics 89(1–2), 131–157.

    Google Scholar 

  25. Hill, D., M. Perry, and R. J. Willis. (2004). “Estimating Knightian Uncertainty from Probability Questions on the HRS,” unpublished manuscript, University of Michigan.

  26. Hsiao, C. and B.-H. Sun. (1999), “Modeling Response Bias – With an Analysis of the Demand for an Advanced Electronic Device,” Journal of Econometrics 89(1–2), 15–39.

    Google Scholar 

  27. Hurd, M. D., D. McFadden, H. Chand, L. Gan, A. Merrill, and M. Roberts. (1998). “Consumption and Savings Balances of the Elderly: Experimental Evidence on Survey Response Bias.” in D. Wise (ed.), Frontiers in the Economics of Aging, Chicago, IL: University of Chicago Press, pp. 353–387.

    Google Scholar 

  28. Hurd, M. D. and K. McGarry. (1995). “Evaluation of the Subjective Probabilities of Survival in the Health and Retirement Study,” Journal of Human Resources 30(Supplement), S268–S292.

    Google Scholar 

  29. Hurd, M. D. and K. McGarry. (2002). “The Predictive Validity of Subjective Probabilities of Survival,” Economic Journal 112(482), 966–985.

    Article  Google Scholar 

  30. Infosino, W. J. (1986). “Forecasting New Product Sales from Likelihood of Purchase Ratings,” Marketing Science 5(4), 372–384.

    Google Scholar 

  31. Kalwani, M. U. and A. J. Silk. (1982). “On the Reliability and Predictive Validity of Purchase Intention Measures,” Marketing Science 1(3), 243–286.

    Google Scholar 

  32. King, G., C. Murray, J. A. Salomon, and A. Tandon. (2004). “Enhancing the Validity and Cross-cultural Comparability of Measurement in Survey Research,” American Political Science Review 98(1), 191–207.

    Article  Google Scholar 

  33. Klein, R. and R. Sherman. (1997). “Estimating New Product Demand from Biased Survey Data,” Journal of Econometrics 76(1–2), 53–76.

    Google Scholar 

  34. Lewbel, A. (2004). “Simple Estimators for Hard Problems: Endogeneity in Binary Choice and Other Nonlinear Models,” unpublished manuscript, Boston College.

  35. Louviere, J. J., D. A. Hensher, and J. D. Swait (2000). Stated Choice Methods: Analysis and Application. Cambridge, UK: Cambridge University Press.

    Google Scholar 

  36. Louviere, J. J., D. Street, R. Carson, A. Ainslie, J. R. Deshazo, T. Cameron, D. Hensher, R. Kohn, and T. Marley. (2002). “Dissecting the Random Component of Utility”, Marketing Letters 13(3), 163–176.

    Article  Google Scholar 

  37. Manski, C. F. (1999). “Analysis of Choice Expectations in Incomplete Scenarios,” Journal of Risk and Uncertainty 19(1–3), 49–65.

    Google Scholar 

  38. Manski, C. F. (2004). “Measuring Expectations,” Econometrica 72(5), 1329–1376.

    Article  Google Scholar 

  39. Matzkin, R. (2004). “Unobservable Instruments,” unpublished manuscript, Northwestern University.

  40. McFadden, D. (2004). “Foundations of Economic Survey Research,” unpublished manuscript, Berkeley: University of California.

  41. McFadden, D., N. Schwarz, and J. Winter. (2004), “Measuring Perceptions and Behavior in Household Surveys,” paper presented at the Annual Meeting of the American Economic Association, San Diego, CA.

  42. McFadden, D. and K. Train (2000), “Mixed MNL Models for Discrete Response,” Journal of Applied Econometrics 15(5), 447–470.

    Article  Google Scholar 

  43. McGarry, K. (2004). “Health and Retirement: Do Changes in Health Affect Retirement Expectations?” Journal of Human Resources 39(3), 624–648.

    Google Scholar 

  44. Menon, G., O. Raghubir, and N. Schwarz. (1997). “How Much Will I Spend? Factors Affecting Consumers' Estimates of Future Expense,” Journal of Consumer Psychology 6(2), 141–164.

    Article  Google Scholar 

  45. Morwitz, V. G. and G. J. Fitzsimons. (2004). “The Mere Measurement Effect: Why Does Measuring Intentions Change Actual Behavior?” Journal of Consumer Psychology 14(1–2), 64–74.

    Google Scholar 

  46. Molinari, F. (2005). “Cognitive Processes and Self/proxy Response Status: Evidence from HRS 2000,” unpublished manuscript, Cornell University.

  47. Poterba, J. and L. Summers (1986). “Reporting Errors and Labor Market Dynamics,” Econometrica 54(6), 1319–1338.

    Google Scholar 

  48. Schwarz, N. (2003). “Self-Reports in Consumer Research: The Challenge of Comparing Cohorts and Cultures,” Journal of Consumer Research 29(4), 588–594.

    Article  Google Scholar 

  49. Schwarz, N., R. M. Groves, and H. Schuman. (1998). “Survey Methods.” In N. Schwarz, R. Groves, and H. Schuman (eds.), Handbook of Social Psychology, 4th Edition, New York: McGraw-Hill, pp. 143–179.

    Google Scholar 

  50. Schwarz, N., H. J. Hippler, B. Deutsch, and F. Strack. (1985). “Response Categories: Effects on Behavioral Reports and Comparative Judgements,” Public Opinion Quarterly 49(3), 388–395.

    Article  Google Scholar 

  51. Smith, V. K., D. H. Taylor, and F. A. Sloan. (2001). “Longevity Expectations and Death: Can People Predict Their Own Demise?” American Economic Review 91(4), 1126–1134.

    Google Scholar 

  52. Tourangeau, R., L. J. Rips, and K. Rasinski. (2000). The Psychology of Survey Response. New York and Cambridge, UK: Cambridge University Press.

    Google Scholar 

  53. Tversky, A. and D. Kahneman (1974). “Judgement Under Uncertainty: Heuristics and Biases,” Science 185(4157), 1124–1131.

    Google Scholar 

  54. Wansbeek, T. and E. Meijer (2000). Measurement Error and Latent Variables in Econometrics, Amsterdam: Elsevier.

    Google Scholar 

  55. Wason, K., Polonsky, M., and Hyman, M. (2002). “Designing Vignette Studies in Marketing,” Australasian Marketing Journal 10(3), 41–58.

    Google Scholar 

  56. Winter, J. (2002). “Bracketing Effects in Categorized Survey Questions and the Measurement of Economic Quantities,” Discussion Paper No. 02-35, Sonderforschungsbereich 504, University of Mannheim.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Joachim K. Winter.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

McFadden, D.L., Bemmaor, A.C., Caro, F.G. et al. Statistical Analysis of Choice Experiments and Surveys. Market Lett 16, 183–196 (2005). https://doi.org/10.1007/s11002-005-5884-2

Download citation

Keywords

  • consumer surveys
  • survey response error
  • hypothetical choice
  • applied econometrics