Marketing Letters

, Volume 16, Issue 3–4, pp 183–196 | Cite as

Statistical Analysis of Choice Experiments and Surveys

  • Daniel L. McFadden
  • Albert C. Bemmaor
  • Francis G. Caro
  • Jeff Dominitz
  • Byung-Hill Jun
  • Arthur Lewbel
  • Rosa L. Matzkin
  • Francesca Molinari
  • Norbert Schwarz
  • Robert J. Willis
  • Joachim K. WinterEmail author


Measures of households' past behavior, their expectations with respect to future events and contingencies, and their intentions with respect to future behavior are frequently collected using household surveys. These questions are conceptually difficult. Answering them requires elaborate cognitive and social processes, and often respondents report only their “best” guesses and/or estimates, using more or less sophisticated heuristics. A large body of literature in psychology and survey research shows that as a result, responses to such questions may be severely biased. In this paper, (1) we describe some of the problems that are typically encountered, (2) provide some empirical illustrations of these biases, and (3) develop a framework for conceptualizing survey response behavior and for integrating structural models of response behavior into the statistical analysis of the underlying economic behavior.


consumer surveys survey response error hypothetical choice applied econometrics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Adams, P., M. D. Hurd, D. McFadden, A. Merrill, and T. Ribeiro. (2003). “Healthy, Wealthy, and Wise? Tests for Direct Causal Paths Between Health and Socioeconomic Status,” Journal of Econometrics 112(1), 3–56.Google Scholar
  2. Alexander, C. S. and H. J. Becker. (1978). “The Use of Vignettes in Survey Research,” Public Opinion Quarterly 42(1), 93–104.CrossRefGoogle Scholar
  3. Allais, M. (1953). “Le Comportement De L'homme Rationnel Devant Le Risque: Critique Des Postulats Et Axiomes De L'école Américaine,” Econometrica 21(4), 503–546.Google Scholar
  4. Banks, J., A. Kapteyn, J. P. Smith, and A. van Soest. (2004). “International Comparisons of Work Disability,” Working Paper No. 155, Labor and Population Group, RAND, Santa Monica, CA.Google Scholar
  5. Bemmaor, A. C. (1995). “Predicting Behavior from Intention-to-buy Measures: The Parametric Case,” Journal of Marketing Research 32(May), 176–191.Google Scholar
  6. Ben-Akiva, M., D. McFadden, M. Abe, U. BÖckenholt, D. Bolduc, D. Gopinath, T. Morikawa, V. Ramaswamy, V. Rao, D. Revelt, and D. Steinberg. (1997). “Modeling Methods for Discrete Choice Analysis,” Marketing Letters 8(3), 273–286.CrossRefGoogle Scholar
  7. Ben-Akiva, M., D. McFadden, T. Gärling, D. Gopinath, D. Bolduc, A. BÖrsch-Supan, P. Delquié, O. Larichev, T. Morikawa, A. Polydoropoulou, and V. Rao. (1999). “Extended Framework for Modeling Choice Behavior,” Marketing Letters 10(3), 187–203.CrossRefGoogle Scholar
  8. Ben-Akiva, M., D. McFadden, K. Train, J. Walker, C. Bhat, M. Bierlaire, D. Bolduc, A. BÖrsch-Supan, D. Brownstone, D. S. Bunch, A. Daly, A. de Palma, D. Gopinath, A. Karlstrom, and M. Munizaga. (2002). “Hybrid Choice Models: Progress and Challenges,” Marketing Letters 13(3), 163–175.CrossRefGoogle Scholar
  9. Bergadaá, M. M. (1990). “The role of Time in the Action of the Consumer,” Journal of Consumer Research 17(3), 289–302.Google Scholar
  10. Bound, J., C. Brown, and N. Mathiowetz. (2001). “Measurement Error in Survey Data.” In J. J. Heckman and E. Leamer (eds.), Handbook of Econometrics, Volume 5, Amsterdam: Elsevier, pp. 3705–3843.Google Scholar
  11. Bruine de Bruin, W., P. S. Fischbeck, N. A. Stiber, and B. Fischhoff. (2002). “What Number is Fifty-Fifty? Redistributing Excessive 50% Responses in Elicited Probabilities,” Risk Analysis 22(4), 713–723.Google Scholar
  12. Buchanan, B. and G. Morrison. (1987). “Sampling Properties of Rate Questions with Implications for Survey Research,” Marketing Science 16(3), 286–298.Google Scholar
  13. Chandon, P., V. G. Morwitz, and W. J. Reinartz. (2005). “Do intentions Really Predict Behavior? Self-generated Validity Effects in Survey Research,” Journal of Marketing 69(2), 1–14.CrossRefGoogle Scholar
  14. Cowing, T. and D. McFadden. (1984). Microeconomic Modeling and Policy Analysis: Studies in Residential Energy Demand. New York: Academic Press.Google Scholar
  15. Crossley, T. F. and S. Kennedy. (2002). “The Reliability of Self-assessed Health Status,” Journal of Health Economics 21(4), 643–658.CrossRefGoogle Scholar
  16. Davidson, D., P. Suppes, and S. Siegel. (1956). Decision Making: An Experimental Approach. Stanford, CA: Stanford University Press.Google Scholar
  17. Dex, S. (1995). “The Reliability of Recall Data: A literature Review,” Bulletin de Méthodologie Sociologique, 49, December, 58–89.Google Scholar
  18. Dominitz, J., and C. F. Manski. (1997). “Using Expectations Data to Study Subjective Income Expectations,” Journal of the American Statistical Association 92(439), 855–867.Google Scholar
  19. Dominitz, J. and C. F. Manski. (2004). “How Should We Measure Consumer Confidence?” Journal of Economic Perspectives 18(2), 51–66.CrossRefGoogle Scholar
  20. Feldman, J. M. and J. G. Lynch Jr. (1988). “Self-generated Validity and Other Effects of Measurement on Belief, Attitude, Intention, and Behavior,” Journal of Applied Psychology 73(3), 421–435.CrossRefGoogle Scholar
  21. Gan, L., M. D. Hurd, and D. McFadden. (2003). “Individual Subjective Survival Curves,” Working Paper No. 9480, National Bureau of Economic Research (NBER), Cambridge, MA.Google Scholar
  22. Gardiner, J. M., and A. Richardson-Klavehn. (2000). “Remembering and Knowing,” in E. Tulving and F. I. M. Craik (eds.), Handbook of Memory, New York: Oxford University Press, pp. 229–244.Google Scholar
  23. Granbois, D. H. and J. O. Summers. (1975). “Primary and Secondary Validity of Consumer Purchase Probabilities,” Journal of Consumer Research 1(4), 31–38.CrossRefGoogle Scholar
  24. Harris, K. M. and M. P. Keane. (1999). “A Model of Health Plan Choice: Inferring Preferences and Perceptions from a Combination of Revealed Preference and Attitudinal Data,” Journal of Econometrics 89(1–2), 131–157.Google Scholar
  25. Hill, D., M. Perry, and R. J. Willis. (2004). “Estimating Knightian Uncertainty from Probability Questions on the HRS,” unpublished manuscript, University of Michigan.Google Scholar
  26. Hsiao, C. and B.-H. Sun. (1999), “Modeling Response Bias – With an Analysis of the Demand for an Advanced Electronic Device,” Journal of Econometrics 89(1–2), 15–39.Google Scholar
  27. Hurd, M. D., D. McFadden, H. Chand, L. Gan, A. Merrill, and M. Roberts. (1998). “Consumption and Savings Balances of the Elderly: Experimental Evidence on Survey Response Bias.” in D. Wise (ed.), Frontiers in the Economics of Aging, Chicago, IL: University of Chicago Press, pp. 353–387.Google Scholar
  28. Hurd, M. D. and K. McGarry. (1995). “Evaluation of the Subjective Probabilities of Survival in the Health and Retirement Study,” Journal of Human Resources 30(Supplement), S268–S292.Google Scholar
  29. Hurd, M. D. and K. McGarry. (2002). “The Predictive Validity of Subjective Probabilities of Survival,” Economic Journal 112(482), 966–985.CrossRefGoogle Scholar
  30. Infosino, W. J. (1986). “Forecasting New Product Sales from Likelihood of Purchase Ratings,” Marketing Science 5(4), 372–384.Google Scholar
  31. Kalwani, M. U. and A. J. Silk. (1982). “On the Reliability and Predictive Validity of Purchase Intention Measures,” Marketing Science 1(3), 243–286.Google Scholar
  32. King, G., C. Murray, J. A. Salomon, and A. Tandon. (2004). “Enhancing the Validity and Cross-cultural Comparability of Measurement in Survey Research,” American Political Science Review 98(1), 191–207.CrossRefGoogle Scholar
  33. Klein, R. and R. Sherman. (1997). “Estimating New Product Demand from Biased Survey Data,” Journal of Econometrics 76(1–2), 53–76.Google Scholar
  34. Lewbel, A. (2004). “Simple Estimators for Hard Problems: Endogeneity in Binary Choice and Other Nonlinear Models,” unpublished manuscript, Boston College.Google Scholar
  35. Louviere, J. J., D. A. Hensher, and J. D. Swait (2000). Stated Choice Methods: Analysis and Application. Cambridge, UK: Cambridge University Press.Google Scholar
  36. Louviere, J. J., D. Street, R. Carson, A. Ainslie, J. R. Deshazo, T. Cameron, D. Hensher, R. Kohn, and T. Marley. (2002). “Dissecting the Random Component of Utility”, Marketing Letters 13(3), 163–176.CrossRefGoogle Scholar
  37. Manski, C. F. (1999). “Analysis of Choice Expectations in Incomplete Scenarios,” Journal of Risk and Uncertainty 19(1–3), 49–65.Google Scholar
  38. Manski, C. F. (2004). “Measuring Expectations,” Econometrica 72(5), 1329–1376.CrossRefGoogle Scholar
  39. Matzkin, R. (2004). “Unobservable Instruments,” unpublished manuscript, Northwestern University.Google Scholar
  40. McFadden, D. (2004). “Foundations of Economic Survey Research,” unpublished manuscript, Berkeley: University of California.Google Scholar
  41. McFadden, D., N. Schwarz, and J. Winter. (2004), “Measuring Perceptions and Behavior in Household Surveys,” paper presented at the Annual Meeting of the American Economic Association, San Diego, CA.Google Scholar
  42. McFadden, D. and K. Train (2000), “Mixed MNL Models for Discrete Response,” Journal of Applied Econometrics 15(5), 447–470.CrossRefGoogle Scholar
  43. McGarry, K. (2004). “Health and Retirement: Do Changes in Health Affect Retirement Expectations?” Journal of Human Resources 39(3), 624–648.Google Scholar
  44. Menon, G., O. Raghubir, and N. Schwarz. (1997). “How Much Will I Spend? Factors Affecting Consumers' Estimates of Future Expense,” Journal of Consumer Psychology 6(2), 141–164.CrossRefGoogle Scholar
  45. Morwitz, V. G. and G. J. Fitzsimons. (2004). “The Mere Measurement Effect: Why Does Measuring Intentions Change Actual Behavior?” Journal of Consumer Psychology 14(1–2), 64–74.Google Scholar
  46. Molinari, F. (2005). “Cognitive Processes and Self/proxy Response Status: Evidence from HRS 2000,” unpublished manuscript, Cornell University.Google Scholar
  47. Poterba, J. and L. Summers (1986). “Reporting Errors and Labor Market Dynamics,” Econometrica 54(6), 1319–1338.Google Scholar
  48. Schwarz, N. (2003). “Self-Reports in Consumer Research: The Challenge of Comparing Cohorts and Cultures,” Journal of Consumer Research 29(4), 588–594.CrossRefGoogle Scholar
  49. Schwarz, N., R. M. Groves, and H. Schuman. (1998). “Survey Methods.” In N. Schwarz, R. Groves, and H. Schuman (eds.), Handbook of Social Psychology, 4th Edition, New York: McGraw-Hill, pp. 143–179.Google Scholar
  50. Schwarz, N., H. J. Hippler, B. Deutsch, and F. Strack. (1985). “Response Categories: Effects on Behavioral Reports and Comparative Judgements,” Public Opinion Quarterly 49(3), 388–395.CrossRefGoogle Scholar
  51. Smith, V. K., D. H. Taylor, and F. A. Sloan. (2001). “Longevity Expectations and Death: Can People Predict Their Own Demise?” American Economic Review 91(4), 1126–1134.Google Scholar
  52. Tourangeau, R., L. J. Rips, and K. Rasinski. (2000). The Psychology of Survey Response. New York and Cambridge, UK: Cambridge University Press.Google Scholar
  53. Tversky, A. and D. Kahneman (1974). “Judgement Under Uncertainty: Heuristics and Biases,” Science 185(4157), 1124–1131.Google Scholar
  54. Wansbeek, T. and E. Meijer (2000). Measurement Error and Latent Variables in Econometrics, Amsterdam: Elsevier.Google Scholar
  55. Wason, K., Polonsky, M., and Hyman, M. (2002). “Designing Vignette Studies in Marketing,” Australasian Marketing Journal 10(3), 41–58.Google Scholar
  56. Winter, J. (2002). “Bracketing Effects in Categorized Survey Questions and the Measurement of Economic Quantities,” Discussion Paper No. 02-35, Sonderforschungsbereich 504, University of Mannheim.Google Scholar

Copyright information

© Springer Science + Business Media, Inc. 2005

Authors and Affiliations

  • Daniel L. McFadden
    • 1
  • Albert C. Bemmaor
    • 2
  • Francis G. Caro
    • 3
  • Jeff Dominitz
    • 4
  • Byung-Hill Jun
    • 5
  • Arthur Lewbel
    • 6
  • Rosa L. Matzkin
    • 7
  • Francesca Molinari
    • 8
  • Norbert Schwarz
    • 9
  • Robert J. Willis
    • 10
  • Joachim K. Winter
    • 11
    Email author
  1. 1.Econometrics LaboratoryUniversity of CaliforniaBerkeley
  2. 2.ESSECCergy-Pontoise CedexFrance
  3. 3.Gerontology InstituteUniversity of Massachusetts BostonBoston
  4. 4.H. John Heinz III, School of Public Policy and ManagementCarnegie Mellon UniversityPittsburgh
  5. 5.Econometrics LaboratoryUniversity of CaliforniaBerkeley
  6. 6.Department of EconomicsBoston CollegeChestnut Hill
  7. 7.Department of EconomicsNorthwestern UniversityEvanston
  8. 8.Department of EconomicsCornell UniversityIthaca
  9. 9.Institute for Social ResearchUniversity of MichiganAnn Arbor
  10. 10.Department of Economics and Institute for Social ResearchUniversity of MichiganAnn Arbor
  11. 11.Department of EconomicsUniversity of MunichMunichGermany

Personalised recommendations