Question order sensitivity of subjective well-being measures: focus on life satisfaction, self-rated health, and subjective life expectancy in survey instruments
This study examines the effect of question context created by order in questionnaires on three subjective well-being measures: life satisfaction, self-rated health, and subjective life expectancy.
We conducted two Web survey experiments. The first experiment (n = 648) altered the order of life satisfaction and self-rated health: (1) life satisfaction asked immediately after self-rated health; (2) self-rated health immediately after life satisfaction; and (3) two items placed apart. We examined their correlation coefficient by experimental condition and further examined its interaction with objective health. The second experiment (n = 479) asked life expectancy before and after parental mortality questions. Responses to life expectancy were compared by order using ANOVA, and we examined interaction with parental mortality status using ANCOVA. Additionally, response time and probes were examined.
Correlation coefficients between self-rated health and life satisfaction differed significantly by order: 0.313 (life satisfaction first), 0.508 (apart), and 0.643 (self-rated health first). Differences were larger among respondents with chronic conditions. Response times were the shortest when self-rated health was asked first. When life expectancy asked after parental mortality questions, respondents reported considering parents more for answering life expectancy; and respondents with deceased parents reported significantly lower expectancy, but not those whose parents were alive.
Question context effects exist. Findings suggest placing life satisfaction and self-rated health apart to avoid artificial attenuation or inflation in their association. Asking about parental mortality prior to life expectancy appears advantageous as this leads respondents to consider parental longevity more, an important factor for true longevity.
KeywordsInstrument design Question order Self-rated health Global life satisfaction Subjective life expectancy Survey research
The data used in this study were collected with funding from the Regula Herzog Young Investigator Fund awarded to Sunghee Lee.
Compliance with ethical standards
Conflict of interest
All authors declare no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This study protocol was approved by the Institutional Review Board at the University of Michigan.
- 2.Campbell, A., Converse, P. E., & Rodgers, W. L. (1976). The query of American-life. New York, NY: Russell Sage.Google Scholar
- 4.Kahneman, D., Diener, E., & Schwarz, N. (1999). Well-being: The foundations of hedonic psychology. New York, NY: Russell Sage Foundation.Google Scholar
- 5.Schwarz, N., & Strack, F. (1999). Report of subjective well-being: Judgmental processes and their methodological implications. In D. Kahneman, E. Diener, & N. Schwarz (Eds.), Well-being: The foundations of hedonic psychology (pp. 61–84). New York, NY: Russell Sage Foundation.Google Scholar
- 6.Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys. London: Academic Press.Google Scholar
- 7.Schwarz, N. (1996). Cognition and communication: judgmental biases, research methods, and the logic of conversation. Hillsdale, NJ: Erlbaum.Google Scholar
- 14.Sudman, S., Bradburn, N. M., & Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass.Google Scholar
- 16.Higgins, E. T. (1989). Knowledge accessibility and activation: Subjectivity and suffering from unconscious sources. In J. S. Uleman & J. A. Barghs (Eds.), Unintended thought (pp. 75–123). New York, NY: Guilford Press.Google Scholar
- 21.Organisation for Economic Cooperation and Development (OECD). (2013). OECD guidelines on measuring subjective well-being. OECD Publishing. Retrieved from http://dx.doi.org/10.1787/9789264191655-en.
- 26.Organisation for Economic Cooperation and Development (OECD). (2015). Life satisfaction. Retrieved from http://www.oecdbetterlifeindex.org/topics/life-satisfaction/.
- 38.Mohan, R., Beydoun, H. A., Beydoun, M. A., Barnes-Eley, M., Davis, J., Lance, R., & Schellhammer, P. (2011). Self-rated health as a tool for estimating health-adjusted life expectancy among patients newly diagnosed with localized prostate cancer: A preliminary study. Quality of Life Research, 20(5), 713–721.PubMedCrossRefGoogle Scholar
- 39.Kleinman, A. (1980). Patients and healers in the context of cultures. Berkeley, CA: University of California Press.Google Scholar
- 40.Bury, M. (1982). Chronic illness as biological disruption. Sociology of Health and Illness, 12, 451–468.Google Scholar
- 44.Lawton, M. P. (1977). Morale: What are we measuring? In C. N. Nydeggar (Ed.), Measuring morale: A guide to effective measurement. Washington, DC: The Gerontological Society.Google Scholar
- 46.Hurd, M. D., & McGarry, K. (1995). Evaluation of the subjective probabilities of survival in the health and retirement study. Journal of Human Resources, 30, S268–S292.Google Scholar
- 49.Lee, S., & Smith, J. (2016). Methodological aspects of subjective life expectancy: Effects of culture-specific reporting heterogeneity among older adults in the U.S. The Journals of Gerontology: Series B Psychological Sciences and Social Sciences, 71(3), 558–568.Google Scholar
- 50.Smith, T. W. (1982). Conditional order effect. GSS technical report. no. 33. Chicago, IL: NORC.Google Scholar
- 53.Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics 3: Speech acts (pp. 41–58). New York, NY: Academic Press.Google Scholar
- 63.Murphy, J., Link, M. W., Childs, J. H., Tesfaye, C. L., Dean, E., Stern, M., & Schober, M. F. (2014). Social media in public opinion research: Report of the AAPOR task force on emerging technologies in public opinion research. American Association for Public Opinion Research. Retrieved from https://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/AAPOR_Social_Media_Report_FNL.pdf.
- 67.Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.Google Scholar
- 68.Horton, J. J., & Chilton, L. B. (2010). The labor economics of paid crowdsourcing. In Proceedings of the 11th ACM Conference on Electronic Commerce (pp. 209–218). Association for Computing Machinery. Retrieved from http://arxiv.org/pdf/1001.0627.pdf.
- 71.Groves, R. M., Fultz, N. H., & Martin, E. (1992). Direct questioning about comprehension in a survey setting. Questions about questions: Inquiries into the cognitive bases of surveys. New York, NY: Russell Sage, 49–61.Google Scholar
- 77.Shannon, C. E. (1948). A mathematical theory of communication. The Bell Systems Technical Journal, 27, 379–423 and 623–656.Google Scholar
- 78.Fisher, R. A. (1921). On the probable error of a coefficient of correlation deduced from a small sample. Metron, 1, 3–32.Google Scholar
- 80.Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar