Abstract
Purpose
This study examines the effect of question context created by order in questionnaires on three subjective well-being measures: life satisfaction, self-rated health, and subjective life expectancy.
Methods
We conducted two Web survey experiments. The first experiment (n = 648) altered the order of life satisfaction and self-rated health: (1) life satisfaction asked immediately after self-rated health; (2) self-rated health immediately after life satisfaction; and (3) two items placed apart. We examined their correlation coefficient by experimental condition and further examined its interaction with objective health. The second experiment (n = 479) asked life expectancy before and after parental mortality questions. Responses to life expectancy were compared by order using ANOVA, and we examined interaction with parental mortality status using ANCOVA. Additionally, response time and probes were examined.
Results
Correlation coefficients between self-rated health and life satisfaction differed significantly by order: 0.313 (life satisfaction first), 0.508 (apart), and 0.643 (self-rated health first). Differences were larger among respondents with chronic conditions. Response times were the shortest when self-rated health was asked first. When life expectancy asked after parental mortality questions, respondents reported considering parents more for answering life expectancy; and respondents with deceased parents reported significantly lower expectancy, but not those whose parents were alive.
Conclusion
Question context effects exist. Findings suggest placing life satisfaction and self-rated health apart to avoid artificial attenuation or inflation in their association. Asking about parental mortality prior to life expectancy appears advantageous as this leads respondents to consider parental longevity more, an important factor for true longevity.
Similar content being viewed by others
References
Diener, E., Oishi, S., & Lucas, R. E. (2003). Personality, culture, and subjective well-being: Emotional and cognitive evaluations of life. Annual Review of Psychology, 54(1), 403–425.
Campbell, A., Converse, P. E., & Rodgers, W. L. (1976). The query of American-life. New York, NY: Russell Sage.
Ryan, R. M., & Deci, E. L. (2001). On happiness and human potentials: A review of research on hedonic and eudaimonic well-being. Annual Review of Psychology, 52(1), 141–166.
Kahneman, D., Diener, E., & Schwarz, N. (1999). Well-being: The foundations of hedonic psychology. New York, NY: Russell Sage Foundation.
Schwarz, N., & Strack, F. (1999). Report of subjective well-being: Judgmental processes and their methodological implications. In D. Kahneman, E. Diener, & N. Schwarz (Eds.), Well-being: The foundations of hedonic psychology (pp. 61–84). New York, NY: Russell Sage Foundation.
Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys. London: Academic Press.
Schwarz, N. (1996). Cognition and communication: judgmental biases, research methods, and the logic of conversation. Hillsdale, NJ: Erlbaum.
Schuman, H. (1992). Context effects: State of the past/state of the art. In N. Schwarz & S. Sudman (Eds.), Context effects in social and psychological research (pp. 5–20). New York, NY: Springer.
Schwarz, N. (1999). Self-reports: how the questions shape the answers. American Psychologist, 54(2), 93.
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.
Strack, F., Schwarz, N., & Wänke, M. (1991). Semantic and pragmatic aspects of context effects in social and psychological research. Social Cognition, 9(1), 111–125.
Lee, S., & Schwarz, N. (2014). Question context and priming meaning of health: Effect on differences in self-rated health between Hispanics and non-Hispanic whites. American Journal of Public Health, 104(1), 179–185.
Garbarski, D., Schaeffer, N. C., & Dykema, J. (2015). The effects of response option order and question order on self-rated health. Quality of Life Research, 24, 1443–1453.
Sudman, S., Bradburn, N. M., & Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass.
Zaller, J. (1992). The nature and origins of mass opinion. Cambridge: Cambridge University Press.
Higgins, E. T. (1989). Knowledge accessibility and activation: Subjectivity and suffering from unconscious sources. In J. S. Uleman & J. A. Barghs (Eds.), Unintended thought (pp. 75–123). New York, NY: Guilford Press.
Tourangeau, R., & Rasinski, K. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103(3), 299–314.
Wyer, R. S., & Srull, T. K. (1986). Human cognition in its social context. Psychological Review, 93(3), 322–359.
Kahneman, D., & Tversky, A. (1996). On the reality of cognitive illusions. Psychological Review, 103(3), 582–591.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.
Organisation for Economic Cooperation and Development (OECD). (2013). OECD guidelines on measuring subjective well-being. OECD Publishing. Retrieved from http://dx.doi.org/10.1787/9789264191655-en.
Pavot, W., & Diener, E. (1993). Review of the satisfaction with life scale. Psychological Assessment, 5(2), 164–172.
Diener, E. (1994). Assessing subjective well-being: Progress and opportunities. Social Indicators Research, 31, 103–157.
Barbato, A., Monzani, E., & Schiavi, T. (2004). Life satisfaction in a sample of outpatients with severe mental disorders: A survey in northern Italy. Quality of Life Research, 13, 969–973.
Bjørnskov, C. (2010). How comparable are the Gallup World Poll life satisfaction data? Journal of Happiness Studies, 11(1), 41–60.
Organisation for Economic Cooperation and Development (OECD). (2015). Life satisfaction. Retrieved from http://www.oecdbetterlifeindex.org/topics/life-satisfaction/.
Idler, E. L., & Kasl, S. (1991). Health perceptions and survival: Do global evaluations of health status really predict mortality? Journal of Gerontology, 46(2), S55–S65.
Ferraro, K. F., Farmer, M. M., & Wybraniec, J. A. (1997). Health trajectories: Long-term dynamics among black and white adults. Journal of Health and Social Behavior, 38(1), 38–54.
Kuusio, H., Heponiemi, T., Aalto, A. M., Sinervo, T., & Elovainio, M. (2012). Differences in well-being between GPs, medical specialists, and private physicians: The role of psychosocial factors. Health Services Research, 47(1), 68–85.
Riise, H. K. R., Riise, T., Natvig, G. K., & Daltveit, A. K. (2014). Poor self-rated health associated with an increased risk of subsequent development of lung cancer. Quality of Life Research, 23, 145–153.
Shi, L., Starfield, B., Politzer, R., & Regan, J. (2002). Primary care, self-rated health, and reductions in social disparities in health. Health Services Research, 37(3), 529–550.
DeSalvo, K. B., Fan, V. S., McDonell, M. B., & Fihn, S. D. (2005). Predicting mortality and healthcare utilization with a single question. Health Services Research, 40(4), 1234–1246.
Veenstra, M., Moum, T., & Garratt, A. M. (2006). Patient experiences with information in a hospital setting: Associations with coping and self-rated health in chronic illness. Quality of Life Research, 15, 967–978.
Andersen, F. K., Christensen, K., & Frederiksen, H. (2007). Self-rated health and age: A cross-sectional and longitudinal study of 11,000 Danes aged 45–102. Scandinavian Journal of Public Health, 35(2), 164–171.
Fleishman, J. A., & Cohen, J. W. (2010). Using information on clinical conditions to predict high-cost patients. Health Services Research, 45(2), 532–552.
Mossey, J. M., & Shapiro, E. (1982). Self-rated health: A predictor of mortality among the elderly. American Journal of Public Health, 72(8), 800–808.
Idler, E. L., & Benyamini, Y. (1997). Self-rated health and mortality: A review of twenty-seven community studies. Journal of Health and Social Behavior, 38(1), 21–37.
Mohan, R., Beydoun, H. A., Beydoun, M. A., Barnes-Eley, M., Davis, J., Lance, R., & Schellhammer, P. (2011). Self-rated health as a tool for estimating health-adjusted life expectancy among patients newly diagnosed with localized prostate cancer: A preliminary study. Quality of Life Research, 20(5), 713–721.
Kleinman, A. (1980). Patients and healers in the context of cultures. Berkeley, CA: University of California Press.
Bury, M. (1982). Chronic illness as biological disruption. Sociology of Health and Illness, 12, 451–468.
Zullig, K. J., & White, R. J. (2011). Physical activity, life satisfaction, and self-rated health of middle school students. Applied Research in Quality of Life, 6(3), 277–289.
Herman, K. M., Hopman, W. M., & Rosenberg, M. W. (2013). Self-rated health and life satisfaction among Canadian adults: associations of perceived weight status versus BMI. Quality of Life Research, 22, 2693–2705.
Okun, M. A., & George, L. K. (1984). Physician-and self-ratings of health, neuroticism and subjective well-being among men and women. Personality and Individual Differences, 5(5), 533–539.
Lawton, M. P. (1977). Morale: What are we measuring? In C. N. Nydeggar (Ed.), Measuring morale: A guide to effective measurement. Washington, DC: The Gerontological Society.
Elder, T. E. (2013). The predictive validity of subjective mortality expectations: evidence from the health and retirement study. Demography, 50(2), 569–589.
Hurd, M. D., & McGarry, K. (1995). Evaluation of the subjective probabilities of survival in the health and retirement study. Journal of Human Resources, 30, S268–S292.
Perozek, M. (2008). Using subjective expectations to forecast longevity: Do survey respondents know something we don’t know? Demography, 45(1), 95–113.
Van Doorn, C., & Kasl, S. V. (1998). Can parental longevity and self-rated life expectancy predict mortality among older persons? Results from an Australian cohort. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 53(1), S28–S34.
Lee, S., & Smith, J. (2016). Methodological aspects of subjective life expectancy: Effects of culture-specific reporting heterogeneity among older adults in the U.S. The Journals of Gerontology: Series B Psychological Sciences and Social Sciences, 71(3), 558–568.
Smith, T. W. (1982). Conditional order effect. GSS technical report. no. 33. Chicago, IL: NORC.
Strack, F., Martin, L. L., & Schwarz, N. (1988). Priming and communication: Social determinants of information use in judgments of life satisfaction. European Journal of Social Psychology, 18, 429–442.
Schwarz, N., Strack, F., & Mai, H. P. (1991). Assimilation and contrast effects in part–whole question sequences: A conversational logic analysis. Public Opinion Quarterly, 55(1), 3–23.
Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics 3: Speech acts (pp. 41–58). New York, NY: Academic Press.
Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good. Review of General Psychology, 5(4), 323–370.
Griffin, B., Loh, V., & Hesketh, B. (2013). A mental model of factors associated with subjective life expectancy. Social Science and Medicine, 82, 79–86.
Pyszczynski, T., Greenberg, J., & Solomon, S. (1999). A dual-process model of defense against conscious and unconscious death-related thoughts: An extension of terror management theory. Psychological Review, 106(4), 835.
Jonas, E., Schimel, J., Greenberg, J., & Pyszczynski, T. (2002). The Scrooge effect: Evidence that mortality salience increases prosocial attitudes and behavior. Personality and Social Psychology Bulletin, 28(10), 1342–1353.
Renkema, L. J., Stapel, D. A., & Van Yperen, N. W. (2008). Go with the flow: Conforming to others in the face of existential threat. European Journal of Social Psychology, 38(4), 747–756.
Han, S., Qin, J., & Ma, Y. (2010). Neurocognitive processes of linguistic cues related to death. Neuropsychologia, 48(12), 3436–3442.
Klackl, J., Jonas, E., & Kronbichler, M. (2013). Existential neuroscience: Neurophysiological correlates of proximal defenses against death-related thoughts. Social Cognitive and Affective Neuroscience, 8(3), 333–340.
Luo, S., Shi, Z., Yang, X., Wang, X., & Han, S. (2014). Reminders of mortality decrease midcingulate activity in response to others’ suffering. Social Cognitive and Affective Neuroscience, 9(4), 477–486.
Mathews, P., & Sear, R. (2008). Life after death: An investigation into how mortality perceptions influence fertility preferences using evidence from an internet-based experiment. Journal of Evolutionary Psychology, 6(3), 155–172.
Murphy, J., Link, M. W., Childs, J. H., Tesfaye, C. L., Dean, E., Stern, M., & Schober, M. F. (2014). Social media in public opinion research: Report of the AAPOR task force on emerging technologies in public opinion research. American Association for Public Opinion Research. Retrieved from https://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/AAPOR_Social_Media_Report_FNL.pdf.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.
Antoun, C., Zhang, C., Conrad, F. G., & Schober, M. F. (2015). Comparisons of online recruitment strategies for convenience samples: Craigslist, Google AdWords, Facebook, and Amazon Mechanical Turk. Field Methods,. doi:10.1177/1525822X15603149.
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.
Horton, J. J., & Chilton, L. B. (2010). The labor economics of paid crowdsourcing. In Proceedings of the 11th ACM Conference on Electronic Commerce (pp. 209–218). Association for Computing Machinery. Retrieved from http://arxiv.org/pdf/1001.0627.pdf.
Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23.
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1(2), 213–220.
Groves, R. M., Fultz, N. H., & Martin, E. (1992). Direct questioning about comprehension in a survey setting. Questions about questions: Inquiries into the cognitive bases of surveys. New York, NY: Russell Sage, 49–61.
Behr, D., Kaczmirek, L., Bandilla, W., & Braun, M. (2012). Asking probing questions in web surveys: Which factors have an impact on the quality of responses? Social Science Computer Review, 30(4), 487–498.
Bassili, J. N., & Scott, B. S. (1996). Response latency as a signal to question problems in survey research. Public Opinion Quarterly, 60(3), 390–399.
Tourangeau, R., Couper, M. P., & Conrad, F. G. (2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68, 368–393.
Malhotra, N. (2008). Completion time and response order effects in web surveys. Public Opinion Quarterly, 72(5), 914–934.
Yan, T., & Tourangeau, R. (2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22(1), 51–68.
Shannon, C. E. (1948). A mathematical theory of communication. The Bell Systems Technical Journal, 27, 379–423 and 623–656.
Fisher, R. A. (1921). On the probable error of a coefficient of correlation deduced from a small sample. Metron, 1, 3–32.
Silver, N. C., & Dunlap, W. P. (1987). Averaging correlation coefficients: Should Fisher’s z transformation be used? Journal of Applied Psychology, 72(1), 146.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Funding
The data used in this study were collected with funding from the Regula Herzog Young Investigator Fund awarded to Sunghee Lee.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This study protocol was approved by the Institutional Review Board at the University of Michigan.
Appendices
Appendix 1. Description of experimental design
Main experiments
Experiment 1
In the original study, there were six conditions as in Table 4. While there is no clear guidance, some variations of the question order as in Condition 1 appear to be practiced (e.g., Health and Retirement Study). Hence, we randomly assigned 2/7 of the sample to Condition 1 and 1/7 to the remaining conditions. For this study, Condition 2 is not applicable as it was implemented to study the effect of response scales on measuring self-rated health, hence, removed from the analysis. Conditions 3, 4, and 5 collectively asked life satisfaction immediately after self-rated health. These three conditions produced similar correlation coefficients between self-rated health and life satisfaction and, hence, were grouped into one. Condition 6 asked self-rated health immediately after life satisfaction and was retained as a separate condition in the analysis.
Experiment 2
Subjective life expectancy was asked in two different questions in the original study: one using percent chance, and the other using actual years in age (See Appendix 2 for details of the wording). As the one with percent chance is used more popularly, we assign 2/3 of the sample to the percent chance wording (conditions 1 and 2 in Table 4) and the remainder to the actual years wording (conditions 3 and 4). Within each subjective life expectancy wording, we randomly split the sample half and asked subjective life expectancy either before parental mortality questions (conditions 1 and 3) or after them (conditions 2 and 4). Due to the differences in the question type, we dropped conditions 3 and 4 from the analysis.
Ancillary experiments
The following experiments were implemented in the survey and not included in this study as they address aspects other than question contexts.
Experiment 3: response scale for self-rated health
Condition 2 in Table 4.I was implemented to be compared to Condition 1 only on the distribution of self-rated health.
Experiment 4: types of anchoring vignettes for self-rated health
Different types of vignettes questions were asked in order to examine whether vignette types affect the adjustment of self-rated health.
Experiment 5: question types of subjective life expectancy
Although labeled as “subjective life expectancy” in the literature, some use actual age and others use percent chance. This experiment was included to examine the difference in the question type of subjective life expectancy.
Appendix 2. Question wording
Self-rated health with a 5-point response scale
Would you say your health is excellent, very good, good, fair, or poor in general?
□ Excellent
□ Very good
□ Good
□ Fair
□ Poor
Self-rated health with a 100-point response scale
On a scale from 0 to 100, where 0 is worst possible health and 100 is perfect possible health, how would you rate your health in general? Please provide the number (from 0 to 100).
________
Life satisfaction
Please think about your life and situation right now. How satisfied are you with your life as a whole these days?
□ Completely satisfied
□ Very satisfied
□ Somewhat satisfied
□ Not very satisfied
□ Not at all satisfied
Vignette introduction
We are interested in how you would rate the health of other people your age. Now we are going to describe the health of some people your age and then ask you to rate their health in the way you would to rate your own health.
Physical health vignettes
[Randomize: Mary/Steve] has no problem with walking, running, or moving his/her limbs. [She/He] jogs 3 miles twice a week. The last time [she/he] spent time in bed due to illness was more than a year ago. In general, would you say [Mary’s/Steve’s] health is excellent, very good, good, fair, or poor?
□ Excellent
□ Very good
□ Good
□ Fair
□ Poor
[Kate/Mike] has no problem walking 1 mile. Occasionally, [she/he] feels fatigued and has some trouble bending and lifting, but his/her occasional pain does not affect his/her daily activities. [She/He] spent a couple of days in bed due to illness last year. In general, would you say [Kate’s/Mike’s] health is excellent, very good, good, fair, or poor?
[Nancy/John] has no problem walking a couple of blocks or climbing a couple of flights of stairs. However, [she/he] feels tired after walking or climbing stairs. He/she has a great deal of trouble with daily activities. Each week, [she/he] experiences pain that limit some of his/her daily activities. [She/He] spent a week in bed due to illness last year. In general, would you say [Nancy’s/John’s] health is excellent, very good, good, fair, or poor?
Mental health vignettes
[Jennifer/Peter] enjoys his/her work and social activities and is generally satisfied with [her/his] life. [She/He] gets depressed rarely and loses interest in what [she/he] usually enjoys. In general, would you say [Jennifer’s/Peter’s] health is excellent, very good, good, fair, or poor?
[Kelly/David] enjoys social activities, but does not like [her/his] work. Although [she/he] is content with [her/his] life, [she/he] gets depressed once a month or so but is able to carry on with [her/his] day-to-day activities. In general, would you say [Kelly’s/David’s] health is excellent, very good, good, fair, or poor?
[Jane/Jack] feels depressed most of the time. [She/He] weeps frequently and feels hopeless about the future. [She/He] feels that [she/he] has become a burden on others and that [she/he] would be better dead. In general, would you say [Jane’s/Jack’s] health is excellent, very good, good, fair, or poor?
Combined physical and mental health vignettes
[Jennifer/Peter] enjoys [her/his] work and social activities and is generally satisfied with [her/his] life. [She/He] has no problem with walking, running, or moving [her/his] limbs and jogs 3 miles twice a week. In general, would you say [Jennifer’s/Peter’s] health is excellent, very good, good, fair, or poor?
[Kelly/David] feels fatigued occasionally and has some trouble bending and lifting, but [her/his] occasional pain does not affect [her/his] daily activities. Although [she/he] is content with [her/his] life, [she/he] gets depressed once a month or so. In general, would you say [Kelly’s/David’s] health is excellent, very good, good, fair, or poor?
[Jane/Jack] feels depressed most of the time. Although [she/he] has no problem walking a couple of blocks or climbing a couple of flights of stairs, [she/he] feels tired after walking or climbing stairs. In general, would you say [Jane’s/Jack’s] health is excellent, very good, good, fair, or poor?
Chronic conditions
Next, we’d like to learn more about you. Has a doctor ever told you that you have diabetes or high blood sugar?
□ Yes
□ No
Has a doctor ever told you that you have asthma?
□ Yes
□ No
Has a doctor ever told you that you have high blood pressure or hypertension?
□ Yes
□ No
Subjective life expectancy with percent chance response
The next few questions ask about your views of the chances that various events will happen. Your answers can range from zero to one hundred, where zero means there is absolutely no chance, and one hundred means that it is absolutely certain. For example, when weather forecasters report the chance of rain, a number like 20 % means “a small chance,” a number around 50 % means “a pretty even chance,” and a number like 80 % means “a very good chance.”
What is the percent chance that you will live to be [75 or more (IF AGE IS ≤ 64)/80 or more (IF AGE IS 65–69)/
85 or more (IF AGE IS 70–74)/90 or more (IF AGE IS 75–79)/95 or more (IF AGE IS 80–84)/100 or more (IF AGE IS ≥ 85)]? If you can’t think of an answer, you can choose “Don’t know.”
________ percent chance
□ Don’t know
Subjective life expectancy with actual age response
Now, we want to ask you about your expectations that various events will happen.To what age do you expect to live? If you can’t think of an answer, you can choose “Don’t know.”
_______ years old
□ Don’t know
Probing for subjective life expectancy with percent chance response
{SHOW IF SUBJECTIVE LIFE EXPECTANCY IS ANSWERED} You said there is a [FILL ANSWER FROM SUBJECTIVE LIFE EXPECTANCY] % chance that you will live to be [FILL AGE USED IN SUBJECTIVE LIFE EXPECTANCY] or more. How did you arrive at this answer? What did you consider to say that there is a [FILL ANSWER FROM SUBJECTIVE LIFE EXPECTANCY] % chance that you will live to be [FILL AGE USED IN SUBJECTIVE LIFE EXPECTANCY] or more?
______________________________________________________________
{SHOW IF SUBJECTIVE LIFE EXPECTANCY = Don’t know} You said you don’t know the chance that you will live to be [FILL AGE USED IN SUBJECTIVE LIFE EXPECTANCY] or more. How did you arrive at this answer? What made you say “don’t know”?
_____________________________________________________________
Parental mortality
Is your mother alive now?
□ Yes
□ No
{SHOW IF MOTHER NOT ALIVE} How old was your mother when she passed away?
_________ years old
Is your father alive now?
□ Yes
□ No
{SHOW IF FATHER NOT ALIVE} How old was your father when he passed away?
_________ years old
Rights and permissions
About this article
Cite this article
Lee, S., McClain, C., Webster, N. et al. Question order sensitivity of subjective well-being measures: focus on life satisfaction, self-rated health, and subjective life expectancy in survey instruments. Qual Life Res 25, 2497–2510 (2016). https://doi.org/10.1007/s11136-016-1304-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11136-016-1304-8