The effects of response option order and question order on self-rated health
This study aims to assess the impact of response option order and question order on the distribution of responses to the self-rated health (SRH) question and the relationship between SRH and other health-related measures.
In an online panel survey, we implement a 2-by-2 between-subjects factorial experiment, manipulating the following levels of each factor: (1) order of response options (“excellent” to “poor” versus “poor” to “excellent”) and (2) order of SRH item (either preceding or following the administration of domain-specific health items). We use Chi-square difference tests, polychoric correlations, and differences in means and proportions to evaluate the effect of the experimental treatments on SRH responses and the relationship between SRH and other health measures.
Mean SRH is higher (better health) and proportion in “fair” or “poor” health lower when response options are ordered from “excellent” to “poor” and SRH is presented first compared to other experimental treatments. Presenting SRH after domain-specific health items increases its correlation with these items, particularly when response options are ordered “excellent” to “poor.” Among participants with the highest level of current health risks, SRH is worse when it is presented last versus first.
While more research on the presentation of SRH is needed across a range of surveys, we suggest that ordering response options from “poor” to “excellent” might reduce positive clustering. Given the question order effects found here, we suggest presenting SRH before domain-specific health items in order to increase inter-survey comparability, as domain-specific health items will vary across surveys.
KeywordsSelf-rated health Response option order Question order Assimilation effects Validity USA
- 7.Canfield, B., Miller, K., Beatty, P., Whitaker, K., Calvillo, A., & Wilson, B. (2003). Adult questions on the health interview survey—results of cognitive testing interviews conducted April–May 2003. Hyattsville, MD: National Center for Health Statistics, Cognitive Methods Staff., pp. 1–41.Google Scholar
- 10.Groves, R. M., Fultz, F. N., & Martin, E. (1992). Direct questioning about comprehension in a survey setting. In J. M. Tanur (Ed.), Questions about questions: Inquiries into the cognitive bases of surveys (pp. 49–61). New York: Russell Sage Foundation.Google Scholar
- 16.Sudman, S., & Bradburn, N. M. (1982). Asking questions. New Jersey: Jossey-Bass.Google Scholar
- 17.Means, B., Nigam, A., Zarrow, M., Loftus, E. F., & Donaldson, M. S. (1989). Autobiographical memory for health-related events. Cognition and Survey Research, 6, 2, DHHS (PHS) 89-1077, 6(2), 1–38.Google Scholar
- 18.Keller, S. D., & Ware, J. E. (1996). Questions and answers about SF-36 and SF-12. Medical Outcomes Trust Bulletin, 4(3), 1–13.Google Scholar
- 28.National Institute on Alcohol, A., & Alcoholism. (2007). Helping patients who drink too much. Bethesda, MD: US Department of Health and Human Services.Google Scholar
- 29.Krosnick, J. A. (2012). Improving question design to maximize reliability and validity. Conference on the Future of Survey Research, Arlington, VA.Google Scholar
- 31.Preacher, K. J. (2002). Calculation for the test of the difference between two independent correlation coefficients [computer software]. Available from http://quantpsy.org.
- 33.Lee, S., Schwarz, N., & Goldstein, L. S. (2014). Culture-sensitive question order effects of self-rated health between older Hispanic and non-Hispanic adults in the United States. Journal of Aging and Health, 26(5), 860–883.Google Scholar