Skip to main content
Log in

The effect of response option order on self-rated health: a replication study

  • Brief Communication
  • Published:
Quality of Life Research Aims and scope Submit manuscript



Following calls for replication of research studies, this study documents the results of two studies that experimentally examine the impact of response option order on self-rated health (SRH).


Two studies from an online panel survey examined how the order of response options (positive to negative versus negative to positive) influences the distribution of SRH answers.


The results of both studies indicate that the distribution of SRH varies across the experimental treatments, and mean SRH is lower (worse) when the response options start with “poor” rather than “excellent.” In addition, there are differences across the two studies in the distribution of SRH and mean SRH when the response options begin with “excellent,” but not when the response options begin with “poor.”


The similarities in the general findings across the two studies strengthen the claim that SRH will be lower (worse) when the response options are ordered beginning with “poor” rather than “excellent” in online self-administered questionnaires, with implications for the validity of SRH. The slight differences in the administration of the seemingly identical studies further strengthen the claim and also serve as a reminder of the inherent variability of a single permutation of any given study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.


  1. BMI in this study is calculated from self-reported height and weight, and is thus subject to measurement error.


  1. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

  2. Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137–141.

    Article  Google Scholar 

  3. Idler, E. L., & Benyamini, Y. (1997). Self-rated health and mortality: A review of twenty-seven community studies. Journal of Health and Social Behavior, 38(1), 21–37.

    Article  CAS  PubMed  Google Scholar 

  4. Carp, F. M. (1974). Position effects on interview responses. Journal of Gerontology, 29(5), 581–587.

    Article  CAS  PubMed  Google Scholar 

  5. Chan, J. C. (1991). Response-order effects in Likert-type scales. Educational and Psychological Measurement, 51(3), 531–540.

    Article  Google Scholar 

  6. Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236.

    Article  Google Scholar 

  7. Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2), 201–219.

    Article  Google Scholar 

  8. Sudman, S., & Bradburn, N. M. (1982). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.

    Google Scholar 

  9. Garbarski, D., Schaeffer, N., & Dykema, J. (2015). The effects of response option order and question order on self-rated health. Quality of Life Research, 24(6), 1443–1453.

    Article  PubMed  Google Scholar 

  10. Means, B., Nigam, A., Zarrow, M., Loftus, E. F., & Donaldson, M. S. (1989). Autobiographical memory for health-related events (No. DHHS (PHS) 89-1077). Rockville, MD: National Center for Health Statistics.

  11. Callegaro, M., & DiSogra, C. (2008). Computing response metrics for online panels. Public Opinion Quarterly, 72(5), 1008–1032.

    Article  Google Scholar 

  12. Mutz, D. C. (2011). Population-based survey experiments. Princeton: Princeton University Press.

    Book  Google Scholar 

  13. Lee, S., & Schwarz, N. (2014). Question context and priming meaning of health: Effect on differences in self-rated health between Hispanics and non-Hispanic whites. American Journal of Public Health, 104(1), 179–185.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Sanchez, G. R., & Vargas, E. D. Forthcoming. Language bias and self-rated health status among the Latino population: Evidence of the influence of translation in a wording experiment. Quality of Life Research.

Download references


This research was supported in part by funding from the Eunice Kennedy Shriver National Institute of Child Health and Human Development Grants to the Center for Demography and Ecology (T32 HD007014) and the Health Disparities Research Scholars training program (T32 HD049302), and from core funding to the Center for Demography and Ecology (R24 HD047873) at the University of Wisconsin–Madison. The data used in this study were collected by GfK with funding from Time-sharing Experiments for the Social Sciences (NSF Grant SES-0818839, Jeremy Freese and James Druckman, Principal Investigators) and RTI International as part of their 2012 Research Challenge. This study was approved by the Social and Behavioral Sciences Institutional Review Board at the University of Wisconsin–Madison. Opinions expressed here are those of the authors and do not necessarily reflect those of the sponsors or related organizations.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Dana Garbarski.

Ethics declarations

Conflict of interest

Authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garbarski, D., Schaeffer, N.C. & Dykema, J. The effect of response option order on self-rated health: a replication study. Qual Life Res 25, 2117–2121 (2016).

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: