Abstract
Comparison standards that people use when responding to survey questions, also called Frames of Reference (FoRs), can influence the validity of self-report responses. The effects of FoRs might be the strongest for items using vague quantifier (VQ) scales, which are particularly prominent in quality of life research, compared with numeric responses. This study aims to investigate the impact of FoRs on self-report measures by examining how imposing a specific FoR in survey questions affects (a) the response levels of VQ and numeric scales and (b) the relationship between VQs and a quantitative responses to the same question. A sample of 1869 respondents rated their education, commute and sleep duration, medication use, and level of physical activity using both VQ and numeric formats. Participants were asked to compare themselves with the average US adult, with their friends who are about their age, or did not receive specific instructions regarding a reference for comparison. We found that FoR conditions did not influence the numeric responses. Among the VQ responses, only education attainment was affected by FoR. The association between the numeric responses and vague quantifiers was comparable across different FoR conditions. Our results showed that manipulating the use of interpersonal FoRs had limited effect on the responses, which suggests that at least some comparisons do not have a strong biasing effect on self-report measures. However, future research should confirm this finding for using other FoRs (e.g., historical or hypothetical comparisons) and other outcome measures.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Data can be obtained from authors upon request.
References
Al Baghal, T. (2014). Is vague valid? The comparative predictive validity of vague quantifiers and numeric response options. Survey Research Methods, 8(3), 169–179. https://doi.org/10.18148/srm/2014.v8i3.5813
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s mechanical Turk. Political Analysis, 20(3), 351–368. https://doi.org/10.1093/pan/mpr057.
Biernat, M., Manis, M., & Nelson, T. E. (1991). Stereotypes and standards of judgment. Journal of Personality and Social Psychology., 60, 485–499. https://doi.org/10.1037/0022-3514.60.4.485.
Bradburn, N., & Miles, C. (1979). Vague quantifiers. Public Opinion Quarterly, 43(1), 92–101.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. https://doi.org/10.1177/1745691610393980.
DeWalt, D. A., Rothrock, N., Yount, S., & Stone, A. A. (2007). Evaluation of item candidates. Medical Care, 45(Suppl 1), S12–S21. https://doi.org/10.1097/01.mlr.0000254567.79743.e2.
Fayers, P. M., Langston, A. L., & Robertson, C. (2007). Implicit self-comparisons against others could bias quality of life assessments. Journal of Clinical Epidemiology, 60(10), 1034–1039. https://doi.org/10.1016/j.jclinepi.2007.03.005.
Feitosa, J., Joseph, D. L., & Newman, D. A. (2015). Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75, 47–52. https://doi.org/10.1016/j.paid.2014.11.017.
Fienberg, S. E., Loftus, E. F., & Tanur, J. M. (1985). Cognitive aspects of health survey methodology: An overview. Milbank Memorial Fund Quarterly, Health and Society, 63(3), 547–564.
Filus, A., Junghaenel, D. U., Schneider, S., Broderick, J. E., & Stone, A. A. (2020). Age effects of frames of reference in self-reports of health, well-being, fatigue and pain. Applied Research in Quality of Life, 15(1), 35–54. https://doi.org/10.1007/s11482-018-9663-7.
Junghaenel, D. U., Broderick, J. E., Schneider, S., May, M., Bolton, A., McCarrier, K. P., Stassek, L. M., Keithly, S. C., & Stone, A. A. (2018). Frames of reference in self-reports of health, well-being, fatigue, and pain: A qualitative examination. Applied Research in Quality of Life, 13(3), 585–601. https://doi.org/10.1007/s11482-017-9546-3.
Junghaenel, D. U., Schneider, S., & Stone, A. A. (2019). Do people with arthritis differ from healthy controls in their internal comparison standards for self-reports of health, fatigue, and pain? Journal of Patient-Reported Outcomes, 3(1), 21. https://doi.org/10.1186/s41687-019-0108-3.
Kaplan, G., & Baron-Epel, O. (2003). What lies behind the subjective evaluation of health status? Social Science & Medicine, 56(8), 1669–1676. https://doi.org/10.1016/S0277-9536(02)00179-X.
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017). An analysis of data quality: Professional panels, student subject pools, and Amazon’s mechanical Turk. Journal of Advertising, 46(1), 141–155. https://doi.org/10.1080/00913367.2016.1269304.
Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48(1), 61–83. https://doi.org/10.1016/j.jrp.2013.09.008.
McCredie, M. N., & Morey, L. C. (2019). Who are the Turkers? A characterization of MTurk workers using the personality assessment inventory. Assessment, 26(5), 759–766. https://doi.org/10.1177/1073191118760709.
Osborne, J. W., & Blanchard, M. R. (2011). Random responding from participants is a threat to the validity of social science research. Frontiers in Psychology, 1(JAN), 1–7. https://doi.org/10.3389/fpsyg.2010.00220.
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon mechanical Turk. Behavior Research Methods, 46(4), 1023–1031. https://doi.org/10.3758/s13428-013-0434-y.
Ramsey, S. R., Thompson, K. L., McKenzie, M., & Rosenbaum, A. (2016). Psychological research in the internet age: The quality of web-based data. Computers in Human Behavior, 58, 354–360. https://doi.org/10.1016/j.chb.2015.12.049.
Roberts, G. (1999). Age effects and health appraisal: A meta-analysis. The journals of gerontology. Series B, Psychological sciences and social sciences, 54(1), S24–S30.
Rothrock, N. E., Hays, R. D., Spritzer, K., Yount, S. E., Riley, W., & Cella, D. (2010). Relative to the general US population, chronic diseases are associated with poorer health-related quality of life as measured by the patient-reported outcomes measurement information system (PROMIS). Journal of Clinical Epidemiology, 63(11), 1195–1204. https://doi.org/10.1016/j.jclinepi.2010.04.012.
Sargent-Cox, K. A., Anstey, K. J., & Luszcz, M. A. (2008). Determinants of self-rated health items with different points of reference. Journal of Aging and Health, 20(6), 739–761. https://doi.org/10.1177/0898264308321035.
Schaeffer, N. C. (1991). Hardly ever or constantly? Group comparisons using vague quantifier. The Public Opinion Quarterly, 55(3), 395–423.
Schaeffer, N. C., & Charng, H. W. (1991). Two experiments in simplifying response categories: Intensity ratings and behavioral frequencies. Sociological Perspectives, 34(34), 165–182. https://doi.org/10.2307/1388989.
Schneider, S., & Stone, A. A. (2016). The meaning of vaguely quantified frequency response options on a quality of life scale depends on respondents’ medical status and age. Quality of Life Research, 25(10), 2511–2521. https://doi.org/10.1007/s11136-016-1293-7.
Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105. https://doi.org/10.1037//0003-066X.54.2.93.
Schwarz, N., Hippler, H.-J., Deutsch, B., & Strack, F. (1985). Response scales: Effects of category range on reported behavior and comparative judgments. Public Opinion Quarterly, 49(3), 388. https://doi.org/10.1086/268936.
Schwarz, N., & Oyserman, D. (2001). Asking questions about behavior. American Journal of Evaluation, 22(2), 127–160. https://doi.org/10.1177/109821400102200202.
Stone, A. A., Bachrach, C. A., Jobe, J. B., Kurtzmann, H. S., & Cain, V. S. (Eds.). (2000). The science of self-report: Implications for research and practice. Mahwah, NJ: Erlbaum.
Stone, A. A., Broderick, J. E., Schwartz, J. E., & Schwarz, N. (2008). Context effects in survey ratings of health, symptoms, and satisfaction. Medical Care, 46(7), 662–667. https://doi.org/10.1097/MLR.0b013e3181789387.
Stone, A. A., Walentynowicz, M., Schneider, S., Junghaenel, D. U., & Wen, C. K. (2019). MTurk participants have substantially lower evaluative subjective well-being than other survey participants. Computers in Human Behavior, 94, 1–8. https://doi.org/10.1016/j.chb.2018.12.042.
Thomas, K. A., & Clifford, S. (2017). Validity and mechanical Turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior, 77(August), 184–197. https://doi.org/10.1016/j.chb.2017.08.038.
Walentynowicz, M., Schneider, S., & Stone, A. A. (2018). The effects of time frames on self-report. PLoS One, 13(8), e0201655. https://doi.org/10.1371/journal.pone.0201655.
Wänke, M. (2002). Conversational norms and the interpretation of vague quantifiers. Applied Cognitive Psychology, 16(3), 301–307. https://doi.org/10.1002/acp.787.
Wright, D. B., Gaskell, G. D., & O’Muircheartaigh, C. A. (1994). How much is ‘quite a bit’? Mapping between numerical values and vague quantifiers. Applied Cognitive Psychology, 8(5), 479–496. https://doi.org/10.1002/acp.2350080506.
Funding
This work was supported by a grant from the National Institute on Aging, R01AG042407 (AAS).
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation and data collection were performed by Marta Walentynowicz. Analysis was performed by Marta Walentynowicz, Stefan Schneider, and Arthur Stone. The first draft of the manuscript was written by Marta Walentynowicz and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of Interest
MW, SS, and DJ have no conflict of interest to declare. AS is a senior scientist with the Gallup Organization and a consultant with Adelphi Values, Inc.
Ethics Approval
The University of Southern California Institutional Review Board approved this study.
Consent to Participate
Informed consent was obtained from all individual participants included in the study.
Consent for Publication
Not applicable.
Code Availability
Not applicable.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
ESM 1
(PDF 235 kb)
Rights and permissions
About this article
Cite this article
Walentynowicz, M., Schneider, S., Junghaenel, D.U. et al. Vague Quantifiers Demonstrate Little Susceptibility to Frame of Reference Effects. Applied Research Quality Life 17, 317–331 (2022). https://doi.org/10.1007/s11482-020-09889-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11482-020-09889-0