Demography

, Volume 52, Issue 5, pp 1703–1728 | Cite as

Promises and Pitfalls of Anchoring Vignettes in Health Survey Research

  • Hanna Grol-Prokopczyk
  • Emese Verdes-Tennant
  • Mary McEniry
  • Márton Ispány
Article

Abstract

Data harmonization is a topic of growing importance to demographers, who increasingly conduct domestic or international comparative research. Many self-reported survey items cannot be directly compared across demographic groups or countries because these groups differ in how they use subjective response categories. Anchoring vignettes, already appearing in numerous surveys worldwide, promise to overcome this problem. However, many anchoring vignettes have not been formally evaluated for adherence to the key measurement assumptions of vignette equivalence and response consistency. This article tests these assumptions in some of the most widely fielded anchoring vignettes in the world: the health vignettes in the World Health Organization (WHO) Study on Global AGEing and Adult Health (SAGE) and World Health Survey (WHS) (representing 10 countries; n = 52,388), as well as similar vignettes in the Health and Retirement Study (HRS) (n = 4,528). Findings are encouraging regarding adherence to response consistency, but reveal substantial violations of vignette equivalence both cross-nationally and across socioeconomic groups. That is, members of different sociocultural groups appear to interpret vignettes as depicting fundamentally different levels of health. The evaluated anchoring vignettes do not fulfill their promise of providing interpersonally comparable measures of health. Recommendations for improving future implementations of vignettes are discussed.

Keywords

Anchoring vignettes Survey methods Self-rated health Comparative health research Reporting heterogeneity 

Supplementary material

13524_2015_422_MOESM1_ESM.docx (28 kb)
Online Resource 1(DOCX 27.7 kb)
13524_2015_422_MOESM2_ESM.docx (131 kb)
Online Resource 2(DOCX 131 kb)
13524_2015_422_MOESM3_ESM.docx (68 kb)
Online Resource 3(DOCX 68.4 kb)
13524_2015_422_MOESM4_ESM.zip (180 kb)
Online Resource 4(ZIP 179 kb)

References

  1. Angel, R. (2013). After Babel: Language and the fundamental challenges of comparative aging research. Journal of Cross-Cultural Gerontology, 28, 223–238.CrossRefGoogle Scholar
  2. Au, N., & Lorgelly, P. K. (2014). Anchoring vignettes for health comparisons: An analysis of response consistency. Quality of Life Research, 23, 1721–1731.CrossRefGoogle Scholar
  3. Bago D’Uva, T., Lindeboom, M., O’Donnell, O., & van Doorslaer, E. (2011a). Education-related inequity in healthcare with heterogeneous reporting of health. Journal of the Royal Statistical Society: Series A, 174, 639–664.CrossRefGoogle Scholar
  4. Bago D’Uva, T., Lindeboom, M., O’Donnell, O., & van Doorslaer, E. (2011b). Slipping anchor? Testing the vignettes approach to identification and correction of reporting heterogeneity. Journal of Human Resources, 46, 875–906.CrossRefGoogle Scholar
  5. Biss, E. (2005). The pain scale. Seneca Review, 35(1), 5–25.Google Scholar
  6. Burgard, S. A., & Chen, P. V. (2014). Challenges of health measurement in studies of health disparities. Social Science & Medicine, 106, 143–150.CrossRefGoogle Scholar
  7. Corrado, L., & Weeks, M. (2010). Identification strategies in survey response using vignettes (Cambridge Working Papers in Economics No. 1031). Cambridge, UK: Cambridge University. Retrieved from http://ideas.repec.org/p/cam/camdae/1031.html
  8. Dong, H., Campbell, C., Kurosu, S., Yang, W., & Lee, J. Z. (2015). New sources for comparative social science: Historical population panel data from East Asia. Demography, 52, 1061–1088.CrossRefGoogle Scholar
  9. Dowd, J. B., & Zajacova, A. (2007). Does the predictive power of self-rated health for subsequent mortality risk vary by socioeconomic status in the US? International Journal of Epidemiology, 36, 1214–1221.CrossRefGoogle Scholar
  10. Grol-Prokopczyk, H. (2014). Age and sex effects in anchoring vignette studies: Methodological and empirical contributions. Survey Research Methods, 8, 1–17.Google Scholar
  11. Grol-Prokopczyk, H., Freese, J., & Hauser, R. M. (2011). Using anchoring vignettes to assess group differences in self-rated health. Journal of Health and Social Behavior, 52, 246–261.CrossRefGoogle Scholar
  12. Hanna, L. C., Hunt, S. M., & Bhopal, R. S. (2012). Using the Rose Angina Questionnaire cross-culturally: The importance of consulting lay people when translating epidemiological questionnaires. Ethnicity & Health, 17, 241–251.CrossRefGoogle Scholar
  13. Hopkins, D. J., & King, G. (2010). Improving anchoring vignettes: Designing surveys to correct interpersonal incomparability. Public Opinion Quarterly, 74, 201–222.CrossRefGoogle Scholar
  14. Hunt, S. M., & Bhopal, R. (2004). Self report in clinical and epidemiological studies with non-English speakers: The challenge of language and culture. Journal of Epidemiology and Community Health, 58, 618–622.CrossRefGoogle Scholar
  15. Iburg, K. M., Salomon, J. A., Tandon, A., & Murray, C. J. L. (2002). Cross-population comparability of physician-assessed and self-reported measures of health. In C. J. L. Murray, J. A. Salomon, C. D. Mathers, & A. D. Lopez (Eds.), Summary measures of population health: Concepts, ethics, measurement and applications (pp. 433–448). Geneva, Switzerland: World Health Organization.Google Scholar
  16. Inglehart, R., & Welzel, C. (2005). Modernization, cultural change and democracy. New York, NY: Cambridge University Press.Google Scholar
  17. Jürges, H. (2007). True health vs response styles: Exploring cross-country differences in self-reported health. Health Economics, 16, 163–178.CrossRefGoogle Scholar
  18. Jylhä, M., Guralnik, J. M., Ferrucci, L., Jokela, J., & Heikkinen, E. (1998). Is self-rated health comparable across cultures and genders? Journals of Gerontology, Series B: Psychological Sciences and Social Sciences, 53, S144–S152.CrossRefGoogle Scholar
  19. Kapteyn, A. (2010). What can we learn from (and about) global aging? Demography, 47(Suppl.), S191–S209.CrossRefGoogle Scholar
  20. King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of survey research. American Political Science Review, 98, 191–207.CrossRefGoogle Scholar
  21. King, G., & Wand, J. (2007). Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis, 15, 46–66.CrossRefGoogle Scholar
  22. Kowal, P., Chatterji, S., Naidoo, N., Biritwum, R., Fan, W., Lopez Ridaura, R., . . . Boerma, J. T. (2012). Data resource profile: The World Health Organization Study on Global AGEing and Adult Health (SAGE). International Journal of Epidemiology, 41, 1639–1649.Google Scholar
  23. Kristensen, N., & Johansson, E. (2008). New evidence on cross-country differences in job satisfaction using anchoring vignettes. Labour Economics, 15, 96–117.CrossRefGoogle Scholar
  24. Menec, V. H., Shooshtari, S., & Lambert, P. (2007). Ethnic differences in self-rated health among older adults: A cross-sectional and longitudinal analysis. Journal of Aging and Health, 19, 62–86.CrossRefGoogle Scholar
  25. Murray, C. J. L., Özaltin, E., Tandon, A., Salomon, J. A., Sadana, R., & Chatterji, S. (2003). Empirical evaluation of the anchoring vignette approach in health surveys. In C. J. L. Murray & D. B. Evans (Eds.), Health systems performance assessment: Debates, methods and empiricism (pp. 369–399). Geneva, Switzerland: World Health Organization.Google Scholar
  26. Murray, C. J. L., Tandon, A., Salomon, J. A., Mathers, C. D., & Sadana, R. (2002). New approaches to enhance cross-population comparability of survey results. In C. J. L. Murray, J. A. Salomon, C. D. Mathers, & A. D. Lopez (Eds.), Summary measures of population health: Concepts, ethics, measurement and applications (pp. 421–431). Geneva, Switzerland: World Health Organization.Google Scholar
  27. National Institute on Aging (NIA). (2012). Harmonization strategies for behavioral, social science, and genetic research (Workshop Summary Report). Retrieved from https://www.nia.nih.gov/sites/default/files/nia_bssg_harmonization_summary_version_2-5-20122.pdf
  28. Pan, Y., & Fond, M. (2014). Evaluating multilingual questionnaires: A sociolinguistic perspective. Survey Research Methods, 8, 181–194.Google Scholar
  29. Pasick, R. J., Stewart, S. L., Bird, J. A., & D’Onofrio, C. N. (2001). Quality of data in multiethnic health surveys. Public Health Reports, 116(Suppl. 1), 223–243.CrossRefGoogle Scholar
  30. Rabe-Hesketh, S., & Skrondal, A. (2002). Estimating chopit models in gllamm: Political efficacy example from King et al. Retrieved from http://www.gllamm.org/chopit.pdf
  31. Rice, N., Robone, S., & Smith, P. (2011). Analysis of the validity of the vignette approach to correct for heterogeneity in reporting health system responsiveness. European Journal of Health Economics: HEPAC: Health economics in prevention and care, 12, 141–162.CrossRefGoogle Scholar
  32. Ruggles, S. (2014). Big microdata for population research. Demography, 51, 287–297.CrossRefGoogle Scholar
  33. Sadana, R., Mathers, C. D., Lopez, A. D., Murray, C. J. L., & Moesgaard Iburg, K. (2002). Comparative analyses of more than 50 household surveys on health status. In C. J. L. Murray, J. A. Salomon, C. D. Mathers, & A. D. Lopez (Eds.), Summary measures of population health: Concepts, ethics, measurement and applications (pp. 369–386). Geneva, Switzerland: World Health Organization.Google Scholar
  34. Schenker, N., Raghunathan, T. E., & Bondarenko, I. (2010). Improving on analyses of self-reported data in a large-scale health survey by using information from an examination-based survey. Statistics in Medicine, 29, 533–545.Google Scholar
  35. Schiavenato, M., & Craig, K. D. (2010). Pain assessment as a social transaction: Beyond the “gold standard.” Clinical Journal of Pain, 26, 667–676.Google Scholar
  36. Sen, A. (2002). Health: Perception versus observation. BMJ, 324, 860–861.CrossRefGoogle Scholar
  37. Shetterly, S. M., Baxter, J., Mason, L. D., & Hamman, R. F. (1996). Self-rated health among Hispanic vs non-Hispanic white adults: The San Luis Valley Health and Aging Study. American Journal of Public Health, 86, 1798–1801.CrossRefGoogle Scholar
  38. Skevington, S. M. (2002). Advancing cross-cultural research on quality of life: Observations drawn from the WHOQOL development. Quality of Life Research, 11, 135–144.CrossRefGoogle Scholar
  39. Smith, T. W. (2003). Developing comparable questions in cross-national surveys. In J. A. Harkness, F. J. R. van der Vijver, & P. P. Mohler (Eds.), Cross-cultural survey methods (pp. 69–91). Hoboken, NJ: John Wiley & Sons.Google Scholar
  40. Tandon, A., Murray, C. J. L., Salomon, J. A., & King, G. (2003). Statistical models for enhancing cross-population comparability. In C. J. L. Murray & D. B. Evans (Eds.), Health systems performance assessment: Debates, methods and empiricism (pp. 727–741). Geneva, Switzerland: World Health Organization.Google Scholar
  41. United Nations Development Programme (UNDP). (2008). Human development report 2007/2008. Retrieved from http://hdr.undp.org/en/reports/global/hdr2007-2008/
  42. van Soest, A., Delaney, L., Harmon, C., Kapteyn, A., & Smith, J. P. (2011). Validating the use of anchoring vignettes for the correction of response scale differences in subjective questions. Journal of the Royal Statistical Society: Series A, 174, 575–595.Google Scholar
  43. van Soest, A., & Vonkova, H. (2014). Testing the specification of parametric models by using anchoring vignettes. Journal of the Royal Statistical Society: Series A, 177, 115–133.Google Scholar
  44. Zimmer, Z., Natividad, J., Lin, H.-S., & Chayovan, N. (2000). A cross-national examination of the determinants of self-assessed health. Journal of Health and Social Behavior, 41, 465–481.CrossRefGoogle Scholar

Copyright information

© Population Association of America 2015

Authors and Affiliations

  • Hanna Grol-Prokopczyk
    • 1
  • Emese Verdes-Tennant
    • 2
  • Mary McEniry
    • 3
  • Márton Ispány
    • 4
  1. 1.Department of SociologyUniversity at Buffalo, State University of New YorkBuffaloUSA
  2. 2.World Health OrganizationGenevaSwitzerland
  3. 3.Center for Demography & EcologyUniversity of WisconsinMadisonUSA
  4. 4.Faculty of InformaticsUniversity of DebrecenDebrecenHungary

Personalised recommendations