Skip to main content

Advertisement

SpringerLink
Log in

Measurement equivalence using a mixed-mode approach to administer health-related quality of life instruments

  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Purpose

To evaluate the effects of mode, order of administration, and the interaction of mode and order on health-related quality of life scales when self-administered by mixed mode (paper-mode and web-mode) for measurement equivalence.

Methods

Health-related quality of life data was analyzed from the Cancer of the Prostate Strategic Urologic Research Endeavor using the Medical Outcomes Study (MOS) Short Form-36 (SF-36) and the University of California Los Angeles Prostate Cancer Index (UCLA-PCI). A randomized crossover design assigned participants to two groups with a preferred 2–5-day washout period. Cognitive debriefing evaluated participants’ mode preference.

Results

Of the 245 men enrolled, 85 % completed both modes. The majority were White (97 %), college educated (66 %), reported an annual income >$75,000 (46 %), and a median age of 69 years. Intraclass correlation coefficients were high for each item on both instruments (r = .54–.97). Exact percentage agreement for yes/no items was high (≥.88). For the SF-36, significant differences were observed for order of administration (physical component and physical function scores) and for the interaction between mode and order (mental component, role emotional, social function, vitality, and mental health scores). For the UCLA-PCI, the largest difference was 12.8 points lower for sexual bother for order of administration by web-mode first (p = .03). Seventy percent preferred the web-mode, 21 % had no preference, and 9 % preferred the paper-mode.

Conclusion

Web-mode and paper-mode administrations of the SF-36 and UCLA-PCI are equivalent in men with prostate cancer, implying that mixed-mode survey administration is warranted.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Abbreviations

CaPSURE:

Cancer of the Prostate Strategic Urologic Research Endeavor

HRQOL:

Health-related quality of life

ME:

Measurement equivalence

PRO:

Patient-reported outcome

SAQ:

Self-administered questionnaire

SF-36:

Short Form-36

UCLA-PCI:

University of California Los Angeles Prostate Cancer Index

References

  1. Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report. Value in Health, 12(4), 419–429. doi:10.1111/j.1524-4733.2008.00470.x.

    Article  PubMed  Google Scholar 

  2. Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27(3), 281–291. doi:10.1093/pubmed/fdi031.

    Article  PubMed  Google Scholar 

  3. deLeeuw, E., Hox, J., & Dillman, D. (2008). Mixed-mode surveys: When and why. In E. deLeeuw, J. Hox, & D. Dillman (Eds.), International handbook of survey methodology (pp. 299–316). New York: Psychology Press Taylor and Francis Group.

    Google Scholar 

  4. deLeeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(2), 233–255.

    Google Scholar 

  5. Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value in Health, 11(2), 322–333. doi:10.1111/j.1524-4733.2007.00231.x.

    Article  PubMed  Google Scholar 

  6. Arpinelli, F., & Bamfi, F. (2006). The FDA guidance for industry on PROs: The point of view of a pharmaceutical company. Health and Quality of Life Outcomes, 4, 85. doi:10.1186/1477-7525-4-85.

    Article  PubMed Central  PubMed  Google Scholar 

  7. FDA. (2006). Guidance for industry patient-reported outcome measures: use in medical product development to support labeling claims. (pp. i–32). Rockville, MD: U.S. Department of Health and Human Services, Food and Drug Administration.

  8. Zickuhr, K. (2010). Generations 2010. http://www.pewinternet.org/~/media//Files/Reports/2010/PIP_Generations_and_Tech10.pdf. Accessed September 19, 2012.

  9. Zickuhr, K., & Smith, A. (2012). Digital differences. http://www.pewinternet.org/~/media//Files/Reports/2012/PIP_Digital_differences_041312.pdf. Accessed September 19, 2012.

  10. Couper, M. P., Kapteyn, A., Schonlau, M., & Winter, J. (2007). Noncoverage and nonresponse in an internet survey. Social Science Research, 36, 131–148.

    Article  Google Scholar 

  11. Abernethy, A. P., Herndon, J. E, 2nd, Wheeler, J. L., Patwardhan, M., Shaw, H., Lyerly, H. K., et al. (2008). Improving health care efficiency and quality using tablet personal computers to collect research-quality, patient-reported data. Health Services Research, 43(6), 1975–1991. doi:10.1111/j.1475-6773.2008.00887.x.

    Article  PubMed Central  PubMed  Google Scholar 

  12. Dupont, A., Wheeler, J., Herndon, J. E, 2nd, Coan, A., Zafar, S. Y., Hood, L., et al. (2009). Use of tablet personal computers for sensitive patient-reported information. The Journal of Supportive Oncology, 7(3), 91–97.

    PubMed  Google Scholar 

  13. Velikova, G., Wright, E. P., Smith, A. B., Cull, A., Gould, A., Forman, D., et al. (1999). Automated collection of quality-of-life data: A comparison of paper and computer touch-screen questionnaires. Journal of Clinical Oncology, 17(3), 998–1007.

    PubMed  CAS  Google Scholar 

  14. Basch, E., Artz, D., Iasonos, A., Speakman, J., Shannon, K., Lin, K., et al. (2007). Evaluation of an online platform for cancer patient self-reporting of chemotherapy toxicities. Journal of the American Medical Informatics Association, 14(3), 264–268. doi:10.1197/jamia.M2177.

    Article  PubMed Central  PubMed  Google Scholar 

  15. Farnell, D. J., Routledge, J., Hannon, R., Logue, J. P., Cowan, R. A., Wylie, J. P., et al. (2010). Efficacy of data capture for patient-reported toxicity following radiotherapy for prostate or cervical cancer. European Journal of Cancer, 46(3), 534–540. doi:10.1016/j.ejca.2009.11.017.

    Article  PubMed  Google Scholar 

  16. Dillman, A., Gertseva, A., & Mahon-Haft, T. (2005). Achieving usability in establishment surveys through the application of visual design principles. Journal of Official Statistics, 21(2), 183–214.

    Google Scholar 

  17. Dillman, D., Smyth, J., & Christian, L. (2009). Internet, mail and mixed-mode surveys: The tailored design method (3rd ed.). Hoboken, NJ: Wiley.

    Google Scholar 

  18. Dillman, D. A., & Smyth, J. D. (2007). Design effects in the transition to web-based surveys. American Journal of Preventive Medicine, 32(5 Suppl), S90–S96. doi:10.1016/j.amepre.2007.03.008.

    Article  PubMed  Google Scholar 

  19. Kwak, N., & Radler, B. (2002). A comparison between mail and web surveys: Response pattern, respondent profile and data quality. Journal of Official Statistics, 18(2), 257–273.

    Google Scholar 

  20. Schulenberg, S. E., & Yutrzenka, B. A. (1999). The equivalence of computerized and paper-and-pencil psychological instruments: Implications for measures of negative affect. Behavior Research Methods, Instruments, & Computers, 31(2), 315–321.

    Article  CAS  Google Scholar 

  21. Tseng, H. M., Tiplady, B., Macleod, H. A., & Wright, P. (1998). Computer anxiety: A comparison of pen-based personal digital assistants, conventional computer and paper assessment of mood and performance. British Journal of Psychology, 89(Pt 4), 599–610.

    Article  PubMed  Google Scholar 

  22. Urologic Diseases in America (2012). In Litwin MS, Saigal CS, & et.al. (Eds.), Chapter 3 Prostate Cancer, Tables 3–10 & 3–11 (Vol., pp. 73–96, NIH Publication No. 12-7865 Vol. June 4, 2012). Washington, DC: US Department of Health and Human Services, Public Health Service, National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases.

  23. Siegel, R., Naishadham, D., & Jemal, A. (2012). Cancer statistics, 2012. CA: A Cancer Journal for Clinicians, 62(1), 10–29. doi:10.3322/caac.20138.

    Google Scholar 

  24. Lipscomb, J., Gotay, C., & Snyder, C. (2005). Introduction to outcomes assessment in cancer. In J. Lipscomb, C. Gotay, & C. Snyder (Eds.), Outcome assessment in cancer: Measures, methods and applications (pp. 1–13). New York: Cambridge University Press.

    Google Scholar 

  25. Lipscomb, J., Snyder, C., & Gotay, C. (2007). Cancer outcomes measurement: Through the lens of the medical outcomes trust framework. Quality of Life Research, 16(1), 143–164. doi:10.1007/s11136-006-9116-x.

    Article  PubMed  Google Scholar 

  26. Hays, R. D., Sherbourne, C. D., & Mazel, R. M. (1993). The RAND 36-Item Health Survey 1.0. Health Economics, 2(3), 217–227.

    Article  PubMed  CAS  Google Scholar 

  27. McHorney, C. A., Ware, J. E, Jr, Lu, J. F., & Sherbourne, C. D. (1994). The MOS 36-item Short-Form Health Survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patient groups. Medical Care, 32(1), 40–66.

    Article  PubMed  CAS  Google Scholar 

  28. McHorney, C. A., Ware, J. E, Jr, & Raczek, A. E. (1993). The MOS 36-Item Short-Form Health Survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Medical Care, 31(3), 247–263.

    Article  PubMed  CAS  Google Scholar 

  29. Stewart, A. L., Hays, R. D., & Ware, J. E, Jr. (1988). The MOS short-form general health survey. Reliability and validity in a patient population. Medical Care, 26(7), 724–735.

    Article  PubMed  CAS  Google Scholar 

  30. Ware, J. E, Jr, Kosinski, M., & Gandek, B. (2004). SF-36 ® health survey: Manual & interpretation guide. Lincoln, RI: QualityMetric Incorporated.

    Google Scholar 

  31. Ware, J. E, Jr, & Sherbourne, C. D. (1992). The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Medical Care, 30(6), 473–483.

    Article  PubMed  Google Scholar 

  32. Litwin, M. S., Hays, R. D., Fink, A., Ganz, P. A., Leake, B., & Brook, R. H. (1998). The UCLA Prostate Cancer Index: Development, reliability, and validity of a health-related quality of life measure. Medical Care, 36(7), 1002–1012.

    Article  PubMed  CAS  Google Scholar 

  33. Litwin, M. S., Hays, R. D., Fink, A., Ganz, P. A., Leake, B., Leach, G. E., et al. (1995). Quality-of-life outcomes in men treated for localized prostate cancer. JAMA, the Journal of the American Medical Association, 273(2), 129–135.

    Article  CAS  Google Scholar 

  34. Lubeck, D. P., Litwin, M. S., Henning, J. M., Stier, D. M., Mazonson, P., Fisk, R., et al. (1996). The CaPSURE database: A methodology for clinical practice and research in prostate cancer. Urology, 48(5), 773–777.

    Article  PubMed  CAS  Google Scholar 

  35. Rosner, B. (2006). The cross-over design. In Fundamentals of biostatistics, (6th Ed., pp. 699–707). Belmont, CA: Thomson Higher Education.

  36. Connare, V. (1996). Trebuchet MS. http://www.microsoft.com/typography/fonts/family.aspx?FID=2. Accessed March 11, 2013.

  37. Ware, J. E, Jr, & Kosinski, M. (2001). SF-36 ® physical and mental health summary scales: A manual for users of version 1.0 (2nd ed.). Lincoln: RI QualityMetric Incorporated.

    Google Scholar 

  38. Rosner, B. (2006). The intraclass correlation coefficient. In Fundamentals of biostatistics (6th ed., pp. 613–617). Belmont, CA: Thomson Higher Education.

  39. Rosner, B. (2006). The paired t test. In Fundamentals of biostatistics (6th Ed., pp. 298–302). Belmont, CA: Thomson Higher Education.

  40. Cicchetti, D. V., & Sparrow, S. A. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86(2), 127–137.

    PubMed  CAS  Google Scholar 

  41. Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: Wiley.

    Google Scholar 

  42. Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: SAGE Publications Inc.

    Google Scholar 

  43. D’Amico, A. V., Whittington, R., Malkowicz, S. B., Schultz, D., Blank, K., Broderick, G. A., et al. (1998). Biochemical outcome after radical prostatectomy, external beam radiation therapy, or interstitial radiation therapy for clinically localized prostate cancer. JAMA, the Journal of the American Medical Association, 280(11), 969–974.

    Article  Google Scholar 

  44. Cohen, J. (1988). Calculation of effect size. In 2nd (Ed.), Statistical power analysis for the behavioral sciences (pp. 24–27). Hillsdale, NJ: Lawrence Erlbaum Associates.

  45. Farivar, S. S., Liu, H., & Hays, R. D. (2004). Half standard deviation estimate of the minimally important difference in HRQOL scores. Expert Review of Pharmacoeconomics & Outcomes Research, 4(5), 515–523. doi:10.1586/14737167.4.5.515.

    Article  Google Scholar 

  46. Norman, G. R., Sloan, J. A., & Wyrwich, K. W. (2003). Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Medical Care, 41(5), 582–592. doi:10.1097/01.MLR.0000062554.74615.4C.

    PubMed  Google Scholar 

  47. Revicki, D. A., Cella, D., Hays, R. D., Sloan, J. A., Lenderking, W. R., & Aaronson, N. K. (2006). Responsiveness and minimal important differences for patient reported outcomes. Health and Quality of Life Outcomes, 4, 70. doi:10.1186/1477-7525-4-70.

    Article  PubMed Central  PubMed  Google Scholar 

  48. Sloan, J. A., Frost, M. H., Berzon, R., Dueck, A., Guyatt, G., Moinpour, C., et al. (2006). The clinical significance of quality of life assessments in oncology: A summary for clinicians. Supportive Care in Cancer, 14(10), 988–998. doi:10.1007/s00520-006-0085-y.

    Article  PubMed  Google Scholar 

  49. Bliven, B. D., Kaufman, S. E., & Spertus, J. A. (2001). Electronic collection of health-related quality of life data: Validity, time benefits, and patient preference. Quality of Life Research, 10(1), 15–22.

    Article  PubMed  CAS  Google Scholar 

  50. Ryan, J. M., Corry, J. R., Attewell, R., & Smithson, M. J. (2002). A comparison of an electronic version of the SF-36 General Health Questionnaire to the standard paper version. Quality of Life Research, 11(1), 19–26.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

The Cancer of the Prostate Strategic Urologic Research Endeavor (CaPSURE) longitudinal study is supported by an unrestricted educational grant from Abbott Laboratories (Chicago, IL), by the National Institutes of Health/National Cancer Institute (Grant No. 5RC1CA146596) and by the Agency for Healthcare Research and Quality (Grant No. 1U01CA88160). The funding sources had no role in the study design, data collection, analyses, interpretation, or writing of this manuscript. Additionally, the authors would like to acknowledge the contributions of the staff from the UCSF Urology Data Operations group. Specifically, Suzanne C. Lessard, Paige L. Marr, Eric P. Elkin, Ali Zargham, Lydia K. Moody, and Brian Lanzman for their work on the design and data collection with this study.

Conflict of interest

None.

Author information

Authors and Affiliations

  1. Department of Urology, UCSF Helen Diller Family Comprehensive Cancer Center, School of Medicine, University of California San Francisco (UCSF), 3333 California Street, Suite 282, San Francisco, CA, 94143-1355, USA

    Jeanette M. Broering, Alan Paciorek & Peter R. Carroll

  2. Departments of Clinical Pharmacy and Medicine, Schools of Pharmacy and Medicine, University of California San Francisco (UCSF), San Francisco, CA, USA

    Leslie S. Wilson

  3. Department of Urology, David Geffen School of Medicine, Los Angeles, CA, USA

    Mark S. Litwin

  4. Department of Health Services, Fielding School of Public Health, University of California Los Angeles (UCLA), Los Angeles, CA, USA

    Mark S. Litwin

  5. Department of Physiological Nursing, School of Nursing, University of California San Francisco (UCSF), San Francisco, CA, USA

    Christine Miaskowski

Authors
  1. Jeanette M. Broering
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alan Paciorek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter R. Carroll
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leslie S. Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mark S. Litwin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christine Miaskowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeanette M. Broering.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Broering, J.M., Paciorek, A., Carroll, P.R. et al. Measurement equivalence using a mixed-mode approach to administer health-related quality of life instruments. Qual Life Res 23, 495–508 (2014). https://doi.org/10.1007/s11136-013-0493-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-013-0493-7

Keywords

Search

Navigation