Skip to main content

Advertisement

SpringerLink
Log in

Mode of administration does not cause bias in patient-reported outcome results: a meta-analysis

  • Special Section: PROs in Non-Standard Settings (by invitation only)
  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Purpose

Technological advances in recent decades have led to the availability of new modes to administer patient-reported outcomes (PROs). To aid selecting optimal modes of administration (MOA), we undertook a systematic review to determine whether differences in bias (both size and direction) exist among modes.

Methods

We searched five electronic databases from 2004 (date of last comprehensive review on this topic) to April 2014, cross-referenced and searched reference lists. Studies that compared two or more MOA for a health-related PRO measure in adult samples were included. Two reviewers independently applied inclusion and quality criteria and extracted findings. Meta-analyses and meta-regressions were conducted using random-effects models.

Results

Of 5100 papers screened, 222 were considered potentially relevant and 56 met eligibility criteria. No evidence of bias was found for: (1) paper versus electronic self-complete; and (2) self-complete versus assisted MOA. Heterogeneity for paper versus electronic comparison was explained by type of construct (i.e. physical vs. psychological). Heterogeneity for self-completion versus assisted modes was in part explained by setting (clinic vs. home); the largest bias was introduced when assisted completion occurred in the clinic and follow-up was by self-completion (either electronic or paper) in the home.

Conclusions

Self-complete paper and electronic MOA can be used interchangeably for research in clinic and home settings. Self-completion and assisted completion produce equivalent scores overall, although heterogeneity may be induced by setting. These results support the use of mixed MOAs within a research study, which may be a useful strategy for reducing missing PRO data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Food and Drug Administration. (2009). Patient reported outcome measures: Use in medical product development to support labelling claims. MD: US Department of Health & Human Support Food & Drug Administration.

    Google Scholar 

  2. Hood, K., Robling, M., Ingledew, D., Gillespie, D., Greene, G., Ivins, R., et al. (2012). Mode of data elicitation, acquisition and response to surveys: A systematic review. Health Technology Assessment, 16(27), 1–162.

    Article  CAS  PubMed  Google Scholar 

  3. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.

    Article  PubMed  Google Scholar 

  4. Basch, E., Abernethy, A. P., Mullins, C. D., Reeve, B. B., Smith, M. L., Coons, S. J., et al. (2012). Recommendations for incorporating patient-reported outcomes into clinical comparative effectiveness research in adult oncology. Journal of Clinical Oncology, 30(34), 4249–4255.

    Article  PubMed  Google Scholar 

  5. Stukenborg, G. J., Blackhall, L., Harrison, J., Barclay, J. S., Dillon, P., Davis, M. A., et al. (2014). Cancer patient-reported outcomes assessment using wireless touch screen tablet computers. Quality of Life Research, 23(5), 1603–1607.

    Article  CAS  PubMed  Google Scholar 

  6. Scientific Advisory Committee of the Medical Outcomes Trust. (2002). Assessing health status and quality-of-life instruments: Attributes and review criteria. Quality of Life Research, 11(3), 193–205.

    Article  Google Scholar 

  7. Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27(3), 281–291.

    Article  PubMed  Google Scholar 

  8. Bernhard, J., Cella, D. F., Coates, A. S., Fallowfield, L., Ganz, P. A., Moinpour, C. M., et al. (1998). Missing quality of life data in cancer clinical trials: Serious problems and challenges. Statistics in Medicine, 17(5–7), 517–532.

    Article  CAS  PubMed  Google Scholar 

  9. Smith, A. B., King, M., Butow, P., & Olver, I. (2013). A comparison of data quality and practicality of online versus postal questionnaires in a sample of testicular cancer survivors. Psycho-Oncology, 22(1), 233–237.

    Article  PubMed  Google Scholar 

  10. Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value Health, 11(2), 322–333.

    Article  PubMed  Google Scholar 

  11. McColl, E., Jacoby, A., Thomas, L., Soutter, J., Bamford, C., Steen, N., et al. (2001). Design and use of questionnaires: A review of best practice applicable to surveys of health service staff and patients. Health Technology Assessment, 5(31), 1–256.

    Article  CAS  PubMed  Google Scholar 

  12. Puhan, M. A., Ahuja, A., Van Natta, M. L., Ackatz, L. E., & Meinert, C. (2011). Interviewer versus self-administered health-related quality of life questionnaires—Does it matter? Health and Quality of Life Outcomes,. doi:10.1186/1477-7525-1189-1130.

    Google Scholar 

  13. Kmet, L., Lee, R., & Cook, L. (2004). Standard quality assessment criteria for evaluating primary research papers from a variety of fields. Health Technology Assessment, 13, 1–294.

  14. Lipsey, M., & Wilson, D. (2001). Practical meta-analysis. Thousand Oaks: Sage.

    Google Scholar 

  15. Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2009). Introduction to meta-analysis. Oxford: Wiley.

    Book  Google Scholar 

  16. Kongsved, S. M., Basnov, M., Holm-Christensen, K., & Hjollund, N. H. (2007). Response rate and completeness of questionnaires: A randomized study of Internet versus paper-and-pencil versions. Journal of Medical Internet Research, 9(3), e25.

    Article  PubMed Central  PubMed  Google Scholar 

  17. Reichmann, W. M., Losina, E., Seage, G. R., Arbelaez, C., Safren, S. A., Katz, J. N., et al. (2010). Does modality of survey administration impact data quality: Audio computer assisted self interview (ACASI) versus self-administered pen and paper? PLoS One, 5(1), e8728.

    Article  PubMed Central  PubMed  Google Scholar 

  18. Lannin, N. A., Anderson, C., Lim, J., Paice, K., Price, C., Faux, S., et al. (2013). Telephone follow-up was more expensive but more efficient than postal in a national stroke registry. Journal of Clinical Epidemiology, 66(8), 896–902.

    Article  PubMed  Google Scholar 

  19. Wijndaele, K., Matton, L., Duvigneaud, N., Lefevre, J., Duquet, W., Thomis, M., et al. (2007). Reliability, equivalence and respondent preference of computerized versus paper-and-pencil mental health questionnaires. Computers in Human Behavior, 23(4), 1958–1970.

    Article  Google Scholar 

  20. Rodriguez, H. P., von Glahn, T., Rogers, W. H., Chang, H., Fanjiang, G., & Safran, D. G. (2006). Evaluating patients’ experiences with individual physicians: A randomized trial of mail, internet, and interactive voice response telephone administration of surveys. Medical Care, 44(2), 167–174.

    Article  PubMed  Google Scholar 

  21. Zuidgeest, M., Hendriks, M., Koopman, L., Spreeuwenberg, P., & Rademakers, J. (2011). A comparison of a postal survey and mixed-mode survey using a questionnaire on patients’ experiences with breast care. Journal of Medical Internet Research, 13(3), e68.

    Article  PubMed Central  PubMed  Google Scholar 

  22. Rutherford, C., Nixon, J., Brown, J. M., Lamping, D. L., & Cano, S. J. (2014). Using mixed methods to select optimal mode of administration for a patient-reported outcome instrument for people with pressure ulcers. BMC Medical Research Methodology, 14(22), 1471–2288.

    Google Scholar 

  23. Shea, J. A., Guerra, C. E., Weiner, J., Aguirre, A. C., Ravenell, K. L., & Asch, D. A. (2008). Adapting a patient satisfaction instrument for low literate and Spanish-speaking populations: Comparison of three formats. Patient Education and Counseling, 73(1), 132–140.

    Article  PubMed  Google Scholar 

  24. Shih, T., & Fan, X. (2007). Response rates and mode preferences in web-mail mixed-mode surveys: A meta-analysis. International Journal of Internet Science, 2, 59–82.

    Google Scholar 

  25. Ashley, L., Keding, A., Brown, J., Velikova, G., & Wright, P. (2013). Score equivalence of electronic and paper versions of the Social Difficulties Inventory (SDI-21): A randomised crossover trial in cancer patients. Quality of Life Research, 22(6), 1435–1440.

    Article  PubMed  Google Scholar 

  26. Austin, J., Alvero, A. M., Fuchs, M. M., Patterson, L., & Anger, W. K. (2009). Pre-training to improve workshop performance in supervisor skills: An exploratory study of Latino agricultural workers. Journal of Agricultural Safety and Health, 15(3), 273–281.

    Article  CAS  PubMed  Google Scholar 

  27. Bjorner, J. B., Rose, M., Gandek, B., Stone, A. A., Junghaenel, D. U., & Ware, J. E, Jr. (2014). Method of administration of PROMIS scales did not significantly impact score level, reliability, or validity. Journal of Clinical Epidemiology, 67(1), 108–113.

    Article  PubMed Central  PubMed  Google Scholar 

  28. Caute, A., Northcott, S., Clarkson, L., Pring, T., & Hilari, K. (2012). Does mode of administration affect health-related quality-of-life outcomes after stroke? International Journal of Speechlanguage Pathology, 14(4), 329–337.

    Article  Google Scholar 

  29. Cerrada, C. J., Weinberg, J., Sherman, K. J., & Saper, R. B. (2014). Inter-method reliability of paper surveys and computer assisted telephone interviews in a randomized controlled trial of yoga for low back pain. BMC Research Notes, 7, 227. doi:10.1186/1756-0500-7-227.

    Article  PubMed Central  PubMed  Google Scholar 

  30. Chang, Y. J., Chang, C. H., Peng, C. L., Wu, H. C., Lin, H. C., Wang, J. Y., et al. (2014). Measurement equivalence and feasibility of the EORTC QLQ-PR25: Paper-and-pencil versus touch-screen administration. Health and Quality of Life Outcomes, 12, 23. doi:10.1186/1477-7525-12-23.

    Article  Google Scholar 

  31. Cheung, Y. B., Goh, C., Thumboo, J., Khoo, K. S., & Wee, J. (2006). Quality of life scores differed according to mode of administration in a review of three major oncology questionnaires. Journal of Clinical Epidemiology, 59(2), 185–191.

    Article  PubMed  Google Scholar 

  32. Clayton, J. A., Eydelman, M., Vitale, S., Manukyan, Z., Kramm, R., Datiles, M., et al. (2013). Web-based versus paper administration of common ophthalmic questionnaires: Comparison of subscale scores. Ophthalmology, 120(10), 2151–2159.

    Article  PubMed  Google Scholar 

  33. Coles, M. M., Cook, L. M., & Blake, T. R. (2007). Assessing obsessive compulsive symptoms and cognitions on the internet: Evidence for the comparability of paper and Internet administration. Behaviour Research and Therapy, 45(9), 2232–2240.

    Article  PubMed  Google Scholar 

  34. Collins, F. E., & Jones, K. V. (2004). Investigating dissociation online: Validation of a web-based version of the dissociative experiences scale. Journal of Trauma & Dissociation, 5(1), 133–147.

    Article  Google Scholar 

  35. Cook, A. J., Roberts, D. A., Henderson, M. D., Van Winkle, L. C., Chastain, D. C., & Hamill-Ruth, R. J. (2004). Electronic pain questionnaires: a randomized, crossover comparison with paper questionnaires for chronic pain assessment. Pain, 110(1–2), 310–317.

    Article  PubMed  Google Scholar 

  36. Greene, J., Speizer, H., & Wiitala, W. (2008). Telephone and web: Mixed-mode challenge. Health Services Research, 43(1 Pt 1), 230–248.

    PubMed Central  PubMed  Google Scholar 

  37. Grieve, R., & de Groot, H. T. (2011). Does online psychological test administration facilitate faking? Computers in Human Behavior, 27(6), 2386–2391.

    Article  Google Scholar 

  38. Gundy Cm, A. N. K. (2010). Effects of mode of administration (MOA) on the measurement properties of the EORTC QLQ-C30: A randomized study. Health and Quality of Life Outcomes, 8, 35. doi:10.1186/1477-7525-8-35.

    Article  PubMed Central  PubMed  Google Scholar 

  39. Handa, V. L., Barber, M. D., Young, S. B., Aronson, M. P., Morse, A., & Cundiff, G. W. (2008). Paper versus web-based administration of the pelvic floor distress inventory 20 and pelvic floor impact questionnaire 7. International Urogynecology Journal, 19(10), 1331–1335.

    Article  Google Scholar 

  40. Hauer, K., Yardley, L., Beyer, N., Kempen, G., Dias, N., Campbell, M., et al. (2010). Validation of the falls efficacy scale and falls efficacy scale international in geriatric patients with and without cognitive impairment: results of self-report and interview-based questionnaires. Gerontology, 56(2), 190–199.

    Article  CAS  PubMed  Google Scholar 

  41. Hayes, J., & Grieve, R. (2013). Faked depression: Comparing malingering via the internet, pen-and-paper, and telephone administration modes. Telemedicine and e Health, 19(9), 714–716.

    Article  PubMed  Google Scholar 

  42. Hedman, E., Ljotsson, B., Ruck, C., Furmark, T., Carlbring, P., Lindefors, N., & Andersson, G. (2010). Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation. Computers in Human Behavior, 26(4), 736–740.

    Article  Google Scholar 

  43. Hedman, E., Ljotsson, B., Blom, K., Alaoui, S. E., Kraepelien, M., Ruck, C., et al. (2013). Telephone versus internet administration of self-report measures of social anxiety, depressive symptoms, and insomnia: Psychometric evaluation of a method to reduce the impact of missing data. Journal of Medical Internet Research, 15(10), 131–138.

    Article  Google Scholar 

  44. Hollandare, F., Andersson, G., & Engstrom, I. (2010). A comparison of psychometric properties between internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. Journal of Medical Internet Research, 12(5), e49.

  45. Kobak, K. A. (2004). A comparison of face-to-face and videoconference administration of the Hamilton Depression Rating Scale. Journal of Telemedicine and Telecare, 10(4), 231–235.

    Article  PubMed  Google Scholar 

  46. Kobak, K. A., Williams, J. B. W., Jeglic, E., Salvucci, D., & Sharp, I. R. (2008). Face-to-face versus remote administration of the Montgomery–Asberg Depression Rating Scale using videoconference and telephone. Depression and Anxiety, 25(11), 913–919.

    Article  PubMed  Google Scholar 

  47. Lall, R., Mistry, D., Bridle, C., & Lamb, S. E. (2012). Telephone interviews can be used to collect follow-up data subsequent to no response to postal questionnaires in clinical trials. Journal of Clinical Epidemiology, 65(1), 90–99.

    Article  PubMed  Google Scholar 

  48. Lundy, J. J., & Coons, S. J. (2011). Measurement equivalence of interactive voice response and paper versions of the EQ-5D in a cancer patient sample. Value in Health, 14(6), 867–871.

    Article  PubMed  Google Scholar 

  49. Lungenhausen, M., Lange, S., Maier, C., Schaub, C., Trampisch, H. J., & Endres. H. G. (2007). Randomised controlled comparison of the Health Survey Short Form (SF-12) and the Graded Chronic Pain Scale (GCPS) in telephone interviews versus self-administered questionnaires. Are the results equivalent? BMC Medical Research Methodology, 7(50). doi:10.1186/1471-2288-7-50.

    PubMed Central  PubMed  Google Scholar 

  50. Marceau, L. D., Link, C., Jamison, R. N., & Carolan, S. (2007). Electronic diaries as a tool to improve pain management: Is there any evidence? Pain Medicine, 8(Suppl 3), S101–S109.

    Article  PubMed  Google Scholar 

  51. Matthew, A. G., Currie, K. L., Irvine, J., Ritvo, P., Santa Mina, D., Jamnicky, L., et al. (2007). Serial personal digital assistant data capture of health-related quality of life: A randomized controlled trial in a prostate cancer clinic. Health Qual Life Outcomes, 5, 38.

    Article  PubMed Central  PubMed  Google Scholar 

  52. Naus, M. J., Philipp, L. M., & Samsi, M. (2009). From paper to pixels: A comparison of paper and computer formats in psychological assessment. Computers in Human Behavior, 25(1), 1–7.

    Article  Google Scholar 

  53. Pinnock, H., Juniper, E. F., & Sheikh, A. (2005). Concordance between supervised and postal administration of the Mini Asthma Quality of Life Questionnaire (MiniAQLQ) and Asthma Control Questionnaire (ACQ) was very high. Journal of Clinical Epidemiology, 58(8), 809–814.

    Article  PubMed  Google Scholar 

  54. Ramachandran, S., Lundy, J. J., & Coons, S. J. (2008). Testing the measurement equivalence of paper and touch-screen versions of the EQ-5D visual analog scale (EQ VAS). Quality of Life Research, 17(8), 1117–1120.

    Article  PubMed  Google Scholar 

  55. Reissmann, D. R., John, M. T., & Schierz, O. (2011). Influence of administration method on oral health-related quality of life assessment using the Oral Health Impact Profile. European Journal of Oral Sciences, 119(1), 73–78.

    Article  PubMed  Google Scholar 

  56. Richter, J. G., Becker, A., Koch, T., Nixdorf, M., Willers, R., Monser, R., et al. (2008). Self-assessments of patients via Tablet PC in routine patient care: Comparison with standardised paper questionnaires. Annals of the Rheumatic Diseases, 67(12), 1739–1741.

    Article  CAS  PubMed  Google Scholar 

  57. Ritter, P., Lorig, K., Laurent, D., & Matthews, K. (2004). Internet versus mailed questionnaires: A randomized comparison. Journal of Medical Internet Research, 6(3), e29.

    Article  PubMed Central  PubMed  Google Scholar 

  58. Salaffi, F., Gasparini, S., & Grassi, W. (2009). The use of computer touch-screen technology for the collection of patient-reported outcome data in rheumatoid arthritis: Comparison with standardized paper questionnaires. Clinical and Experimental Rheumatology, 27(3), 459–468.

    CAS  PubMed  Google Scholar 

  59. Salaffi, F., Gasparini, S., Ciapetti, A., Gutierrez, M., & Grassi, W. (2013). Usability of an innovative and interactive electronic system for collection of patient-reported data in axial spondyloarthritis: Comparison with the traditional paper-administered format. Rheumatology, 52(11), 2062–2070.

    Article  PubMed  Google Scholar 

  60. Sikorski, A., Given, C. W., Given, B., Jeon, S., & You, M. (2009). Differential symptom reporting by mode of administration of the assessment: Automated voice response system versus a live telephone interview. Medical Care, 47(8), 866–874.

    Article  Google Scholar 

  61. Sousa, P. C., Mendes, F. M., Imparato, J. C., & Ardenghi, T. M. (2009). Differences in responses to the Oral Health Impact Profile (OHIP14) used as a questionnaire or in an interview. Pesquisa Odontologica Brasileira—Brazilian Oral Research, 23(4), 358–364.

    Google Scholar 

  62. Suris, A., Borman, P. D., Lind, L., & Kashner, T. M. (2007). Aggression, impulsivity, and health functioning in a veteran population: Equivalency and test-retest reliability of computerized and paper-and-pencil administrations. Computers in Human Behavior, 23(1), 97–110.

    Article  Google Scholar 

  63. Swartz, R. J., de Moor, C., Cook, K. F., Fouladi, R. T., Basen-Engquist, K., Eng, C., & Taylor, C. L. C. (2007). Mode effects in the center for epidemiologic studies depression (CES-D) scale: Personal digital assistant vs. paper and pencil administration. Quality of Life Research, 16(5), 803–813.

    Article  PubMed  Google Scholar 

  64. Tiplady, B., Goodman, K., Cummings, G., Lyle, D., Carrington, R., Battersby, C., & Ralston, S. H. (2010). Patient-reported outcomes in rheumatoid arthritis: Assessing the equivalence of electronic and paper data collection. The Patient: Patient Centered Outcomes Research, 3(3), 133–143.

    Article  Google Scholar 

  65. Weiler, K., Christ, A. M., Woodworth, G. G., Weiler, R. L., & Weiler, J. M. (2004). Quality of patient-reported outcome data captured using paper and interactive voice response diaries in an allergic rhinitis study: Is electronic data capture really better? Annals of Allergy, Asthma & Immunology, 92(3), 335–339.

    Article  Google Scholar 

  66. Whitehead, L. (2011). Methodological issues in Internet-mediated research: A randomized comparison of internet versus mailed questionnaires. Journal of Medical Internet Research, 13(4), e109.

  67. Wu, L. T., Pan, J. J., Blazer, D. G., Tai, B., Brooner, R. K., Stitzer, M. L., et al. (2009). The construct and measurement equivalence of cocaine and opioid dependences: A National Drug Abuse Treatment Clinical Trials Network (CTN) study. Drug and Alcohol Dependence, 103(3), 114–123.

    Article  PubMed Central  PubMed  Google Scholar 

  68. Yu, S. C., & Yu, M. N. (2007). Comparison of Internet-based and paper-based questionnaires in Taiwan using multisample invariance approach. Cyberpsychology & Behavior, 10(4), 501–507.

    Article  Google Scholar 

  69. Zimmerman, M., & Martinez, J. H. (2012). Web-based assessment of depression in patients treated in clinical practice: Reliability, validity, and patient acceptance. Journal of Clinical Psychiatry, 73(3), 333–338.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

We thank HR and LG for their contribution to this project as volunteers. We also acknowledge the support from our faculty librarian, Matthew Davis, in developing the search strategy.

Funding

C.R., D.C., R.M.B. and M.K. are supported by the Australian Government through Cancer Australia. No additional funding was sought for this review.

Author information

Authors and Affiliations

  1. Quality of Life Office, Psycho-oncology Co-operative Research Group, School of Psychology, University of Sydney, Level 6 North, Chris O’Brien Lifehouse (C39Z), Sydney, NSW, Australia

    Claudia Rutherford, Daniel Costa, Rebecca Mercieca-Bebber & Madeleine King

  2. Sydney Medical School, University of Sydney, Sydney, NSW, Australia

    Rebecca Mercieca-Bebber & Madeleine King

  3. Centre for Medical Psychology and Evidence-Based Decision-Making, School of Psychology, University of Sydney, Sydney, NSW, Australia

    Holly Rice

  4. Kings’ College London, London, UK

    Liam Gabb

Authors
  1. Claudia Rutherford
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rebecca Mercieca-Bebber
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Holly Rice
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Liam Gabb
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Madeleine King
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claudia Rutherford.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical approval

This article is a secondary analysis of published literature. It does not contain any studies with human participants or animals performed by any of the authors.

Electronic supplementary material

Below is the link to the electronic supplementary material (Search strategy).

Supplementary material 1 (DOCX 21 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rutherford, C., Costa, D., Mercieca-Bebber, R. et al. Mode of administration does not cause bias in patient-reported outcome results: a meta-analysis. Qual Life Res 25, 559–574 (2016). https://doi.org/10.1007/s11136-015-1110-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-015-1110-8

Keywords

Search

Navigation