Quality of Life Research

, Volume 25, Issue 3, pp 559–574 | Cite as

Mode of administration does not cause bias in patient-reported outcome results: a meta-analysis

  • Claudia RutherfordEmail author
  • Daniel Costa
  • Rebecca Mercieca-Bebber
  • Holly Rice
  • Liam Gabb
  • Madeleine King
Special Section: PROs in Non-Standard Settings (by invitation only)



Technological advances in recent decades have led to the availability of new modes to administer patient-reported outcomes (PROs). To aid selecting optimal modes of administration (MOA), we undertook a systematic review to determine whether differences in bias (both size and direction) exist among modes.


We searched five electronic databases from 2004 (date of last comprehensive review on this topic) to April 2014, cross-referenced and searched reference lists. Studies that compared two or more MOA for a health-related PRO measure in adult samples were included. Two reviewers independently applied inclusion and quality criteria and extracted findings. Meta-analyses and meta-regressions were conducted using random-effects models.


Of 5100 papers screened, 222 were considered potentially relevant and 56 met eligibility criteria. No evidence of bias was found for: (1) paper versus electronic self-complete; and (2) self-complete versus assisted MOA. Heterogeneity for paper versus electronic comparison was explained by type of construct (i.e. physical vs. psychological). Heterogeneity for self-completion versus assisted modes was in part explained by setting (clinic vs. home); the largest bias was introduced when assisted completion occurred in the clinic and follow-up was by self-completion (either electronic or paper) in the home.


Self-complete paper and electronic MOA can be used interchangeably for research in clinic and home settings. Self-completion and assisted completion produce equivalent scores overall, although heterogeneity may be induced by setting. These results support the use of mixed MOAs within a research study, which may be a useful strategy for reducing missing PRO data.


Systematic review Patient-reported outcome Mode of administration Bias 



We thank HR and LG for their contribution to this project as volunteers. We also acknowledge the support from our faculty librarian, Matthew Davis, in developing the search strategy.


C.R., D.C., R.M.B. and M.K. are supported by the Australian Government through Cancer Australia. No additional funding was sought for this review.

Compliance with ethical standards

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical approval

This article is a secondary analysis of published literature. It does not contain any studies with human participants or animals performed by any of the authors.

Supplementary material

11136_2015_1110_MOESM1_ESM.docx (21 kb)
Supplementary material 1 (DOCX 21 kb)


  1. 1.
    Food and Drug Administration. (2009). Patient reported outcome measures: Use in medical product development to support labelling claims. MD: US Department of Health & Human Support Food & Drug Administration.Google Scholar
  2. 2.
    Hood, K., Robling, M., Ingledew, D., Gillespie, D., Greene, G., Ivins, R., et al. (2012). Mode of data elicitation, acquisition and response to surveys: A systematic review. Health Technology Assessment, 16(27), 1–162.CrossRefPubMedGoogle Scholar
  3. 3.
    Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.CrossRefPubMedGoogle Scholar
  4. 4.
    Basch, E., Abernethy, A. P., Mullins, C. D., Reeve, B. B., Smith, M. L., Coons, S. J., et al. (2012). Recommendations for incorporating patient-reported outcomes into clinical comparative effectiveness research in adult oncology. Journal of Clinical Oncology, 30(34), 4249–4255.CrossRefPubMedGoogle Scholar
  5. 5.
    Stukenborg, G. J., Blackhall, L., Harrison, J., Barclay, J. S., Dillon, P., Davis, M. A., et al. (2014). Cancer patient-reported outcomes assessment using wireless touch screen tablet computers. Quality of Life Research, 23(5), 1603–1607.CrossRefPubMedGoogle Scholar
  6. 6.
    Scientific Advisory Committee of the Medical Outcomes Trust. (2002). Assessing health status and quality-of-life instruments: Attributes and review criteria. Quality of Life Research, 11(3), 193–205.CrossRefGoogle Scholar
  7. 7.
    Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27(3), 281–291.CrossRefPubMedGoogle Scholar
  8. 8.
    Bernhard, J., Cella, D. F., Coates, A. S., Fallowfield, L., Ganz, P. A., Moinpour, C. M., et al. (1998). Missing quality of life data in cancer clinical trials: Serious problems and challenges. Statistics in Medicine, 17(5–7), 517–532.CrossRefPubMedGoogle Scholar
  9. 9.
    Smith, A. B., King, M., Butow, P., & Olver, I. (2013). A comparison of data quality and practicality of online versus postal questionnaires in a sample of testicular cancer survivors. Psycho-Oncology, 22(1), 233–237.CrossRefPubMedGoogle Scholar
  10. 10.
    Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value Health, 11(2), 322–333.CrossRefPubMedGoogle Scholar
  11. 11.
    McColl, E., Jacoby, A., Thomas, L., Soutter, J., Bamford, C., Steen, N., et al. (2001). Design and use of questionnaires: A review of best practice applicable to surveys of health service staff and patients. Health Technology Assessment, 5(31), 1–256.CrossRefPubMedGoogle Scholar
  12. 12.
    Puhan, M. A., Ahuja, A., Van Natta, M. L., Ackatz, L. E., & Meinert, C. (2011). Interviewer versus self-administered health-related quality of life questionnaires—Does it matter? Health and Quality of Life Outcomes,. doi: 10.1186/1477-7525-1189-1130.Google Scholar
  13. 13.
    Kmet, L., Lee, R., & Cook, L. (2004). Standard quality assessment criteria for evaluating primary research papers from a variety of fields. Health Technology Assessment, 13, 1–294.Google Scholar
  14. 14.
    Lipsey, M., & Wilson, D. (2001). Practical meta-analysis. Thousand Oaks: Sage.Google Scholar
  15. 15.
    Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2009). Introduction to meta-analysis. Oxford: Wiley.CrossRefGoogle Scholar
  16. 16.
    Kongsved, S. M., Basnov, M., Holm-Christensen, K., & Hjollund, N. H. (2007). Response rate and completeness of questionnaires: A randomized study of Internet versus paper-and-pencil versions. Journal of Medical Internet Research, 9(3), e25.PubMedCentralCrossRefPubMedGoogle Scholar
  17. 17.
    Reichmann, W. M., Losina, E., Seage, G. R., Arbelaez, C., Safren, S. A., Katz, J. N., et al. (2010). Does modality of survey administration impact data quality: Audio computer assisted self interview (ACASI) versus self-administered pen and paper? PLoS One, 5(1), e8728.PubMedCentralCrossRefPubMedGoogle Scholar
  18. 18.
    Lannin, N. A., Anderson, C., Lim, J., Paice, K., Price, C., Faux, S., et al. (2013). Telephone follow-up was more expensive but more efficient than postal in a national stroke registry. Journal of Clinical Epidemiology, 66(8), 896–902.CrossRefPubMedGoogle Scholar
  19. 19.
    Wijndaele, K., Matton, L., Duvigneaud, N., Lefevre, J., Duquet, W., Thomis, M., et al. (2007). Reliability, equivalence and respondent preference of computerized versus paper-and-pencil mental health questionnaires. Computers in Human Behavior, 23(4), 1958–1970.CrossRefGoogle Scholar
  20. 20.
    Rodriguez, H. P., von Glahn, T., Rogers, W. H., Chang, H., Fanjiang, G., & Safran, D. G. (2006). Evaluating patients’ experiences with individual physicians: A randomized trial of mail, internet, and interactive voice response telephone administration of surveys. Medical Care, 44(2), 167–174.CrossRefPubMedGoogle Scholar
  21. 21.
    Zuidgeest, M., Hendriks, M., Koopman, L., Spreeuwenberg, P., & Rademakers, J. (2011). A comparison of a postal survey and mixed-mode survey using a questionnaire on patients’ experiences with breast care. Journal of Medical Internet Research, 13(3), e68.PubMedCentralCrossRefPubMedGoogle Scholar
  22. 22.
    Rutherford, C., Nixon, J., Brown, J. M., Lamping, D. L., & Cano, S. J. (2014). Using mixed methods to select optimal mode of administration for a patient-reported outcome instrument for people with pressure ulcers. BMC Medical Research Methodology, 14(22), 1471–2288.Google Scholar
  23. 23.
    Shea, J. A., Guerra, C. E., Weiner, J., Aguirre, A. C., Ravenell, K. L., & Asch, D. A. (2008). Adapting a patient satisfaction instrument for low literate and Spanish-speaking populations: Comparison of three formats. Patient Education and Counseling, 73(1), 132–140.CrossRefPubMedGoogle Scholar
  24. 24.
    Shih, T., & Fan, X. (2007). Response rates and mode preferences in web-mail mixed-mode surveys: A meta-analysis. International Journal of Internet Science, 2, 59–82.Google Scholar
  25. 25.
    Ashley, L., Keding, A., Brown, J., Velikova, G., & Wright, P. (2013). Score equivalence of electronic and paper versions of the Social Difficulties Inventory (SDI-21): A randomised crossover trial in cancer patients. Quality of Life Research, 22(6), 1435–1440.CrossRefPubMedGoogle Scholar
  26. 26.
    Austin, J., Alvero, A. M., Fuchs, M. M., Patterson, L., & Anger, W. K. (2009). Pre-training to improve workshop performance in supervisor skills: An exploratory study of Latino agricultural workers. Journal of Agricultural Safety and Health, 15(3), 273–281.CrossRefPubMedGoogle Scholar
  27. 27.
    Bjorner, J. B., Rose, M., Gandek, B., Stone, A. A., Junghaenel, D. U., & Ware, J. E, Jr. (2014). Method of administration of PROMIS scales did not significantly impact score level, reliability, or validity. Journal of Clinical Epidemiology, 67(1), 108–113.PubMedCentralCrossRefPubMedGoogle Scholar
  28. 28.
    Caute, A., Northcott, S., Clarkson, L., Pring, T., & Hilari, K. (2012). Does mode of administration affect health-related quality-of-life outcomes after stroke? International Journal of Speechlanguage Pathology, 14(4), 329–337.CrossRefGoogle Scholar
  29. 29.
    Cerrada, C. J., Weinberg, J., Sherman, K. J., & Saper, R. B. (2014). Inter-method reliability of paper surveys and computer assisted telephone interviews in a randomized controlled trial of yoga for low back pain. BMC Research Notes, 7, 227. doi: 10.1186/1756-0500-7-227.PubMedCentralCrossRefPubMedGoogle Scholar
  30. 30.
    Chang, Y. J., Chang, C. H., Peng, C. L., Wu, H. C., Lin, H. C., Wang, J. Y., et al. (2014). Measurement equivalence and feasibility of the EORTC QLQ-PR25: Paper-and-pencil versus touch-screen administration. Health and Quality of Life Outcomes, 12, 23. doi: 10.1186/1477-7525-12-23.CrossRefGoogle Scholar
  31. 31.
    Cheung, Y. B., Goh, C., Thumboo, J., Khoo, K. S., & Wee, J. (2006). Quality of life scores differed according to mode of administration in a review of three major oncology questionnaires. Journal of Clinical Epidemiology, 59(2), 185–191.CrossRefPubMedGoogle Scholar
  32. 32.
    Clayton, J. A., Eydelman, M., Vitale, S., Manukyan, Z., Kramm, R., Datiles, M., et al. (2013). Web-based versus paper administration of common ophthalmic questionnaires: Comparison of subscale scores. Ophthalmology, 120(10), 2151–2159.CrossRefPubMedGoogle Scholar
  33. 33.
    Coles, M. M., Cook, L. M., & Blake, T. R. (2007). Assessing obsessive compulsive symptoms and cognitions on the internet: Evidence for the comparability of paper and Internet administration. Behaviour Research and Therapy, 45(9), 2232–2240.CrossRefPubMedGoogle Scholar
  34. 34.
    Collins, F. E., & Jones, K. V. (2004). Investigating dissociation online: Validation of a web-based version of the dissociative experiences scale. Journal of Trauma & Dissociation, 5(1), 133–147.CrossRefGoogle Scholar
  35. 35.
    Cook, A. J., Roberts, D. A., Henderson, M. D., Van Winkle, L. C., Chastain, D. C., & Hamill-Ruth, R. J. (2004). Electronic pain questionnaires: a randomized, crossover comparison with paper questionnaires for chronic pain assessment. Pain, 110(1–2), 310–317.CrossRefPubMedGoogle Scholar
  36. 36.
    Greene, J., Speizer, H., & Wiitala, W. (2008). Telephone and web: Mixed-mode challenge. Health Services Research, 43(1 Pt 1), 230–248.PubMedCentralPubMedGoogle Scholar
  37. 37.
    Grieve, R., & de Groot, H. T. (2011). Does online psychological test administration facilitate faking? Computers in Human Behavior, 27(6), 2386–2391.CrossRefGoogle Scholar
  38. 38.
    Gundy Cm, A. N. K. (2010). Effects of mode of administration (MOA) on the measurement properties of the EORTC QLQ-C30: A randomized study. Health and Quality of Life Outcomes, 8, 35. doi: 10.1186/1477-7525-8-35.PubMedCentralCrossRefPubMedGoogle Scholar
  39. 39.
    Handa, V. L., Barber, M. D., Young, S. B., Aronson, M. P., Morse, A., & Cundiff, G. W. (2008). Paper versus web-based administration of the pelvic floor distress inventory 20 and pelvic floor impact questionnaire 7. International Urogynecology Journal, 19(10), 1331–1335.CrossRefGoogle Scholar
  40. 40.
    Hauer, K., Yardley, L., Beyer, N., Kempen, G., Dias, N., Campbell, M., et al. (2010). Validation of the falls efficacy scale and falls efficacy scale international in geriatric patients with and without cognitive impairment: results of self-report and interview-based questionnaires. Gerontology, 56(2), 190–199.CrossRefPubMedGoogle Scholar
  41. 41.
    Hayes, J., & Grieve, R. (2013). Faked depression: Comparing malingering via the internet, pen-and-paper, and telephone administration modes. Telemedicine and e Health, 19(9), 714–716.CrossRefPubMedGoogle Scholar
  42. 42.
    Hedman, E., Ljotsson, B., Ruck, C., Furmark, T., Carlbring, P., Lindefors, N., & Andersson, G. (2010). Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation. Computers in Human Behavior, 26(4), 736–740.CrossRefGoogle Scholar
  43. 43.
    Hedman, E., Ljotsson, B., Blom, K., Alaoui, S. E., Kraepelien, M., Ruck, C., et al. (2013). Telephone versus internet administration of self-report measures of social anxiety, depressive symptoms, and insomnia: Psychometric evaluation of a method to reduce the impact of missing data. Journal of Medical Internet Research, 15(10), 131–138.CrossRefGoogle Scholar
  44. 44.
    Hollandare, F., Andersson, G., & Engstrom, I. (2010). A comparison of psychometric properties between internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. Journal of Medical Internet Research, 12(5), e49.Google Scholar
  45. 45.
    Kobak, K. A. (2004). A comparison of face-to-face and videoconference administration of the Hamilton Depression Rating Scale. Journal of Telemedicine and Telecare, 10(4), 231–235.CrossRefPubMedGoogle Scholar
  46. 46.
    Kobak, K. A., Williams, J. B. W., Jeglic, E., Salvucci, D., & Sharp, I. R. (2008). Face-to-face versus remote administration of the Montgomery–Asberg Depression Rating Scale using videoconference and telephone. Depression and Anxiety, 25(11), 913–919.CrossRefPubMedGoogle Scholar
  47. 47.
    Lall, R., Mistry, D., Bridle, C., & Lamb, S. E. (2012). Telephone interviews can be used to collect follow-up data subsequent to no response to postal questionnaires in clinical trials. Journal of Clinical Epidemiology, 65(1), 90–99.CrossRefPubMedGoogle Scholar
  48. 48.
    Lundy, J. J., & Coons, S. J. (2011). Measurement equivalence of interactive voice response and paper versions of the EQ-5D in a cancer patient sample. Value in Health, 14(6), 867–871.CrossRefPubMedGoogle Scholar
  49. 49.
    Lungenhausen, M., Lange, S., Maier, C., Schaub, C., Trampisch, H. J., & Endres. H. G. (2007). Randomised controlled comparison of the Health Survey Short Form (SF-12) and the Graded Chronic Pain Scale (GCPS) in telephone interviews versus self-administered questionnaires. Are the results equivalent? BMC Medical Research Methodology, 7(50). doi: 10.1186/1471-2288-7-50.PubMedCentralPubMedGoogle Scholar
  50. 50.
    Marceau, L. D., Link, C., Jamison, R. N., & Carolan, S. (2007). Electronic diaries as a tool to improve pain management: Is there any evidence? Pain Medicine, 8(Suppl 3), S101–S109.CrossRefPubMedGoogle Scholar
  51. 51.
    Matthew, A. G., Currie, K. L., Irvine, J., Ritvo, P., Santa Mina, D., Jamnicky, L., et al. (2007). Serial personal digital assistant data capture of health-related quality of life: A randomized controlled trial in a prostate cancer clinic. Health Qual Life Outcomes, 5, 38.PubMedCentralCrossRefPubMedGoogle Scholar
  52. 52.
    Naus, M. J., Philipp, L. M., & Samsi, M. (2009). From paper to pixels: A comparison of paper and computer formats in psychological assessment. Computers in Human Behavior, 25(1), 1–7.CrossRefGoogle Scholar
  53. 53.
    Pinnock, H., Juniper, E. F., & Sheikh, A. (2005). Concordance between supervised and postal administration of the Mini Asthma Quality of Life Questionnaire (MiniAQLQ) and Asthma Control Questionnaire (ACQ) was very high. Journal of Clinical Epidemiology, 58(8), 809–814.CrossRefPubMedGoogle Scholar
  54. 54.
    Ramachandran, S., Lundy, J. J., & Coons, S. J. (2008). Testing the measurement equivalence of paper and touch-screen versions of the EQ-5D visual analog scale (EQ VAS). Quality of Life Research, 17(8), 1117–1120.CrossRefPubMedGoogle Scholar
  55. 55.
    Reissmann, D. R., John, M. T., & Schierz, O. (2011). Influence of administration method on oral health-related quality of life assessment using the Oral Health Impact Profile. European Journal of Oral Sciences, 119(1), 73–78.CrossRefPubMedGoogle Scholar
  56. 56.
    Richter, J. G., Becker, A., Koch, T., Nixdorf, M., Willers, R., Monser, R., et al. (2008). Self-assessments of patients via Tablet PC in routine patient care: Comparison with standardised paper questionnaires. Annals of the Rheumatic Diseases, 67(12), 1739–1741.CrossRefPubMedGoogle Scholar
  57. 57.
    Ritter, P., Lorig, K., Laurent, D., & Matthews, K. (2004). Internet versus mailed questionnaires: A randomized comparison. Journal of Medical Internet Research, 6(3), e29.PubMedCentralCrossRefPubMedGoogle Scholar
  58. 58.
    Salaffi, F., Gasparini, S., & Grassi, W. (2009). The use of computer touch-screen technology for the collection of patient-reported outcome data in rheumatoid arthritis: Comparison with standardized paper questionnaires. Clinical and Experimental Rheumatology, 27(3), 459–468.PubMedGoogle Scholar
  59. 59.
    Salaffi, F., Gasparini, S., Ciapetti, A., Gutierrez, M., & Grassi, W. (2013). Usability of an innovative and interactive electronic system for collection of patient-reported data in axial spondyloarthritis: Comparison with the traditional paper-administered format. Rheumatology, 52(11), 2062–2070.CrossRefPubMedGoogle Scholar
  60. 60.
    Sikorski, A., Given, C. W., Given, B., Jeon, S., & You, M. (2009). Differential symptom reporting by mode of administration of the assessment: Automated voice response system versus a live telephone interview. Medical Care, 47(8), 866–874.CrossRefGoogle Scholar
  61. 61.
    Sousa, P. C., Mendes, F. M., Imparato, J. C., & Ardenghi, T. M. (2009). Differences in responses to the Oral Health Impact Profile (OHIP14) used as a questionnaire or in an interview. Pesquisa Odontologica Brasileira—Brazilian Oral Research, 23(4), 358–364.Google Scholar
  62. 62.
    Suris, A., Borman, P. D., Lind, L., & Kashner, T. M. (2007). Aggression, impulsivity, and health functioning in a veteran population: Equivalency and test-retest reliability of computerized and paper-and-pencil administrations. Computers in Human Behavior, 23(1), 97–110.CrossRefGoogle Scholar
  63. 63.
    Swartz, R. J., de Moor, C., Cook, K. F., Fouladi, R. T., Basen-Engquist, K., Eng, C., & Taylor, C. L. C. (2007). Mode effects in the center for epidemiologic studies depression (CES-D) scale: Personal digital assistant vs. paper and pencil administration. Quality of Life Research, 16(5), 803–813.CrossRefPubMedGoogle Scholar
  64. 64.
    Tiplady, B., Goodman, K., Cummings, G., Lyle, D., Carrington, R., Battersby, C., & Ralston, S. H. (2010). Patient-reported outcomes in rheumatoid arthritis: Assessing the equivalence of electronic and paper data collection. The Patient: Patient Centered Outcomes Research, 3(3), 133–143.CrossRefGoogle Scholar
  65. 65.
    Weiler, K., Christ, A. M., Woodworth, G. G., Weiler, R. L., & Weiler, J. M. (2004). Quality of patient-reported outcome data captured using paper and interactive voice response diaries in an allergic rhinitis study: Is electronic data capture really better? Annals of Allergy, Asthma & Immunology, 92(3), 335–339.CrossRefGoogle Scholar
  66. 66.
    Whitehead, L. (2011). Methodological issues in Internet-mediated research: A randomized comparison of internet versus mailed questionnaires. Journal of Medical Internet Research, 13(4), e109.Google Scholar
  67. 67.
    Wu, L. T., Pan, J. J., Blazer, D. G., Tai, B., Brooner, R. K., Stitzer, M. L., et al. (2009). The construct and measurement equivalence of cocaine and opioid dependences: A National Drug Abuse Treatment Clinical Trials Network (CTN) study. Drug and Alcohol Dependence, 103(3), 114–123.PubMedCentralCrossRefPubMedGoogle Scholar
  68. 68.
    Yu, S. C., & Yu, M. N. (2007). Comparison of Internet-based and paper-based questionnaires in Taiwan using multisample invariance approach. Cyberpsychology & Behavior, 10(4), 501–507.CrossRefGoogle Scholar
  69. 69.
    Zimmerman, M., & Martinez, J. H. (2012). Web-based assessment of depression in patients treated in clinical practice: Reliability, validity, and patient acceptance. Journal of Clinical Psychiatry, 73(3), 333–338.CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Claudia Rutherford
    • 1
    Email author
  • Daniel Costa
    • 1
  • Rebecca Mercieca-Bebber
    • 1
    • 2
  • Holly Rice
    • 3
  • Liam Gabb
    • 4
  • Madeleine King
    • 1
    • 2
  1. 1.Quality of Life Office, Psycho-oncology Co-operative Research Group, School of PsychologyUniversity of SydneySydneyAustralia
  2. 2.Sydney Medical SchoolUniversity of SydneySydneyAustralia
  3. 3.Centre for Medical Psychology and Evidence-Based Decision-Making, School of PsychologyUniversity of SydneySydneyAustralia
  4. 4.Kings’ College LondonLondonUK

Personalised recommendations