Quality of Life Research

, Volume 24, Issue 8, pp 1949–1961 | Cite as

Equivalence of electronic and paper-based patient-reported outcome measures

  • Niloufar Campbell
  • Faraz Ali
  • Andrew Y. Finlay
  • Sam S. Salek



Electronic formats (ePROs) of paper-based patient-reported outcomes (PROs) should be validated before they can be reliably used. This review aimed to examine studies investigating measurement equivalence between ePROs and their paper originals to identify methodologies used and to determine the extent of such validation.


Three databases (OvidSP, Web of Science and PubMed) were searched using a set of keywords. Results were examined for compliance with inclusion criteria. Articles or abstracts that directly compared screen-based electronic versions of PROs with their validated paper-based originals, with regard to their measurement equivalence, were included. Publications were excluded if the only instruments reported were stand-alone visual analogue scales or interactive voice response formats. Papers published before 2007 were excluded, as a previous meta-analysis examined papers published before this time.


Fifty-five studies investigating 79 instruments met the inclusion criteria. 53 % of the 79 instruments studied were condition specific. Several instruments, such as the SF-36, were reported in more than one publication. The most frequently reported formats for ePROs were Web-based versions. In 78 % of the publications, there was evidence of equivalence or comparability between the two formats as judged by study authors. Of the 30 publications that provided preference data, 87 % found that overall participants preferred the electronic format.


When examining equivalence between paper and electronic versions of PROs, formats are usually judged by authors to be equivalent. Participants prefer electronic formats. This literature review gives encouragement to the further widespread development and use of ePROs.


Patient-reported outcome measures PRO Electronic PROs Validation Equivalence 


Conflict of interest

AYF is joint copyright owner of the DLQI.


  1. 1.
    Center for Drug Evaluation and Research (CDER). (2009). Guidance for industry—patient-reported outcome measures: Use in medical product development to support labeling claims. Silver Spring: Food and Drug Administration.Google Scholar
  2. 2.
    Leidy, N. K., & Vernon, M. (2008). Perspectives on patient-reported outcomes: content validity and qualitative research in a changing clinical trial environment. Pharmacoeconomics, 26(5), 363–370.PubMedCrossRefGoogle Scholar
  3. 3.
    Marshall, S., Haywood, K., & Fitzpatrick, R. (2006). Impact of patient-reported outcome measures on routine practice: A structured review. Journal of Evaluation in Clinical Practice, 12(5), 559–568.PubMedCrossRefGoogle Scholar
  4. 4.
    Lee, S. J., Kavanaugh, A., & Lenert, L. (2007). Electronic and computer-generated patient questionnaires in standard care. Best Practice and Research Clinical Rheumatology, 21(4), 637–647.PubMedCrossRefGoogle Scholar
  5. 5.
    Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429.PubMedCrossRefGoogle Scholar
  6. 6.
    Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value in Health, 11(2), 322–333.PubMedCrossRefGoogle Scholar
  7. 7.
    Arthur, C. (2012). The history of smartphones: timeline. The Guardian, 24 January 2012 (Online). http://www.theguardian.com/technology/2012/jan/24/smartphones-timeline. Accessed 2 May 2014.
  8. 8.
    McLellan, C. (2014). The History of Tablet Computers: A timeline (Online). http://www.zdnet.com/the-history-of-tablet-computers-a-timeline-7000026555/. Accessed 2 May 2014.
  9. 9.
    Salaffi, F., Gasparini, S., & Grassi, W. (2009). The use of computer touch-screen technology for the collection of the patient-reported outcome data in rheumatoid arthritis: Comparison to standardised patient questionnaires. Clinical and Experimental Rheumatology, 27(3), 459–468.PubMedGoogle Scholar
  10. 10.
    MacKenzie, H., Thavaneswaran, A., Chandran, V., & Gladman, D. D. (2011). Patient-reported outcome in psoriatic arthritis: A comparison of web-based versus paper-completed questionnaires. The Journal of Rheumatology, 38(12), 2619–2624.PubMedCrossRefGoogle Scholar
  11. 11.
    Wu, R. C., Thorpe, K., Math, M., Ross, H., Micevski, V., Marquez, C., Straus, S. E. (2009). Comparing administration of questionnaires via the Internet to pen-and-paper in patients with heart failure: Randomised controlled trial. Journal of Medical Internet Research (Online), 11(1). http://www.jmir.org/2009/1/e3/. Accessed 1 May 2014.
  12. 12.
    Juniper, E. F., Riis, B., & Juniper, B. A. (2007). Development and validation of an electronic version of the Rhinoconjunctivitis quality of life questionnaire. Allergy, 62(9), 1091–1093.PubMedCrossRefGoogle Scholar
  13. 13.
    Ring, A. E., Cheong, K. A., Watkins, C. L., Meddis, D., Cella, D., & Harper, P. G. (2008). A randomised study of electronic diary versus paper and pencil collection of patient-reported outcomes in patients with non-small cell lung cancer. The Patient: Patient-Centered Outcomes Research, 1(2), 105–113.CrossRefGoogle Scholar
  14. 14.
    Juniper, E. F., Langlands, J. M., & Juniper, B. A. (2009). Patients may respond differently to paper and electronic versions of the same questionnaires. Respiratory Medicine, 103(6), 932–934.PubMedCrossRefGoogle Scholar
  15. 15.
    Naus, M. J., Philipp, L. M., & Samsi, M. (2009). From paper to pixels: A comparison of paper and computer formats in psychological assessment. Computers in Human Behavior, 25(1), 1–7.CrossRefGoogle Scholar
  16. 16.
    Ribeiro, C., Moreira, L., Silveira, A., Silva, I., Gestal, J., & Vasconcelos, C. (2010). Development and use of touch-screen computer-assisted self-interview in Portuguese patients with chronic immune disease: Evaluation of an electronic version of SF-36v2. Acta Reumatológica Portuguesa, 35(2), 208–214.PubMedGoogle Scholar
  17. 17.
    Matthew, A. G., Currie, K. L., Irvine, J., Ritvo, P., Mina, D. S., Jamnicky, L., Nam, R., Trachtenberg, J. (2007). Serial personal digital assistant data capture of health-related quality of life: A randomised controlled trial in a prostate cancer clinic. Health and Quality of Life Outcomes (Online), 5(38). http://www.hqlo.com/content/5/1/38. Accessed 30 April 2014.
  18. 18.
    Schefte, D. B., & Hetland, M. L. (2010). An open-source, self-explanatory touch screen in routine care. Validity of filling in the bath measures on ankylosing spondylitis disease activity index, function index, the health activity index, function index, the health assessment questionnaire and visual analogue scales in comparison with paper versions. Rheumatology, 49(1), 99–104.PubMedCrossRefGoogle Scholar
  19. 19.
    Handa, V. L., Barber, M. D., Young, S. B., Aronson, M. P., Morse, A., & Cundiff, G. W. (2008). Paper versus web-based administration of the pelvic floor distress inventory 20 and the pelvic floor impact questionnaire 7. International Urogynecology Journal, 19(10), 1331–1335.CrossRefGoogle Scholar
  20. 20.
    Swartz, R. J., Moor, C. D., Cook, K. F., Fouladi, R. T., Basen-Engquist, K., Eng, C., & Taylor, C. L. C. (2007). Mode effects in the center for epidemiological studies depression (CES-D) scale: Personal digital assistant versus paper and pencil administration. Quality of Life Research, 16(5), 803–813.PubMedCrossRefGoogle Scholar
  21. 21.
    Carlbring, P., Brunt, S., Bohman, S., Austin, D., Richards, J., Öst, L.-G., & Andersson, G. (2007). Internet versus paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Computers in Human Behaviour, 23(3), 1421–1434.CrossRefGoogle Scholar
  22. 22.
    Clayer, M., & Davis, A. (2011). Can the Toronto extremity salvage score produce reliable results when used online? Clinical Orthopaedics and Related Research, 469(6), 1750–1756.PubMedCentralPubMedCrossRefGoogle Scholar
  23. 23.
    Frennered, K., Hägg, O., & Wessberg, P. (2010). Validity of a computer touch-screen questionnaire system in back patients. Spine, 35(6), 697–703.PubMedCrossRefGoogle Scholar
  24. 24.
    Oliveira, A., Ferreira, P. L., Antunes, B., & Pimentel, F. L. (2011). OnQol: Electronic device to capture QoL data in oncology: Difference between patients 65 years or older and patients younger than 65 years of age. Journal of Geriatric Oncology, 2(4), 253–258.CrossRefGoogle Scholar
  25. 25.
    Saunders, G., Forsline, A., & Jacobs, P. (2007). The Attitudes towards Loss of Hearing Questionnaire (ALHQ): A comparison of paper and electronic formats. The Journal of the American Academy of Audiology, 18(1), 66–77.CrossRefGoogle Scholar
  26. 26.
    Silveira, A., Gonçalves, J., Sequeira, T., Ribeiro, C., Lopes, C., Monteiro, E., & Pimentel, F. L. (2011). Computer-based quality-of-life monitoring in head and neck cancer patients: A validation model using the EORTC-QLQ C30 and EORTC-H&N35 Portuguese PC-software version. Acta Médica Portuguesa, 24(S2), 347–354.PubMedGoogle Scholar
  27. 27.
    Wright, E. P., Kiely, M., Johnston, C., Smith, A. B., Cull, A., & Selby, P. J. (2005). Development and evaluation of an instrument to assess social difficulties in routine oncology practice. Quality of Life Research, 14(2), 373–386.PubMedCrossRefGoogle Scholar
  28. 28.
    Beaumont, J. L., Victorson, D., Su, J., Baker, C. L., Wortman, K., Shah, H., & Cella, D. (2011). Examining web equivalence and risk factor sensitivity of the COPD Population Screener. Value in Health, 14(4), 506–512.PubMedCrossRefGoogle Scholar
  29. 29.
    Bernstein, A. N., Levinson, A. W., Hobbs, A. R., Lavery, H. J., & Samadi, D. B. (2013). Validation of online administration of the sexual health inventory for men. The Journal of Urology, 189(4), 1456–1461.PubMedCrossRefGoogle Scholar
  30. 30.
    Bruce, B., & Fries, J. F. (2011). Internet versus mailed administration of the health assessment questionnaire disability index. 63(10), pp. S1–S1 256.Google Scholar
  31. 31.
    Chang, Y-J., Chang, C-H., Peng, C-L., Wu, H-C., Lin, H-C., Wang, J-Y., Li, T-C., Yeh, Y-C., Liang, W-M. (2014). Measurement equivalence and feasibility of the EORTC QLQ-PR25: paper-and-pencil versus touch-screen administration. Health and Quality of Life Outcomes (Online), 12(23). http://www.hqlo.com/content/12/1/23. Accessed 29 April 2014.
  32. 32.
    Chen, H-L., Tien, S-W., Shih, C-C. (2011). Paper questionnaire versus Web questionnaire for clinical research impact using the short form of the UDI-6, HQ-7, PISQ-12. International Conference on Engineering and Business Management (EBM2011), 1–6, pp. 1293–1297.Google Scholar
  33. 33.
    Chen, T-h, Li, L., Sigle, J. M., Du, Y.-P., Wang, H.-M., & Lei, J. (2007). Crossover randomised controlled trial of the electronic version of the Chinese SF-36. Journal of Zheijang University Science B, 8(8), 604–608.CrossRefGoogle Scholar
  34. 34.
    Chen, W.-C., Wang, J.-D., Hwang, J.-S., Chen, C.-C., Wu, C.-H., & Yao, G. (2009). Can the web-form WHOQOL-BREF be an alternative to the paper-form? Social Indicators Research, 94(1), 97–114.CrossRefGoogle Scholar
  35. 35.
    Chen, T-h, & Li, L. (2010). Pilot study of equivalence between the electronic and paper version of the Chinese SF-36. Journal of Happiness Studies, 11(2), 151–161.CrossRefGoogle Scholar
  36. 36.
    Dalal, A. A., Nelson, L., Gilligan, T., McLeod, L., Lewis, S., & De Muro-Mercon, C. (2011). Evaluating patient-reported outcome measurement comparability between paper and alternative versions, using the lung function questionnaire as an example. Value in Health, 14(5), 712–720.PubMedCrossRefGoogle Scholar
  37. 37.
    Dinkel, A., Berg, P., Pirker, C., Geinitz, H., Sehlen, S., Emrich, M., et al. (2010). Routine psychosocial distress screening in radiotherapy: implementation and evaluation of a computerised procedure. British Journal of Cancer, 103, 1489–1495.PubMedCentralPubMedCrossRefGoogle Scholar
  38. 38.
    Gudbergsen, H., Bartels, E. M., Krusager, P., Waehrens, E. E., Christensen, R., Danneskoild-Samsöe, B., Bliddal, H. (2011). Test-retest of computerised health status questionnaires frequently used in the monitoring of knee osteoarthritis: a randomised crossover trial. BMC Musculoskeletal Disorders (Online), 12(190). http://www.biomedcentral.com/1471-2474/12/190. Accessed 29 April 2014.
  39. 39.
    Hedman, E., Ljótsson, B., Rück, C., Furmark, T., Carlbring, P., Lindefors, N., & Andersson, G. (2010). Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation. Computers in Human Behaviour, 26(4), 736–740.CrossRefGoogle Scholar
  40. 40.
    Heiberg, T., Kvien, T. K., Dale, Ø., Mowinckel, P., Aanerud, G. J., Songe-Møller, A. B., et al. (2007). Daily health status registration (patient diary) in patients with rheumatoid arthritis: A comparison between personal digital assistant and paper-pencil format. Arthritis Care & Research, 57(3), 454–460.CrossRefGoogle Scholar
  41. 41.
    Holländare, F., Andersson, G., Engström, I. (2010). A comparison of psychometric properties between Internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. Journal of Medical Internet Research (Online), 12(5). http://www.jmir.org/2010/5/e49/. Accessed 1 May 2014.
  42. 42.
    Hollen, P. J., Gralla, R. J., Stewart, J. A., Meharchand, J. M., Wierzbicki, R., & Leighl, N. (2013). Can a computerised format replace a paper form in PRO and HRQL evaluation? Psychometric testing of the computer-assisted LCSS instrument (eLCSS-QL). Supportive Care in Cancer, 21(1), 165–172.PubMedCrossRefGoogle Scholar
  43. 43.
    Howell, R. T., Rodzon, K. S., Kurai, M., & Sanchez, A. H. (2010). A validation of well-being and happiness surveys for administration via the Internet. Behaviour Research Methods, 42(3), 775–784.CrossRefGoogle Scholar
  44. 44.
    Lalanne, C., Herrmann, S., Armstrong, A. R., Cheung-Lung, C., Schwartz, Y., Chassany, O., & Duracinsky, M. (2013). Paper-based and electronic assessment of health-related quality of life specific HIV disease: A reliability study with the PROQOL-HIV questionnaire. Value in Health, 16(7), A362.CrossRefGoogle Scholar
  45. 45.
    Lee, E.-H. (2009). Touch-screen computerised quality-of-life assessment for patients with cancer. Asian Nursing Research, 3(1), 41–48.PubMedCrossRefGoogle Scholar
  46. 46.
    Lee, E.-H. (2009). Computerised measurement for asthma-specific quality of life: Comparison with a conventional paper-and-pencil questionnaire. Journal of Korean Academy Nursing, 39(6), 781–787.CrossRefGoogle Scholar
  47. 47.
    Lee, E-H., Lee, Y.W., Lee, K-W., Kim, D.J., Kim, Y-S., Nam, M-S. (2013). Measurement equivalence of touch-screen computerised and paper-based diabetes-specific quality-of-life questionnaires. International Journal of Nursing Practice (Online). http://onlinelibrary.wiley.com/doi/10.1111/ijn.12184/abstract. Accessed 30 April 2014.
  48. 48.
    Minard, J. P., Thomas, N., Olajos-Clow, J., Juniper, E. F., Jiang, X., Jenkins, B., Taite, A. K., Turcotte, S., Lougheed, M. D. (2011)a. Validation of an electronic version of the Pediatric Caregiver’s Asthma Quality of Life Questionnaire (PACQLQ). American Journal of Respiratory and Critical Care Medicine (Online),183. http://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A1432. Accessed 30 April 2014.
  49. 49.
    Minard, J.P., Thomas, N., Olajos-Clow, J., Juniper, E. F., Jiang, X., Jenkins, B., Taite, A. K., Turcotte, S., Lougheed, M. D. (2011)b. Validation of an electronic version of the Mini Pediatric Asthmas Quality of Life Questionnaire (Mini PAQLQ). American Journal of Respiratory and Critical Care Medicine (Online). http://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A1433. Accessed 30 April 2014.
  50. 50.
    Olajos-Clow, J., Minard, J., Szpiro, K., Juniper, E. F., Turcotte, S., Jiang, X., et al. (2010). Validation of an electronic version of the Mini Asthma quality of life questionnaire. Respiratory Medicine, 104(5), 658–667.PubMedCrossRefGoogle Scholar
  51. 51.
    Oliveira, A., Ferreira, P. L., Antunes, B., & Pimentel, F. L. (2010). Quality of life in oncology: Electronic device to collect data. Acta Médica Portuguesa, 23(6), 1017–1024.PubMedGoogle Scholar
  52. 52.
    Parnell, B. A., Dunivan, G. C., Connolly, A., Jannelli, M. L., Wells, E. C., & Geller, E. J. (2011). Validation of web-based administration of the pelvic organ prolapse/urinary incontinence sexual function questionnaire (PISQ-12). International Urogynecology Journal, 22(3), 357–361.PubMedCrossRefGoogle Scholar
  53. 53.
    Raat, H., Mangunkusumo, R. T., Landgraf, J. M., Kloek, G., & Brug, J. (2007). Feasibility, reliability and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): Internet administration compared with the standard paper version. Quality of Life Research, 16(4), 675–685.PubMedCentralPubMedCrossRefGoogle Scholar
  54. 54.
    Ribeiro, C., Silveira, A., Silva, I., Ribeiro, C., Gestal, J., & Vasconcelos, C. (2011). Computerised information-gathering in patients with lupus: An initial evaluation of an electronic version of the short form 36 version 2. Lupus, 20(4), 402.Google Scholar
  55. 55.
    Richter, J. G., Becker, A., Koch, T., Nixdorf, M., Willers, R., Monser, R., et al. (2008). Self-assessments of patients via Tablet PC in routine patient care: Comparison with standardised paper questionnaires. Annals of the Rheumatic Diseases, 67(12), 1739–1741.PubMedCrossRefGoogle Scholar
  56. 56.
    Sage, J. M., Ali, A., Farrell, J., Huggins, J. L., Covert, K., Eskra, D., et al. (2012). Moving into the electronic age: Validation of rheumatology self-assessment questionnaires on tablet computers. Arthritis and Rheumatism, 64(S10), S1102.Google Scholar
  57. 57.
    Salaffi, F., Gasparini, S., Ciapetti, A., Gutierrez, M., & Grassi, W. (2013). Usability of an innovative and interactive electronic system for collection of patient-reported data in axial spondyloarthritis: Comparison with the traditional paper-administered format. Rheumatology, 52(11), 2062–2070.PubMedCrossRefGoogle Scholar
  58. 58.
    Schemmann, D., Rudolph, J., Haas, H., & Müller-Stromberg, J. (2013). Validation and patient acceptance of a touch tablet version of the iHOT-12 questionnaire. Arthroscopy, 29(12), E188.CrossRefGoogle Scholar
  59. 59.
    Sjöstrom, M., Stenlund, H., Johansson, S., Umefjord, G., & Samuelsson, E. (2012). Stress urinary incontinence and quality of life: A reliability study of a condition-specific instrument in paper and web-based versions. Neurology and Urodynamics, 31(8), 1242–1246.CrossRefGoogle Scholar
  60. 60.
    Twiss, J., McKenna, S., Graham, J. E., Swetz, K. M., Sloan, J., & Gomberg-Maitland, M. (2013). Assessing measurement equivalence of different forms of administration of the Cambridge Pulmonary Hypertension Outcome Review (CAMPHOR) using Rasch analysis. Value in Health, 16(7), A606.CrossRefGoogle Scholar
  61. 61.
    Varni, J. W., Limbers, C. A., Burwinkle, T. M., Bryant, W. P., & Wilson, D. P. (2008). The ePedsQL™ in Type 1 and Type 2 diabetes: Feasibility, reliability, and validity of the Pediatric Quality of Life Inventory™ Internet administration. Diabetes Care, 31(4), 672–677.PubMedCrossRefGoogle Scholar
  62. 62.
    Vinney, L. A., Grade, J. D., & Connor, N. P. (2012). Feasibility of using a handheld electronic device for the collection of patient reported outcomes data from children. Journal of Communication Disorders, 45(1), 12–19.PubMedCentralPubMedCrossRefGoogle Scholar
  63. 63.
    Young, N. L., Varni, J. W., Snider, L., McCormick, A., Sawatzky, B., Scott, M., et al. (2009). The Internet is valid and reliable for child-report: An example using the Activities scale for kids (ASK) and the Pediatric quality of life inventory (PedsQL). Journal of Clinical Epidemiology, 62(3), 314–320.PubMedCrossRefGoogle Scholar
  64. 64.
    Zimmerman, M., & Martinez, J. H. (2012). Web-based assessment of depression in patients treated in clinical practice: Reliability, validity and patient acceptance. Journal of Clinical Psychiatry, 73(3), 333–338.PubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Niloufar Campbell
    • 1
  • Faraz Ali
    • 2
  • Andrew Y. Finlay
    • 2
  • Sam S. Salek
    • 3
  1. 1.Centre for Socioeconomic ResearchSchool of Pharmacy and Pharmaceutical SciencesCardiffUK
  2. 2.Department of Dermatology and Wound Healing, School of MedicineCardiff UniversityCardiffUK
  3. 3.Department of PharmacyUniversity of Hertfordshire, Hatfield and Institute for Medicines DevelopmentCardiffUK

Personalised recommendations