Equivalence of electronic and paper-based patient-reported outcome measures
Electronic formats (ePROs) of paper-based patient-reported outcomes (PROs) should be validated before they can be reliably used. This review aimed to examine studies investigating measurement equivalence between ePROs and their paper originals to identify methodologies used and to determine the extent of such validation.
Three databases (OvidSP, Web of Science and PubMed) were searched using a set of keywords. Results were examined for compliance with inclusion criteria. Articles or abstracts that directly compared screen-based electronic versions of PROs with their validated paper-based originals, with regard to their measurement equivalence, were included. Publications were excluded if the only instruments reported were stand-alone visual analogue scales or interactive voice response formats. Papers published before 2007 were excluded, as a previous meta-analysis examined papers published before this time.
Fifty-five studies investigating 79 instruments met the inclusion criteria. 53 % of the 79 instruments studied were condition specific. Several instruments, such as the SF-36, were reported in more than one publication. The most frequently reported formats for ePROs were Web-based versions. In 78 % of the publications, there was evidence of equivalence or comparability between the two formats as judged by study authors. Of the 30 publications that provided preference data, 87 % found that overall participants preferred the electronic format.
When examining equivalence between paper and electronic versions of PROs, formats are usually judged by authors to be equivalent. Participants prefer electronic formats. This literature review gives encouragement to the further widespread development and use of ePROs.
KeywordsPatient-reported outcome measures PRO Electronic PROs Validation Equivalence
Conflict of interest
AYF is joint copyright owner of the DLQI.
- 1.Center for Drug Evaluation and Research (CDER). (2009). Guidance for industry—patient-reported outcome measures: Use in medical product development to support labeling claims. Silver Spring: Food and Drug Administration.Google Scholar
- 5.Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429.PubMedCrossRefGoogle Scholar
- 7.Arthur, C. (2012). The history of smartphones: timeline. The Guardian, 24 January 2012 (Online). http://www.theguardian.com/technology/2012/jan/24/smartphones-timeline. Accessed 2 May 2014.
- 8.McLellan, C. (2014). The History of Tablet Computers: A timeline (Online). http://www.zdnet.com/the-history-of-tablet-computers-a-timeline-7000026555/. Accessed 2 May 2014.
- 9.Salaffi, F., Gasparini, S., & Grassi, W. (2009). The use of computer touch-screen technology for the collection of the patient-reported outcome data in rheumatoid arthritis: Comparison to standardised patient questionnaires. Clinical and Experimental Rheumatology, 27(3), 459–468.PubMedGoogle Scholar
- 11.Wu, R. C., Thorpe, K., Math, M., Ross, H., Micevski, V., Marquez, C., Straus, S. E. (2009). Comparing administration of questionnaires via the Internet to pen-and-paper in patients with heart failure: Randomised controlled trial. Journal of Medical Internet Research (Online), 11(1). http://www.jmir.org/2009/1/e3/. Accessed 1 May 2014.
- 13.Ring, A. E., Cheong, K. A., Watkins, C. L., Meddis, D., Cella, D., & Harper, P. G. (2008). A randomised study of electronic diary versus paper and pencil collection of patient-reported outcomes in patients with non-small cell lung cancer. The Patient: Patient-Centered Outcomes Research, 1(2), 105–113.CrossRefGoogle Scholar
- 16.Ribeiro, C., Moreira, L., Silveira, A., Silva, I., Gestal, J., & Vasconcelos, C. (2010). Development and use of touch-screen computer-assisted self-interview in Portuguese patients with chronic immune disease: Evaluation of an electronic version of SF-36v2. Acta Reumatológica Portuguesa, 35(2), 208–214.PubMedGoogle Scholar
- 17.Matthew, A. G., Currie, K. L., Irvine, J., Ritvo, P., Mina, D. S., Jamnicky, L., Nam, R., Trachtenberg, J. (2007). Serial personal digital assistant data capture of health-related quality of life: A randomised controlled trial in a prostate cancer clinic. Health and Quality of Life Outcomes (Online), 5(38). http://www.hqlo.com/content/5/1/38. Accessed 30 April 2014.
- 18.Schefte, D. B., & Hetland, M. L. (2010). An open-source, self-explanatory touch screen in routine care. Validity of filling in the bath measures on ankylosing spondylitis disease activity index, function index, the health activity index, function index, the health assessment questionnaire and visual analogue scales in comparison with paper versions. Rheumatology, 49(1), 99–104.PubMedCrossRefGoogle Scholar
- 20.Swartz, R. J., Moor, C. D., Cook, K. F., Fouladi, R. T., Basen-Engquist, K., Eng, C., & Taylor, C. L. C. (2007). Mode effects in the center for epidemiological studies depression (CES-D) scale: Personal digital assistant versus paper and pencil administration. Quality of Life Research, 16(5), 803–813.PubMedCrossRefGoogle Scholar
- 26.Silveira, A., Gonçalves, J., Sequeira, T., Ribeiro, C., Lopes, C., Monteiro, E., & Pimentel, F. L. (2011). Computer-based quality-of-life monitoring in head and neck cancer patients: A validation model using the EORTC-QLQ C30 and EORTC-H&N35 Portuguese PC-software version. Acta Médica Portuguesa, 24(S2), 347–354.PubMedGoogle Scholar
- 30.Bruce, B., & Fries, J. F. (2011). Internet versus mailed administration of the health assessment questionnaire disability index. 63(10), pp. S1–S1 256.Google Scholar
- 31.Chang, Y-J., Chang, C-H., Peng, C-L., Wu, H-C., Lin, H-C., Wang, J-Y., Li, T-C., Yeh, Y-C., Liang, W-M. (2014). Measurement equivalence and feasibility of the EORTC QLQ-PR25: paper-and-pencil versus touch-screen administration. Health and Quality of Life Outcomes (Online), 12(23). http://www.hqlo.com/content/12/1/23. Accessed 29 April 2014.
- 32.Chen, H-L., Tien, S-W., Shih, C-C. (2011). Paper questionnaire versus Web questionnaire for clinical research impact using the short form of the UDI-6, HQ-7, PISQ-12. International Conference on Engineering and Business Management (EBM2011), 1–6, pp. 1293–1297.Google Scholar
- 36.Dalal, A. A., Nelson, L., Gilligan, T., McLeod, L., Lewis, S., & De Muro-Mercon, C. (2011). Evaluating patient-reported outcome measurement comparability between paper and alternative versions, using the lung function questionnaire as an example. Value in Health, 14(5), 712–720.PubMedCrossRefGoogle Scholar
- 38.Gudbergsen, H., Bartels, E. M., Krusager, P., Waehrens, E. E., Christensen, R., Danneskoild-Samsöe, B., Bliddal, H. (2011). Test-retest of computerised health status questionnaires frequently used in the monitoring of knee osteoarthritis: a randomised crossover trial. BMC Musculoskeletal Disorders (Online), 12(190). http://www.biomedcentral.com/1471-2474/12/190. Accessed 29 April 2014.
- 40.Heiberg, T., Kvien, T. K., Dale, Ø., Mowinckel, P., Aanerud, G. J., Songe-Møller, A. B., et al. (2007). Daily health status registration (patient diary) in patients with rheumatoid arthritis: A comparison between personal digital assistant and paper-pencil format. Arthritis Care & Research, 57(3), 454–460.CrossRefGoogle Scholar
- 41.Holländare, F., Andersson, G., Engström, I. (2010). A comparison of psychometric properties between Internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. Journal of Medical Internet Research (Online), 12(5). http://www.jmir.org/2010/5/e49/. Accessed 1 May 2014.
- 42.Hollen, P. J., Gralla, R. J., Stewart, J. A., Meharchand, J. M., Wierzbicki, R., & Leighl, N. (2013). Can a computerised format replace a paper form in PRO and HRQL evaluation? Psychometric testing of the computer-assisted LCSS instrument (eLCSS-QL). Supportive Care in Cancer, 21(1), 165–172.PubMedCrossRefGoogle Scholar
- 44.Lalanne, C., Herrmann, S., Armstrong, A. R., Cheung-Lung, C., Schwartz, Y., Chassany, O., & Duracinsky, M. (2013). Paper-based and electronic assessment of health-related quality of life specific HIV disease: A reliability study with the PROQOL-HIV questionnaire. Value in Health, 16(7), A362.CrossRefGoogle Scholar
- 47.Lee, E-H., Lee, Y.W., Lee, K-W., Kim, D.J., Kim, Y-S., Nam, M-S. (2013). Measurement equivalence of touch-screen computerised and paper-based diabetes-specific quality-of-life questionnaires. International Journal of Nursing Practice (Online). http://onlinelibrary.wiley.com/doi/10.1111/ijn.12184/abstract. Accessed 30 April 2014.
- 48.Minard, J. P., Thomas, N., Olajos-Clow, J., Juniper, E. F., Jiang, X., Jenkins, B., Taite, A. K., Turcotte, S., Lougheed, M. D. (2011)a. Validation of an electronic version of the Pediatric Caregiver’s Asthma Quality of Life Questionnaire (PACQLQ). American Journal of Respiratory and Critical Care Medicine (Online),183. http://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A1432. Accessed 30 April 2014.
- 49.Minard, J.P., Thomas, N., Olajos-Clow, J., Juniper, E. F., Jiang, X., Jenkins, B., Taite, A. K., Turcotte, S., Lougheed, M. D. (2011)b. Validation of an electronic version of the Mini Pediatric Asthmas Quality of Life Questionnaire (Mini PAQLQ). American Journal of Respiratory and Critical Care Medicine (Online). http://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A1433. Accessed 30 April 2014.
- 52.Parnell, B. A., Dunivan, G. C., Connolly, A., Jannelli, M. L., Wells, E. C., & Geller, E. J. (2011). Validation of web-based administration of the pelvic organ prolapse/urinary incontinence sexual function questionnaire (PISQ-12). International Urogynecology Journal, 22(3), 357–361.PubMedCrossRefGoogle Scholar
- 53.Raat, H., Mangunkusumo, R. T., Landgraf, J. M., Kloek, G., & Brug, J. (2007). Feasibility, reliability and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): Internet administration compared with the standard paper version. Quality of Life Research, 16(4), 675–685.PubMedCentralPubMedCrossRefGoogle Scholar
- 54.Ribeiro, C., Silveira, A., Silva, I., Ribeiro, C., Gestal, J., & Vasconcelos, C. (2011). Computerised information-gathering in patients with lupus: An initial evaluation of an electronic version of the short form 36 version 2. Lupus, 20(4), 402.Google Scholar
- 56.Sage, J. M., Ali, A., Farrell, J., Huggins, J. L., Covert, K., Eskra, D., et al. (2012). Moving into the electronic age: Validation of rheumatology self-assessment questionnaires on tablet computers. Arthritis and Rheumatism, 64(S10), S1102.Google Scholar
- 57.Salaffi, F., Gasparini, S., Ciapetti, A., Gutierrez, M., & Grassi, W. (2013). Usability of an innovative and interactive electronic system for collection of patient-reported data in axial spondyloarthritis: Comparison with the traditional paper-administered format. Rheumatology, 52(11), 2062–2070.PubMedCrossRefGoogle Scholar
- 60.Twiss, J., McKenna, S., Graham, J. E., Swetz, K. M., Sloan, J., & Gomberg-Maitland, M. (2013). Assessing measurement equivalence of different forms of administration of the Cambridge Pulmonary Hypertension Outcome Review (CAMPHOR) using Rasch analysis. Value in Health, 16(7), A606.CrossRefGoogle Scholar
- 63.Young, N. L., Varni, J. W., Snider, L., McCormick, A., Sawatzky, B., Scott, M., et al. (2009). The Internet is valid and reliable for child-report: An example using the Activities scale for kids (ASK) and the Pediatric quality of life inventory (PedsQL). Journal of Clinical Epidemiology, 62(3), 314–320.PubMedCrossRefGoogle Scholar