Large-scale clinical implementation of PROMIS computer adaptive testing with direct incorporation into the electronic medical record

  • M. O. Papuga
  • C. Dasilva
  • A. McIntyre
  • D. Mitten
  • S. Kates
  • J. F. BaumhauerEmail author
Original Article


The objective of this research was to assess the implementation of collecting patient-reported outcomes data in the outpatient clinics of a large academic hospital and identify potential barriers and solutions to such an implementation. Three PROMIS computer adaptive test instruments, (1) physical function, (2) pain interference, and (3) depression, were administered at 23,813 patient encounters using a novel software platform on tablet computers. The average time to complete was 3.50 ± 3.12 min, with a median time of 2.60 min. Registration times for new patients did not change significantly, 6.87 ± 3.34 to 7.19 ± 2.69 min. Registration times increased for follow-up (p = .007) from 2.94 ± 1.57 (p < .01) min to 3.32 ± 1.78 min. This is an effective implementation strategy to collect patient-reported outcomes and directly import the results into the electronic medical record in real time for use during the clinical visit.


orthopaedic outcomes patient-reported outcomes PROMIS physical function pain depression 


  1. Benedik E, Korousic SB, Simcic M et al (2014) Comparison of paper- and web-based dietary records: a pilot study. Annals of Nutrition & Metabolism 64, 156–166.CrossRefGoogle Scholar
  2. Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, and Ware JE, Jr (2014) Method of administration of PROMIS scales did not significantly impact score level, reliability, or validity. Journal of Clinical Epidemiology 67, 108–113.CrossRefGoogle Scholar
  3. Black N (2013) Patient reported outcome measures could help transform healthcare. BMJ 346, f167.CrossRefGoogle Scholar
  4. Broderick JE, Schneider S, Junghaenel DU, Schwartz JE and Stone AA (2013) Validity and reliability of patient-reported outcomes measurement information system instruments in osteoarthritis. Arthritis Care Res (Hoboken) 65, 1625–1633.Google Scholar
  5. Bushnell DM, Reilly MC, Galani C et al (2006) Validation of electronic data capture of the Irritable Bowel Syndrome-Quality of Life Measure, the Work Productivity and Activity Impairment Questionnaire for Irritable Bowel Syndrome and the EuroQol. Value Health 9, 98–105.CrossRefGoogle Scholar
  6. Cella D, Riley W, Stone A et al (2010) The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology 63, 1179–1194.CrossRefGoogle Scholar
  7. Cella D, Yount S, Rothrock N et al (2007) The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Medical Care 45, S3–S11.CrossRefGoogle Scholar
  8. Chen J, Ou L and Hollis SJ (2013) A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Services Research 13, 211.CrossRefGoogle Scholar
  9. Coons SJ, Eremenco S, Lundy JJ, O’Donohoe P, O’Gorman H and Malizia W (2015) Capturing patient-reported outcome (PRO) data electronically: the past, present, and promise of ePRO measurement in clinical trials. Patient 8, 301–309.CrossRefGoogle Scholar
  10. Cutler DM and Ghosh K (2012) The potential for cost savings through bundled episode payments. New England Journal of Medicine 366, 1075–1077.CrossRefGoogle Scholar
  11. Dale O and Hagen KB (2007) Despite technical problems personal digital assistants outperform pen and paper when collecting patient diary data. Journal of Clinical Epidemiology 60, 8–17.CrossRefGoogle Scholar
  12. Fries JF, Bruce B and Cella D (2005) The promise of PROMIS: using item response theory to improve assessment of patient-reported outcomes. Clinical and Experimental Rheumatology 23, S53–S57.Google Scholar
  13. Fries JF, Witter J, Rose M, Cella D, Khanna D and Morgan-DeWitt E (2014) Item response theory, computerized adaptive testing, and PROMIS: assessment of physical function. Journal of Rheumatology 41, 153–158.CrossRefGoogle Scholar
  14. Fritz F, Balhorn S, Riek M, Breil B and Dugas M (2012) Qualitative and quantitative evaluation of EHR-integrated mobile patient questionnaires regarding usability and cost-efficiency. International Journal of Medical Informatics 81, 303–313.CrossRefGoogle Scholar
  15. Greenwood MC, Hakim A J, Carson E and Doyle DV (2006) Touch-screen computer systems in the rheumatology clinic offer a reliable and user-friendly means of collecting quality-of-life and outcome data from patients with rheumatoid arthritis. Rheumatology 45, 66–71.CrossRefGoogle Scholar
  16. Gwaltney CJ, Shields AL and Shiffman S (2008) Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health 11, 322–333.CrossRefGoogle Scholar
  17. Hung M, Baumhauer JF, Brodsky JW et al (2014a) Psychometric comparison of the PROMIS physical function CAT with the FAAM and FFI for measuring patient-reported outcomes. Foot and Ankle International 35, 592–599.CrossRefGoogle Scholar
  18. Hung M, Baumhauer JF, Latt LD, Saltzman CL, SooHoo NF and Hunt KJ (2013) Validation of PROMIS (R) Physical Function computerized adaptive tests for orthopaedic foot and ankle outcome research. Clinical Orthopaedics and Related Research 471, 3466–3474.CrossRefGoogle Scholar
  19. Hung M, Clegg DO, Greene T and Saltzman CL (2011) Evaluation of the PROMIS physical function item bank in orthopaedic patients. Journal of Orthopaedic Research 29, 947–953.CrossRefGoogle Scholar
  20. Hung M, Franklin JD, Hon SD, Cheng C, Conrad J and Saltzman CL (2014b) Time for a paradigm shift with computerized adaptive testing of general physical function outcomes measurements. Foot and Ankle International 35, 1–7.CrossRefGoogle Scholar
  21. Hung M, Hon SD, Franklin JD et al (2014c) Psychometric properties of the PROMIS physical function item bank in patients with spinal disorders. Spine 39, 158–163.CrossRefGoogle Scholar
  22. Institue of Medicine (2001) Crossing the Quality Chasm: A New Health System for the 21st Century. Washington DC: National Academy of Science.Google Scholar
  23. Jacomb PA, Jorm AF, Korten AE, Christensen H and Henderson AS (2002) Predictors of refusal to participate: a longitudinal health survey of the elderly in Australia. BMC Public Health 2, 4.CrossRefGoogle Scholar
  24. Khanna D, Maranian P, Rothrock N et al (2012) Feasibility and construct validity of PROMIS and “legacy” instruments in an academic scleroderma clinic. Value Health 15, 128–134.CrossRefGoogle Scholar
  25. Krau SD (2015) Technology in nursing: the mandate for new implementation and adoption approaches. Nursing Clinics 50, 11–12.Google Scholar
  26. May CR (2015) Making sense of technology adoption in healthcare: meso-level considerations. BMC Medicine 13, 92.CrossRefGoogle Scholar
  27. Mihelic AH and Crimmins EM (1997) Loss to follow-up in a sample of Americans 70 years of age and older: the LSOA 1984-1990. Journals of Gerontology 52B, S37–S48.CrossRefGoogle Scholar
  28. Murphy J, Schwerin M, Eyerman J and Kennet J (2008) Barriers to survey participation among older adults in the national survey on drug use and health: the importance of establishing trust. Survey Practice 1(2), 1–6.Google Scholar
  29. Nelson EC, Eftimovska E, Lind C, Hager A, Wasson J and Lindblad S (2015) Patient reported outcome measures in practice. BMJ 350, g7818.CrossRefGoogle Scholar
  30. Norman GR, Sloan JA and Wyrwich KW (2003) Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Medical Care 41, 582–592.Google Scholar
  31. Norton MC, Breitner JC, Welsh KA.and Wyse BW (1994) Characteristics of nonresponders in a community survey of the elderly. Journal of the American Geriatrics Society 42, 1252–1256.CrossRefGoogle Scholar
  32. Overbeek CL, Nota SP, Jayakumar P, Hageman MG and Ring D (2015) The PROMIS physical function correlates with the QuickDASH in patients with upper extremity illness. Clinical Orthopaedics and Related Research 473, 311–317.CrossRefGoogle Scholar
  33. Papuga MO, Beck CA., Kates SL, Schwarz EM and Maloney MD (2014) Validation of GAITRite and PROMIS as high-throughput physical function outcome measures following ACL reconstruction. Journal of Orthopaedic Research 32, 793–801.CrossRefGoogle Scholar
  34. Reeve BB, Burke LB, Chiang YP et al (2007) Enhancing measurement in health outcomes research supported by Agencies within the US Department of Health and Human Services. Quality of Life Research 16(1), 175–186.CrossRefGoogle Scholar
  35. Richard Chin BYL (2008) Introduction to Clinical Trial Statistics. Burlington, MA: Elsevier.Google Scholar
  36. Sandholzer M, Deutsch T, Frese T and Winter A (2015) Predictors of students’ self-reported adoption of a smartphone application for medical education in general practice. BMC Medical Education 15, 91.CrossRefGoogle Scholar
  37. Schick-Makaroff K and Molzahn A (2015) Strategies to use tablet computers for collection of electronic patient-reported outcomes. Health Qual Life Outcomes 13, 2.CrossRefGoogle Scholar
  38. Segal C, Holve E, Sabharwal R (2013) Collecting and Using Patient-Reported Outcomes (PRO) for Comparative Effectiveness Research (CER) and Patient-Centered Outcomes Research (PCOR): Challenges and Opportunities. Issue Briefs and Reports.Google Scholar
  39. Selby JV, Beal AC and Frank L (2012) The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA 307, 1583–1584.CrossRefGoogle Scholar
  40. Sloan JA, Symond T, Vargas-Chanes D and Friendly B (2003) Practical guidelines for assessing the clinical significance of health-related quality of life changes within clinical trials. Therapeutic Innovation and Regulatory Science 37, 23–31.CrossRefGoogle Scholar
  41. Speerin R, Slater H, Li L et al (2014) Moving from evidence to practice: models of care for the prevention and management of musculoskeletal conditions. Best Practice & Research Clinical Rheumatology 28, 479–515.CrossRefGoogle Scholar
  42. Touvier M, Mejean C, Kesse-Guyot E et al (2010) Comparison between web-based and paper versions of a self-administered anthropometric questionnaire. European Journal of Epidemiology 25, 287–296.CrossRefGoogle Scholar
  43. Tyser AR, Beckmann J, Franklin JD et al (2014) Evaluation of the PROMIS physical function computer adaptive test in the upper extremity. The Journal of Hand Surgery 39, 2047–2051.CrossRefGoogle Scholar
  44. Uhlig CE, Seitz B, Eter N, Promesberger J. and Busse H (2014) Efficiencies of Internet-based digital and paper-based scientific surveys and the estimated costs and time for different-sized cohorts. PLoS ONE 9, e108441.CrossRefGoogle Scholar
  45. Velikova G, Wright EP, Smith AB et al (1999) Automated collection of quality-of-life data: a comparison of paper and computer touch-screen questionnaires. Journal of Clinical Oncology 17, 998–1007.CrossRefGoogle Scholar
  46. Wagner LI, Schink J, Bass M et al (2015) Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care. Cancer 121, 927–934.CrossRefGoogle Scholar
  47. Wei DH, Hawker GA, Jevsevar DS and Bozic KJ (2015) Improving value in musculoskeletal care delivery: AOA critical issues. Journal of Bone & Joint Surgery 97, 769–774.CrossRefGoogle Scholar
  48. Wu AW and Snyder C (2011) Getting ready for patient-reported outcomes measures (PROMs) in clinical practice. Healthcare Papers 11, 48–53.CrossRefGoogle Scholar
  49. Yoon S, Wilcox AB and Bakken S (2013) Comparisons among health behavior surveys: implications for the design of informatics infrastructures that support comparative effectiveness research. EGEMS 1, 1021.Google Scholar
  50. Zbrozek A, Hebert J, Gogates G et al (2013) Validation of electronic systems to collect patient-reported outcome (PRO) data-recommendations for clinical trial teams: report of the ISPOR ePRO systems validation good research practices task force. Value Health 16, 480–489.CrossRefGoogle Scholar

Copyright information

© The OR Society 2017

Authors and Affiliations

  • M. O. Papuga
    • 1
    • 2
    • 3
  • C. Dasilva
    • 1
    • 3
    • 4
  • A. McIntyre
    • 1
    • 3
  • D. Mitten
    • 3
    • 4
  • S. Kates
    • 1
    • 3
  • J. F. Baumhauer
    • 1
    • 3
    • 5
    Email author
  1. 1.Center for Musculoskeletal ResearchUniversity of RochesterRochesterUSA
  2. 2.Department of ResearchNew York Chiropractic CollegeSeneca FallsUSA
  3. 3.Department of Orthopaedics and RehabilitationUniversity of RochesterRochesterUSA
  4. 4.Center for Clinical InnovationUniversity of RochesterRochesterUSA
  5. 5.Department of Orthopaedics and RehabilitationUniversity of Rochester Medical CenterRochesterUSA

Personalised recommendations