Skip to main content

Large-scale clinical implementation of PROMIS computer adaptive testing with direct incorporation into the electronic medical record

Health Systems (2017)Cite this article

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

The objective of this research was to assess the implementation of collecting patient-reported outcomes data in the outpatient clinics of a large academic hospital and identify potential barriers and solutions to such an implementation. Three PROMIS computer adaptive test instruments, (1) physical function, (2) pain interference, and (3) depression, were administered at 23,813 patient encounters using a novel software platform on tablet computers. The average time to complete was 3.50 ± 3.12 min, with a median time of 2.60 min. Registration times for new patients did not change significantly, 6.87 ± 3.34 to 7.19 ± 2.69 min. Registration times increased for follow-up (p = .007) from 2.94 ± 1.57 (p < .01) min to 3.32 ± 1.78 min. This is an effective implementation strategy to collect patient-reported outcomes and directly import the results into the electronic medical record in real time for use during the clinical visit.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Figure 1
Figure 2
Figure 3

References

  • Benedik E, Korousic SB, Simcic M et al (2014) Comparison of paper- and web-based dietary records: a pilot study. Annals of Nutrition & Metabolism 64, 156–166.

    Article  Google Scholar 

  • Bjorner JB, Rose M, Gandek B, Stone AA, Junghaenel DU, and Ware JE, Jr (2014) Method of administration of PROMIS scales did not significantly impact score level, reliability, or validity. Journal of Clinical Epidemiology 67, 108–113.

    Article  Google Scholar 

  • Black N (2013) Patient reported outcome measures could help transform healthcare. BMJ 346, f167.

    Article  Google Scholar 

  • Broderick JE, Schneider S, Junghaenel DU, Schwartz JE and Stone AA (2013) Validity and reliability of patient-reported outcomes measurement information system instruments in osteoarthritis. Arthritis Care Res (Hoboken) 65, 1625–1633.

    Google Scholar 

  • Bushnell DM, Reilly MC, Galani C et al (2006) Validation of electronic data capture of the Irritable Bowel Syndrome-Quality of Life Measure, the Work Productivity and Activity Impairment Questionnaire for Irritable Bowel Syndrome and the EuroQol. Value Health 9, 98–105.

    Article  Google Scholar 

  • Cella D, Riley W, Stone A et al (2010) The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology 63, 1179–1194.

    Article  Google Scholar 

  • Cella D, Yount S, Rothrock N et al (2007) The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Medical Care 45, S3–S11.

    Article  Google Scholar 

  • Chen J, Ou L and Hollis SJ (2013) A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Services Research 13, 211.

    Article  Google Scholar 

  • Coons SJ, Eremenco S, Lundy JJ, O’Donohoe P, O’Gorman H and Malizia W (2015) Capturing patient-reported outcome (PRO) data electronically: the past, present, and promise of ePRO measurement in clinical trials. Patient 8, 301–309.

    Article  Google Scholar 

  • Cutler DM and Ghosh K (2012) The potential for cost savings through bundled episode payments. New England Journal of Medicine 366, 1075–1077.

    Article  Google Scholar 

  • Dale O and Hagen KB (2007) Despite technical problems personal digital assistants outperform pen and paper when collecting patient diary data. Journal of Clinical Epidemiology 60, 8–17.

    Article  Google Scholar 

  • Fries JF, Bruce B and Cella D (2005) The promise of PROMIS: using item response theory to improve assessment of patient-reported outcomes. Clinical and Experimental Rheumatology 23, S53–S57.

    Google Scholar 

  • Fries JF, Witter J, Rose M, Cella D, Khanna D and Morgan-DeWitt E (2014) Item response theory, computerized adaptive testing, and PROMIS: assessment of physical function. Journal of Rheumatology 41, 153–158.

    Article  Google Scholar 

  • Fritz F, Balhorn S, Riek M, Breil B and Dugas M (2012) Qualitative and quantitative evaluation of EHR-integrated mobile patient questionnaires regarding usability and cost-efficiency. International Journal of Medical Informatics 81, 303–313.

    Article  Google Scholar 

  • Greenwood MC, Hakim A J, Carson E and Doyle DV (2006) Touch-screen computer systems in the rheumatology clinic offer a reliable and user-friendly means of collecting quality-of-life and outcome data from patients with rheumatoid arthritis. Rheumatology 45, 66–71.

    Article  Google Scholar 

  • Gwaltney CJ, Shields AL and Shiffman S (2008) Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Value Health 11, 322–333.

    Article  Google Scholar 

  • Hung M, Baumhauer JF, Brodsky JW et al (2014a) Psychometric comparison of the PROMIS physical function CAT with the FAAM and FFI for measuring patient-reported outcomes. Foot and Ankle International 35, 592–599.

    Article  Google Scholar 

  • Hung M, Baumhauer JF, Latt LD, Saltzman CL, SooHoo NF and Hunt KJ (2013) Validation of PROMIS (R) Physical Function computerized adaptive tests for orthopaedic foot and ankle outcome research. Clinical Orthopaedics and Related Research 471, 3466–3474.

    Article  Google Scholar 

  • Hung M, Clegg DO, Greene T and Saltzman CL (2011) Evaluation of the PROMIS physical function item bank in orthopaedic patients. Journal of Orthopaedic Research 29, 947–953.

    Article  Google Scholar 

  • Hung M, Franklin JD, Hon SD, Cheng C, Conrad J and Saltzman CL (2014b) Time for a paradigm shift with computerized adaptive testing of general physical function outcomes measurements. Foot and Ankle International 35, 1–7.

    Article  Google Scholar 

  • Hung M, Hon SD, Franklin JD et al (2014c) Psychometric properties of the PROMIS physical function item bank in patients with spinal disorders. Spine 39, 158–163.

    Article  Google Scholar 

  • Institue of Medicine (2001) Crossing the Quality Chasm: A New Health System for the 21st Century. Washington DC: National Academy of Science.

    Google Scholar 

  • Jacomb PA, Jorm AF, Korten AE, Christensen H and Henderson AS (2002) Predictors of refusal to participate: a longitudinal health survey of the elderly in Australia. BMC Public Health 2, 4.

    Article  Google Scholar 

  • Khanna D, Maranian P, Rothrock N et al (2012) Feasibility and construct validity of PROMIS and “legacy” instruments in an academic scleroderma clinic. Value Health 15, 128–134.

    Article  Google Scholar 

  • Krau SD (2015) Technology in nursing: the mandate for new implementation and adoption approaches. Nursing Clinics 50, 11–12.

    Google Scholar 

  • May CR (2015) Making sense of technology adoption in healthcare: meso-level considerations. BMC Medicine 13, 92.

    Article  Google Scholar 

  • Mihelic AH and Crimmins EM (1997) Loss to follow-up in a sample of Americans 70 years of age and older: the LSOA 1984-1990. Journals of Gerontology 52B, S37–S48.

    Article  Google Scholar 

  • Murphy J, Schwerin M, Eyerman J and Kennet J (2008) Barriers to survey participation among older adults in the national survey on drug use and health: the importance of establishing trust. Survey Practice 1(2), 1–6.

    Google Scholar 

  • Nelson EC, Eftimovska E, Lind C, Hager A, Wasson J and Lindblad S (2015) Patient reported outcome measures in practice. BMJ 350, g7818.

    Article  Google Scholar 

  • Norman GR, Sloan JA and Wyrwich KW (2003) Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Medical Care 41, 582–592.

    Google Scholar 

  • Norton MC, Breitner JC, Welsh KA.and Wyse BW (1994) Characteristics of nonresponders in a community survey of the elderly. Journal of the American Geriatrics Society 42, 1252–1256.

    Article  Google Scholar 

  • Overbeek CL, Nota SP, Jayakumar P, Hageman MG and Ring D (2015) The PROMIS physical function correlates with the QuickDASH in patients with upper extremity illness. Clinical Orthopaedics and Related Research 473, 311–317.

    Article  Google Scholar 

  • Papuga MO, Beck CA., Kates SL, Schwarz EM and Maloney MD (2014) Validation of GAITRite and PROMIS as high-throughput physical function outcome measures following ACL reconstruction. Journal of Orthopaedic Research 32, 793–801.

    Article  Google Scholar 

  • Reeve BB, Burke LB, Chiang YP et al (2007) Enhancing measurement in health outcomes research supported by Agencies within the US Department of Health and Human Services. Quality of Life Research 16(1), 175–186.

    Article  Google Scholar 

  • Richard Chin BYL (2008) Introduction to Clinical Trial Statistics. Burlington, MA: Elsevier.

    Google Scholar 

  • Sandholzer M, Deutsch T, Frese T and Winter A (2015) Predictors of students’ self-reported adoption of a smartphone application for medical education in general practice. BMC Medical Education 15, 91.

    Article  Google Scholar 

  • Schick-Makaroff K and Molzahn A (2015) Strategies to use tablet computers for collection of electronic patient-reported outcomes. Health Qual Life Outcomes 13, 2.

    Article  Google Scholar 

  • Segal C, Holve E, Sabharwal R (2013) Collecting and Using Patient-Reported Outcomes (PRO) for Comparative Effectiveness Research (CER) and Patient-Centered Outcomes Research (PCOR): Challenges and Opportunities. Issue Briefs and Reports.

  • Selby JV, Beal AC and Frank L (2012) The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA 307, 1583–1584.

    Article  Google Scholar 

  • Sloan JA, Symond T, Vargas-Chanes D and Friendly B (2003) Practical guidelines for assessing the clinical significance of health-related quality of life changes within clinical trials. Therapeutic Innovation and Regulatory Science 37, 23–31.

    Article  Google Scholar 

  • Speerin R, Slater H, Li L et al (2014) Moving from evidence to practice: models of care for the prevention and management of musculoskeletal conditions. Best Practice & Research Clinical Rheumatology 28, 479–515.

    Article  Google Scholar 

  • Touvier M, Mejean C, Kesse-Guyot E et al (2010) Comparison between web-based and paper versions of a self-administered anthropometric questionnaire. European Journal of Epidemiology 25, 287–296.

    Article  Google Scholar 

  • Tyser AR, Beckmann J, Franklin JD et al (2014) Evaluation of the PROMIS physical function computer adaptive test in the upper extremity. The Journal of Hand Surgery 39, 2047–2051.

    Article  Google Scholar 

  • Uhlig CE, Seitz B, Eter N, Promesberger J. and Busse H (2014) Efficiencies of Internet-based digital and paper-based scientific surveys and the estimated costs and time for different-sized cohorts. PLoS ONE 9, e108441.

    Article  Google Scholar 

  • Velikova G, Wright EP, Smith AB et al (1999) Automated collection of quality-of-life data: a comparison of paper and computer touch-screen questionnaires. Journal of Clinical Oncology 17, 998–1007.

    Article  Google Scholar 

  • Wagner LI, Schink J, Bass M et al (2015) Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care. Cancer 121, 927–934.

    Article  Google Scholar 

  • Wei DH, Hawker GA, Jevsevar DS and Bozic KJ (2015) Improving value in musculoskeletal care delivery: AOA critical issues. Journal of Bone & Joint Surgery 97, 769–774.

    Article  Google Scholar 

  • Wu AW and Snyder C (2011) Getting ready for patient-reported outcomes measures (PROMs) in clinical practice. Healthcare Papers 11, 48–53.

    Article  Google Scholar 

  • Yoon S, Wilcox AB and Bakken S (2013) Comparisons among health behavior surveys: implications for the design of informatics infrastructures that support comparative effectiveness research. EGEMS 1, 1021.

    Google Scholar 

  • Zbrozek A, Hebert J, Gogates G et al (2013) Validation of electronic systems to collect patient-reported outcome (PRO) data-recommendations for clinical trial teams: report of the ISPOR ePRO systems validation good research practices task force. Value Health 16, 480–489.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Musculoskeletal Research, University of Rochester, Rochester, NY, USA

    M. O. Papuga, C. Dasilva, A. McIntyre, S. Kates & J. F. Baumhauer

  2. Department of Research, New York Chiropractic College, Seneca Falls, NY, USA

    M. O. Papuga

  3. Department of Orthopaedics and Rehabilitation, University of Rochester, Rochester, NY, USA

    M. O. Papuga, C. Dasilva, A. McIntyre, D. Mitten, S. Kates & J. F. Baumhauer

  4. Center for Clinical Innovation, University of Rochester, Rochester, NY, USA

    C. Dasilva & D. Mitten

  5. Department of Orthopaedics and Rehabilitation, University of Rochester Medical Center, 601 Elmwood AvenueBox 665, Rochester, NY, 14642, USA

    J. F. Baumhauer

Authors
  1. M. O. Papuga
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. C. Dasilva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. McIntyre
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. D. Mitten
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. S. Kates
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. J. F. Baumhauer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. F. Baumhauer.

Appendix 1: Patient-reported outcomes measurement information system

Appendix 1: Patient-reported outcomes measurement information system

Frequently asked questions

PROMIS

Q. What is the purpose of the PROMIS health assessment?

A. The health assessment gives your provider a clear understanding/snapshot of how you are feeling today and helps them work with you in developing a treatment plan.

Q. Who is requesting this information and why?

A. Your physician is requesting this information and it is Standard of Care (like taking blood pressure).

Q. Where do my answers go?

A. Your answers are electronically uploaded into your medical record and are ready for today’s visit.

Q. Can I see my answers in “My Chart”?

A. No, it is not linked to My Chart; however, your provider can show you YOUR information. Only your provider and other medical professionals with access to your medical record can view your answers.

Q. Am I answering this about my injury/surgery or my health in general?

A. Your general health, which includes what brought you here today. Your answers should be related to how you feel today, a “snapshot” in time.

Q. Do I have to do this every time I come? Why?

Yes. The Health Assessment is Standard of Care and updates your provider on how you are doing today.

Q. Are the questions the same every time?

A. The questions are generated based on the answers that you give. If your prior answer is the same, the next question will be the same. If you answer differently, it triggers a new question.

Q. How long does the assessment take?

A. It only takes a few minutes to complete the assessment.

Q. Why do I have to answer questions that don’t seem to apply to me?

A. It is important for your provider to know if you are able to do the daily activities you normally do or enjoy doing.

Q. Can I decline?

A. Yes, you can opt-out if you wish; however, this is your opportunity to update your provider on how you are feeling today. This information can be used as part of your discussions with your doctor to develop your treatment plan and to monitor your progress.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Papuga, M.O., Dasilva, C., McIntyre, A. et al. Large-scale clinical implementation of PROMIS computer adaptive testing with direct incorporation into the electronic medical record. Health Syst (2017). https://doi.org/10.1057/s41306-016-0016-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1057/s41306-016-0016-1

Keywords

  • orthopaedic outcomes
  • patient-reported outcomes
  • PROMIS
  • physical function
  • pain
  • depression