Journal of General Internal Medicine

, Volume 34, Issue 12, pp 2818–2823 | Cite as

Electronic Health Record Phenotypes for Identifying Patients with Late-Stage Disease: a Method for Research and Clinical Application

  • Natalie C. ErnecoffEmail author
  • Kathryn L. Wessell
  • Laura C. Hanson
  • Adam M. Lee
  • Christopher M. Shea
  • Stacie B. Dusetzina
  • Morris Weinberger
  • Antonia V. Bennett
Original Research



Systematic identification of patients allows researchers and clinicians to test new models of care delivery. EHR phenotypes—structured algorithms based on clinical indicators from EHRs—can aid in such identification.


To develop EHR phenotypes to identify decedents with stage 4 solid-tumor cancer or stage 4–5 chronic kidney disease (CKD).


We developed two EHR phenotypes. Each phenotype included International Classification of Diseases (ICD)-9 and ICD-10 codes. We used natural language processing (NLP) to further specify stage 4 cancer, and lab values for CKD.


Decedents with cancer or CKD who had been admitted to an academic medical center in the last 6 months of life and died August 26, 2017–December 31, 2017.

Main Measure

We calculated positive predictive values (PPV), false discovery rates (FDR), false negative rates (FNR), and sensitivity. Phenotypes were validated by a comparison with manual chart review. We also compared the EHR phenotype results to those admitted to the oncology and nephrology inpatient services.

Key Results

The EHR phenotypes identified 271 decedents with cancer, of whom 186 had stage 4 disease; of 192 decedents with CKD, 89 had stage 4–5 disease. The EHR phenotype for stage 4 cancer had a PPV of 68.6%, FDR of 31.4%, FNR of 0.5%, and 99.5% sensitivity. The EHR phenotype for stage 4–5 CKD had a PPV of 46.4%, FDR of 53.7%, FNR of 0.0%, and 100% sensitivity.


EHR phenotypes efficiently identified patients who died with late-stage cancer or CKD. Future EHR phenotypes can prioritize specificity over sensitivity, and incorporate stratification of high- and low-palliative care need. EHR phenotypes are a promising method for identifying patients for research and clinical purposes, including equitable distribution of specialty palliative care.


Funding Information

This work was supported by the North Carolina Translational and Clinical Sciences Institute (NC TraCS), Clinical and Translational Science Awards (CTSA) grant no. UL1TR002489.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Supplementary material

11606_2019_5219_MOESM1_ESM.docx (61 kb)
ESM 1 (DOCX 60.8 kb)


  1. 1.
    Kavalieratos D, Corbelli J, Zhang D, Dionne-Odom JN, Ernecoff NC, Hanmer J, et al. Association Between Palliative Care and Patient and Caregiver Outcomes. JAMA. 2016 Nov 22;316(20):2104.CrossRefGoogle Scholar
  2. 2.
    Institute of Medicine (US). Dying in America: Improving Quality and Honoring Individual Preferences Near the End of Life : Health and Medicine Division. 2015.Google Scholar
  3. 3.
    O’hare AM, Song M-K, Tamura MK, Moss AH. Priorities in Palliative Care Research II Research Priorities for Palliative Care for Older Adults with Advanced Chronic Kidney Disease.Google Scholar
  4. 4.
    O’Hare AM, Armistead N, Schrag WLF, Diamond L, Moss AH. Patient-Centered Care: An Opportunity to Accomplish the “Three Aims” of the National Quality Strategy in the Medicare ESRD Program. Clin J Am Soc Nephrol. 2014 Dec 5;9(12):2189–94.CrossRefGoogle Scholar
  5. 5.
    Quill TE, Abernethy AP. Generalist plus Specialist Palliative Care — Creating a More Sustainable Model. N Engl J Med. 2013 Mar 28;368(13):1173–5.CrossRefGoogle Scholar
  6. 6.
    Kelley AS, Bollens-Lund E. Identifying the Population with Serious Illness: The “Denominator” Challenge. J Palliat Med. 2018 Mar 1;21(S2):S-7-S-16.CrossRefGoogle Scholar
  7. 7.
    Hanson LC, Gabriel S, Ernecoff NC, Wessell KL, Mitchell SL, Sachs G, et al. Triggered Palliative Care in Advanced Dementia: A Pilot Study. 2017.Google Scholar
  8. 8.
    Ernecoff NC, Wessell KL, Gabriel S, Carey TS, Hanson LC. A Novel Screening Method to Identify Late-Stage Dementia Patients for Palliative Care Research and Practice. J Pain Symptom Manage. 2018;Google Scholar
  9. 9.
    Xu J, Rasmussen L V., Shaw PL, Jiang G, Kiefer RC, Mo H, et al. Review and evaluation of electronic health records-driven phenotype algorithm authoring tools for clinical and translational research. J Am Med Informatics Assoc. 2015 Jul 29;22(6):ocv070CrossRefGoogle Scholar
  10. 10.
    Conway M, Berg RL, Carrell D, Denny JC, Kho AN, Kullo IJ, et al. Analyzing the Heterogeneity and Complexity of Electronic Health Record Oriented Phenotyping Algorithms.Google Scholar
  11. 11.
    Goebel JR, Ahluwalia SC, Chong K, Shreve ST, Goldzweig CL, Austin C, et al. Developing an Informatics Tool To Advance Supportive Care: The Veterans Health Care Administration Palliative Care National Clinical Template. J Palliat Med. 2014 Mar;17(3):266–73.CrossRefGoogle Scholar
  12. 12.
    Lilley EJ, Lindvall ÃC, Lillemoe KD, Tulsky JA, Wiener DC, Cooper Z, et al. Measuring Processes of Care in Palliative Surgery A Novel Approach Using Natural Language Processing. 2017;XX(Xx):1–3.Google Scholar
  13. 13.
    I2b2. Informatics for Integrating Biology and Bedside. Boston, MA;Google Scholar
  14. 14.
    Hanauer DA, Mei Q, Law J, Khanna R, Zheng K. Supporting information retrieval from electronic health records: A report of University of Michigan’s nine-year experience in developing and using the Electronic Medical Record Search Engine (EMERSE). J Biomed Inform. 2015 Jun;55:290–300.CrossRefGoogle Scholar
  15. 15.
    Levey AS, Eckardt K-U, Tsukamoto Y, Levin A, Coresh J, Rossert J, et al. Definition and classification of chronic kidney disease: A position statement from Kidney Disease: Improving Global Outcomes (KDIGO). Vol. 67, Kidney International. 2005.Google Scholar
  16. 16.
    Zeng QT, Goryachev S, Weiss S, Sordo M, Murphy SN, Lazarus R. BMC Medical Informatics and Decision Making Extracting principal diagnosis, co-morbidity and smoking status for asthma research: evaluation of a natural language processing system. 2006;Google Scholar
  17. 17.
    Kelley AS, Morrison RS. Palliative Care for the Seriously Ill. Campion EW, editor. N Engl J Med. 2015 Aug 20;373(8):747–55.Google Scholar
  18. 18.
    Aldridge MD, Hasselaar J, Garralda E, van der Eerden M, Stevenson D, McKendrick K, et al. Education, implementation, and policy barriers to greater integration of palliative care: A literature review. Palliat Med. 2016 Mar 24;30(3):224–39.CrossRefGoogle Scholar
  19. 19.
    Hanson LC, Collichio F, Bernard SA, Wood WA, Milowsky M, Burgess E, et al. Integrating Palliative and Oncology Care for Patients with Advanced Cancer: A Quality Improvement Intervention. J Palliat Med. 2017 Dec 1;20(12):1366–71.CrossRefGoogle Scholar
  20. 20.
    Shiekh Sroujieh L, Monroy D, Warren E. Using Electronic Health Record (EHR) Best Practice Alert (BPA) to Improve RBC Transfusion Practices and Adherence to the Guidelines. Chest. 2016 Oct 1;150(4):599A.CrossRefGoogle Scholar
  21. 21.
    American Academy of Family Physicians. A, Stoeckle J, Diaz V, Valko G, Arenson C. Family practice management. Vol. 24, Family Practice Management. American Academy of Family Physicians; 2017. 30–33 p.Google Scholar
  22. 22.
    Lofthus DM, Gadgil JY, De Lemos JA. Overriding Concerns: The Role of Electronic Medical Record-Based Best Practice Alerts in Reducing Unnecessary Laboratory Testing. 2014;Google Scholar
  23. 23.
    Kukhareva P, Staes C, Noonan KW, Mueller HL, Warner P, Shields DE, et al. Single-reviewer electronic phenotyping validation in operational settings: Comparison of strategies and recommendations. J Biomed Inform. 2017 Feb;66:1–10.CrossRefGoogle Scholar
  24. 24.
    Drawz PE, Archdeacon P, Mcdonald CJ, Powe NR, Smith KA, Norton J, et al. Special Feature CKD as a Model for Improving Chronic Disease Care through Electronic Health Records. Clin J Am Soc Nephrol. 2015;10:1488–99.CrossRefGoogle Scholar
  25. 25.
    Navar AM, Peterson ED, Steen DL, Wojdyla DM, Sanchez RJ, Khan I, et al. Evaluation of Mortality Data from the Social Security Administration Death Master File for Clinical Research. JAMA Cardiol. 2019;Google Scholar
  26. 26.
    Adelson K, Paris J, Horton JR, Hernandez-Tellez L, Ricks D, Morrison RS, et al. Standardized Criteria for Palliative Care Consultation on a Solid Tumor Oncology Service Reduces Downstream Health Care Use. J Oncol Pract. 2017 May 17;13(5):e431–40.CrossRefGoogle Scholar
  27. 27.
    Weissman DE, Meier DE. Special Report Identifying Patients in Need of a Palliative Care Assessment in the Hospital Setting A Consensus Report from the Center to Advance Palliative Care.Google Scholar
  28. 28.
    Avati A, Jung K, Harman S, Downing L, Ng A, Shah NH. Improving palliative care with deep learning. In: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE; 2017. p. 311–6.Google Scholar
  29. 29.
    Sahni N, Simon G, Arora R. Development and Validation of Machine Learning Models for Prediction of 1-Year Mortality Utilizing Electronic Medical Record Data Available at the End of Hospitalization in Multicondition Patients: a Proof-of-Concept Study. J Gen Intern Med. 2018;33(6):921–9.CrossRefGoogle Scholar
  30. 30.
    Cardona-Morrell M, Hillman K. Development of a tool for defining and identifying the dying patient in hospital: Criteria for Screening and Triaging to Appropriate aLternative care (CriSTAL).Google Scholar

Copyright information

© Society of General Internal Medicine 2019

Authors and Affiliations

  • Natalie C. Ernecoff
    • 1
    Email author
  • Kathryn L. Wessell
    • 2
  • Laura C. Hanson
    • 2
    • 3
  • Adam M. Lee
    • 4
  • Christopher M. Shea
    • 5
  • Stacie B. Dusetzina
    • 6
  • Morris Weinberger
    • 5
  • Antonia V. Bennett
    • 5
  1. 1.Division of General Internal MedicineUniversity of Pittsburgh School of MedicinePittsburghUSA
  2. 2.Sheps Center for Health Services ResearchUniversity of North Carolina-Chapel HillChapel HillUSA
  3. 3.Division of Geriatric Medicine & Palliative Care ProgramUniversity of North Carolina-Chapel HillChapel HillUSA
  4. 4.North Carolina Translational and Clinical Sciences InstituteUniversity of North Carolina-Chapel HillChapel HillUSA
  5. 5.Department of Health Policy and Management, Gillings School of Global Public HealthUniversity of North Carolina-Chapel HillChapel HillUSA
  6. 6.Department of Health PolicyVanderbilt University Medical CenterNashvilleUSA

Personalised recommendations