Skip to main content

Advertisement

SpringerLink
Log in

Comparative assessment of manual chart review and ICD claims data in evaluating immunotherapy-related adverse events

  • Original Article
  • Published:
Cancer Immunology, Immunotherapy Aims and scope Submit manuscript

Abstract

Background

The aim of this retrospective study was to demonstrate that irAEs, specifically gastrointestinal and pulmonary, examined through International Classification of Disease (ICD) data leads to underrepresentation of true irAEs and overrepresentation of false irAEs, thereby concluding that ICD claims data are a poor approach to electronic health record (EHR) data mining for irAEs in immunotherapy clinical research.

Methods

This retrospective analysis was conducted in 1,063 cancer patients who received ICIs between 2011 and 2017. We identified irAEs by manual review of medical records to determine the incidence of each of our endpoints, namely colitis, hepatitis, pneumonitis, other irAE, or no irAE. We then performed a secondary analysis utilizing ICD claims data alone using a broad range of symptom and disease-specific ICD codes representative of irAEs.

Results

16% (n = 174/1,063) of the total study population was initially found to have either pneumonitis 3% (n = 37), colitis 7% (n = 81) or hepatitis 5% (n = 56) on manual review. Of these patients, 46% (n = 80/174) did not have ICD code evidence in the EHR reflecting their irAE. Of the total patients not found to have any irAEs during manual review, 61% (n = 459/748) of patients had ICD codes suggestive of possible irAE, yet were not identified as having an irAE during manual review.

Discussion

Examining gastrointestinal and pulmonary irAEs through the International Classification of Disease (ICD) data leads to underrepresentation of true irAEs and overrepresentation of false irAEs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Code availability

All analyses were conducted using the SAS system, version 9.4 (SAS Institute Inc., Cary, NC). No custom codes were used.

Data availability

In accordance with local and/or U.S. Government laws and regulations, any materials and de-identified data that are reasonably requested by others for the purposes of academic research will be made available in a timely fashion.

Abbreviations

ALT:

Alanine aminotransferases

AST:

Aspartate aminotransferases

DILI:

Drug-induced liver injury

EHR:

Electronic health record

FDA:

Food and drug administration

GI:

Gastrointestinal tract

ICIs:

Immune checkpoint inhibitors

ICD:

International classification of disease

irAEs:

Immune-related adverse events

NLP:

Natural language processing

SRS:

Spontaneous reporting systems

References

  1. Hughes MS, Zheng H, Zubiri L et al (2019) Colitis after checkpoint blockade: a retrospective cohort study of melanoma patients requiring admission for symptom control. Cancer Med 8(11):4986–4999. https://doi.org/10.1002/cam4.2397

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Yu Y, Ruddy K, Mansfield A et al (2020) Detecting and filtering immune-related adverse events signal based on text mining and observational health data sciences and informatics common data model: framework development study. JMIR Med Inform 8(6):e17353. https://doi.org/10.2196/17353

    Article  PubMed  PubMed Central  Google Scholar 

  3. Postow MA, Sidlow R, Hellmann MD (2018) Immune-related adverse events associated with immune checkpoint blockade. N Engl J Med 378(2):158–168. https://doi.org/10.1056/NEJMra1703481

    Article  CAS  PubMed  Google Scholar 

  4. Wang DY, Salem JE, Cohen JV et al (2018) Fatal toxic effects associated with immune checkpoint inhibitors: a systematic review and meta-analysis. JAMA Oncol 4(12):1721–1728. https://doi.org/10.1001/jamaoncol.2018.3923

    Article  PubMed  PubMed Central  Google Scholar 

  5. Som A, Mandaliya R, Alsaadi D et al (2019) Immune checkpoint inhibitor-induced colitis: a comprehensive review. World J Clin Cases 7(4):405–418. https://doi.org/10.12998/wjcc.v7.i4.405

    Article  PubMed  PubMed Central  Google Scholar 

  6. Xing P, Zhang F, Wang G et al (2019) Incidence rates of immune-related adverse events and their correlation with response in advanced solid tumours treated with NIVO or NIVO+IPI: a systematic review and meta-analysis. J Immunotherapy Cancer. https://doi.org/10.1186/s40425-019-0779-6

    Article  Google Scholar 

  7. Osorio JC, Ni A, Chaft JE et al (2017) Antibody-mediated thyroid dysfunction during T-cell checkpoint blockade in patients with non-small-cell lung cancer. Ann Oncol 28(3):583–589. https://doi.org/10.1093/annonc/mdw640

    Article  CAS  PubMed  Google Scholar 

  8. Abu-Sbeih H, Ali FS, Naqash AR et al (2019) Resumption of immune checkpoint inhibitor therapy after immune-mediated colitis. J Clin Oncol 37(30):2738–2745. https://doi.org/10.1200/JCO.19.00320

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Michot J, Lappara A, Pavec J, et al. The 2016–2019 ImmunoTOX assessment board report collaborative management of immune-related adverse events, an observational clinical study. European Society for Medical Oncology. 2019, Sept 29. Barcelona, Spain.

  10. Naqash AR, Ricciuti B, Owen DH et al (2020) Outcomes associated with immune-related adverse events in metastatic non-small cell lung cancer treated with nivolumab: a pooled exploratory analysis from a global cohort. Cancer Immunol Immunother 69(7):1177–1187. https://doi.org/10.1007/s00262-020-02536-5

    Article  CAS  PubMed  Google Scholar 

  11. Wang Y, Abu-Sbeih H, Mao E et al (2018) Immune-checkpoint inhibitor-induced diarrhea and colitis in patients with advanced malignancies retrospective: review at MD Anderson. J immunotherapy cancer. https://doi.org/10.1186/s40425-018-0346-6

    Article  Google Scholar 

  12. Dupont R, Bérard E, Puisset F et al (2019) The prognostic impact of immune-related adverse events during anti-PD1 treatment in melanoma and non-small-cell lung cancer: a real-life retrospective study. Oncoimmunology 9(1):1682383. https://doi.org/10.1080/2162402X.2019.1682383

    Article  PubMed  PubMed Central  Google Scholar 

  13. Reddy HG, Schneider BJ, Tai AW (2018) Immune checkpoint inhibitor-associated colitis and hepatitis. Clin Transl Gastroenterol 9(9):180. https://doi.org/10.1038/s41424-018-0049-9

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Chuzi S, Tavora F, Cruz M et al (2017) (2017) Clinical features, diagnostic challenges, and management strategies in checkpoint inhibitor-related pneumonitis. Cancer Manag Res 9:207–213. https://doi.org/10.2147/CMAR.S136818

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Jagannatha A, Liu F, Liu W, Yu H (2019) Overview of the first natural language processing challenge for extracting medication, indication, and adverse drug events from electronic health record notes (MADE 1.0). Drug Saf 42(1):99–111. https://doi.org/10.1007/s40264-018-0762-z

    Article  PubMed  PubMed Central  Google Scholar 

  16. Hohl CM, Karpov A, Reddekopp L, Doyle-Waters M, Stausberg J (2014) ICD-10 codes used to identify adverse drug events in administrative data: a systematic review. J Am Med Inform Assoc 21(3):547–557. https://doi.org/10.1136/amiajnl-2013-002116

    Article  PubMed  Google Scholar 

  17. World Health Organization (2006) The safety of medicines in public health programmes: pharmacovigilance: an essential tool. WHO Publications, Geneva, Switzerland

    Google Scholar 

  18. Cox AR, Anton C, Goh CH, Easter M, Langford NJ, Ferner RE (2001) Adverse drug reactions in patients admitted to hospital identified by discharge ICD-10 codes and by spontaneous reports. Br J Clin Pharmacol 52(3):337–339. https://doi.org/10.1046/j.0306-5251.2001.01454.x

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Coffman C. Using Free-Text Searching and ICD-10 Codes to identify prospective patients with drug-induced liver injury. College of Health and Human Services Eastern Michigan University. Thesis Committee. December 14, 2015

  20. Hohl CM, Kuramoto L, Yu E et al (2013) Evaluating adverse drug event reporting in administrative data from emergency departments: a validation study. BMC Health Serv Res 13:473. https://doi.org/10.1186/1472-6963-13-473

    Article  PubMed  PubMed Central  Google Scholar 

  21. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG (2009) Research electronic data capture (REDCap): a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 42(2):377–381. https://doi.org/10.1016/j.jbi.2008.08.010

    Article  PubMed  Google Scholar 

  22. Owen DH, Wei L, Bertino EM, Edd T, Villalona-Calero MA, He K, Shields PG, Carbone DP, Otterson GA (2018) Incidence, risk factors, and effect on survival of immune-related adverse events in patients with non-small-cell lung cancer. Clin Lung Cancer 19(6):e893–e900. https://doi.org/10.1016/j.cllc.2018.08.008

    Article  PubMed  PubMed Central  Google Scholar 

  23. Cui P, Huang D, Wu Z, Tao H, Zhang S, Ma J, Liu Z, Wang J, Huang Z, Chen S, Zheng X, Hu Y (2020) Association of immune-related pneumonitis with the efficacy of PD-1/PD-L1 inhibitors in non-small cell lung cancer. Ther Adv Med Oncol 12:1–10. https://doi.org/10.1177/1758835920922033

    Article  CAS  Google Scholar 

  24. Hsiehchen D, Watters MK, Lu R, Xie Y, Gerber DE (2019) Variation in the assessment of immune-related adverse event occurrence, grade, and timing in patients receiving immune checkpoint inhibitors. JAMA Netw Open. https://doi.org/10.1001/jamanetworkopen.2019.11519

    Article  PubMed  PubMed Central  Google Scholar 

  25. Hougland P, Nebeker J, Pickard S, et al. Using ICD-9-CM Codes in Hospital Claims Data to Detect Adverse Events in Patient Safety Surveillance. In: Henriksen K, Battles JB, Keyes MA, et al., editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 1: Assessment). Rockville (MD): Agency for Healthcare Research and Quality; 2008 Aug. Available from: https://www.ncbi.nlm.nih.gov/books/NBK43647/

  26. Nadkarni PM (2010) Drug safety surveillance using de-identified EMR and claims data: issues and challenges. J Am Med Inform Assoc 17(6):671–674. https://doi.org/10.1136/jamia.2010.008607

    Article  PubMed  PubMed Central  Google Scholar 

  27. Harpaz R, Callahan A, Tamang S et al (2014) Text mining for adverse drug events: the promise, challenges, and state of the art. Drug Saf 37(10):777–790. https://doi.org/10.1007/s40264-014-0218-z

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  28. Maguire FB, Morris CR, Parikh-Patel A et al (2019) A text-mining approach to obtain detailed treatment information from free-text fields in population-based cancer registries: a study of non-small cell lung cancer in California. PLoS ONE 14(2):e0212454. https://doi.org/10.1371/journal.pone.0212454

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Research support was provided by the REDCap project and The Ohio State University Center for Clinical and Translational Science grant support (National Center for Advancing Translational Sciences, Grant UL1TR002733). Dr. Owen and Dr. Presley are Paul Calabresi Scholars supported by the OSU K12 Training Grant for Clinical Faculty Investigators (K12 CA133250).

Funding

This study was supported by the National Institutes of Health P30CA016058.

Author information

Authors and Affiliations

  1. Department of Internal Medicine, The Ohio State University, A450B Starling Loving Hall Columbus, Columbus, OH, 43210, USA

    Andrew Nashed, Andrew Johns & Mingjia Li

  2. Department of Biomedical Informatics and Center for Biostatistics, The Ohio State University, A450B Starling Loving Hall Columbus, Columbus, OH, 43210, USA

    Shijun Zhang, Chien-Wei Chiang, M. Zitu & Lang Li

  3. Division of Medical Oncology, The Ohio State University, A450B Starling Loving Hall ColumbusA450B Starling Loving Hall Columbus, Columbus, OH, 43210, USA

    Gregory A. Otterson, Carolyn J. Presley, Kari Kendra, Sandip H. Patel, Madison Grogan, Gabrielle Lopez & Dwight H. Owen

Authors
  1. Andrew Nashed
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shijun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chien-Wei Chiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. Zitu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gregory A. Otterson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Carolyn J. Presley
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kari Kendra
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sandip H. Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Andrew Johns
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Mingjia Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Madison Grogan
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Gabrielle Lopez
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Dwight H. Owen
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Lang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AN developed the initial manuscript draft, incorporated all author comments and edits throughout multiple versions, and completed the final draft for submission. LL and DHO developed the overall concept for this paper, contributed to the initial manuscript draft and provided additional edits and final approval. CWC provided significant input into the construct of the entire manuscript, gathered and formulated data and reviewed edits for inclusion and provided final approval. SZ, MMZ, GO, CP, KK, SP, AJ, ML, MG, and GL contributed significantly to manual extraction of data and logistical support of the REDCap database. All authors critically reviewed the manuscript and approved submission.

Corresponding author

Correspondence to Andrew Nashed.

Ethics declarations

Conflict of interest

The authors report no conflict of interest.

Ethics approval

This study was approved by Institutional Review Board at the Ohio State University (IRB Study ID #2016C0070, PI: Dwight H. Owen, MD, MS).

Consent to participate

A waiver of consent was granted by the Institutional Review Board at the Ohio State University for this retrospective study.

Consent for publication

The authors listed have participated in the study, read the final version and given consent for publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nashed, A., Zhang, S., Chiang, CW. et al. Comparative assessment of manual chart review and ICD claims data in evaluating immunotherapy-related adverse events. Cancer Immunol Immunother 70, 2761–2769 (2021). https://doi.org/10.1007/s00262-021-02880-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00262-021-02880-0

Keywords

Search

Navigation