Identifying Breast Cancer Distant Recurrences from Electronic Health Records Using Machine Learning

  • Zexian Zeng
  • Liang Yao
  • Ankita Roy
  • Xiaoyu Li
  • Sasa Espino
  • Susan E Clare
  • Seema A Khan
  • Yuan LuoEmail author
Research Article


Accurately identifying distant recurrences in breast cancer from the electronic health records (EHR) is important for both clinical care and secondary analysis. Although multiple applications have been developed for computational phenotyping in breast cancer, distant recurrence identification still relies heavily on manual chart review. In this study, we aim to develop a model that identifies distant recurrences in breast cancer using clinical narratives and structured data from EHR. We applied MetaMap to extract features from clinical narratives and also retrieved structured clinical data from EHR. Using these features, we trained a support vector machine model to identify distant recurrences in breast cancer patients. We trained the model using 1396 double-annotated subjects and validated the model using 599 double-annotated subjects. In addition, we validated the model on a set of 4904 single-annotated subjects as a generalization test. In the held-out test and generalization test, we obtained F-measure scores of 0.78 and 0.74, area under curve (AUC) scores of 0.95 and 0.93, respectively. To explore the representation learning utility of deep neural networks, we designed multiple convolutional neural networks and multilayer neural networks to identify distant recurrences. Using the same test set and generalizability test set, we obtained F-measure scores of 0.79 ± 0.02 and 0.74 ± 0.004, AUC scores of 0.95 ± 0.002 and 0.95 ± 0.01, respectively. Our model can accurately and efficiently identify distant recurrences in breast cancer by combining features extracted from unstructured clinical narratives and structured clinical data.


Breast cancer Distant recurrence Metastasis NLP, EHR Computational phenotyping Convolutional neural networks Multilayer perceptron 


Funding Information

This project is supported in part by NIH grant R21LM012618-01.

Supplementary material

41666_2019_46_MOESM1_ESM.xlsx (50 kb)
ESM 1 (XLSX 50 kb)
41666_2019_46_MOESM2_ESM.xlsx (13 kb)
ESM 2 (XLSX 13 kb)


  1. 1.
    Egner JR (2010) AJCC cancer staging manual. JAMA 304(15):1726–1727CrossRefGoogle Scholar
  2. 2.
    Lê MG, Arriagada R, Spielmann M, Guinebretière JM, Rochard F (2002) Prognostic factors for death after an isolated local recurrence in patients with early-stage breast carcinoma. Cancer 94(11):2813–2820CrossRefGoogle Scholar
  3. 3.
    Geiger AM, Thwin SS, Lash TL, Buist DSM, Prout MN, Wei F, Field TS, Ulcickas Yood M, Frost FJ, Enger SM, Silliman RA (2007) Recurrences and second primary breast cancers in older women with initial early-stage disease. Cancer 109(5):966–974CrossRefGoogle Scholar
  4. 4.
    Habel LA, Achacoso NS, Haque R, Nekhlyudov L, Fletcher SW, Schnitt SJ, Collins LC, Geiger AM, Puligandla B, Acton L, Quesenberry CP (2009) Declining recurrence among ductal carcinoma in situ patients treated with breast-conserving surgery in the community setting. Breast Cancer Res 11(6):R85CrossRefGoogle Scholar
  5. 5.
    Starren JB, Winter AQ, Lloyd-Jones DM (2015) Enabling a learning health system through a unified enterprise data warehouse: the experience of the Northwestern University Clinical and Translational Sciences (NUCATS) Institute. Clin Transl Sci 8(4):269–271CrossRefGoogle Scholar
  6. 6.
    Birman-Deych E, Waterman AD, Yan Y, Nilasena DS, Radford MJ, Gage BF (2005) Accuracy of ICD-9-CM codes for identifying cardiovascular and stroke risk factors. Med Care 43(5):480–485CrossRefGoogle Scholar
  7. 7.
    Singh JA, Holmgren AR, Noorbaloochi S (2004) Accuracy of Veterans Administration databases for a diagnosis of rheumatoid arthritis. Arthritis Care Res 51(6):952–957CrossRefGoogle Scholar
  8. 8.
    O'malley KJ, Cook KF, Price MD, Wildes KR, Hurdle JF, Ashton CM (2005) Measuring diagnoses: ICD code accuracy. Health Serv Res 40(5p2):1620–1639CrossRefGoogle Scholar
  9. 9.
    Hripcsak G, Albers DJ (2012) Next-generation phenotyping of electronic health records. J Am Med Inform Assoc 20(1):117–121CrossRefGoogle Scholar
  10. 10.
    Greenhalgh T (1999) Narrative based medicine: narrative based medicine in an evidence based world. BMJ Br Med J 318(7179):323–325CrossRefGoogle Scholar
  11. 11.
    Liao KP, Cai T, Gainer V, Goryachev S, Zeng-treitler Q, Raychaudhuri S, Szolovits P, Churchill S, Murphy S, Kohane I, Karlson EW, Plenge RM (2010) Electronic medical records for discovery research in rheumatoid arthritis. Arthritis Care Res 62(8):1120–1127CrossRefGoogle Scholar
  12. 12.
    G. Chao and S. Sun, "Applying a multitask feature sparsity method for the classification of semantic relations between nominals," in Machine Learning and Cybernetics (ICMLC), 2012 International Conference on, 2012, vol. 1, pp. 72–76: IEEEGoogle Scholar
  13. 13.
    Luo Y et al (2017) Natural language processing for EHR-based pharmacovigilance: a structured review. Drug Saf:1–15Google Scholar
  14. 14.
    Zeng Z, Deng Y, Li X, Naumann T, Luo Y (2018) Natural language processing for EHR-based computational phenotyping. IEEE/ACM Transactions on Computational Biology and Bioinformatics:1–1Google Scholar
  15. 15.
    D. S. Carrell, S. Halgrim, D.T. Tran, D. S. M. Buist, J. Chubak, W. W. Chapman, G. Savova, "Using natural language processing to improve efficiency of manual chart abstraction in research: the case of breast cancer recurrence," American journal of epidemiology, p. kwt441, 2014, 179, 749, 758Google Scholar
  16. 16.
    Strauss JA, Chao CR, Kwan ML, Ahmed SA, Schottinger JE, Quinn VP (2013) Identifying primary and recurrent cancers using a SAS-based natural language processing algorithm. J Am Med Inform Assoc 20(2):349–355CrossRefGoogle Scholar
  17. 17.
    Bosco JL et al (2009) Breast cancer recurrence in older women five to ten years after diagnosis. Cancer Epidemiology and Prevention Biomarkers 18(11):2979–2983CrossRefGoogle Scholar
  18. 18.
    Haque R, Shi J, Schottinger JE, Ahmed SA, Chung J, Avila C, Lee VS, Cheetham TC, Habel LA, Fletcher SW, Kwan ML (2015) A hybrid approach to identify subsequent breast cancer using pathology and automated health information data. Med Care 53(4):380–385CrossRefGoogle Scholar
  19. 19.
    Wallner LP, Dibello JR, Li BH, Zheng C, Yu W, Weinmann S, Richert-Boe KE, Ritzwoller DP, VanDenEeden SK, Jacobsen SJ (2014) Development of an algorithm to identify metastatic prostate cancer in electronic medical records using natural language processing. Proc Am Soc Clin Oncol 32:164CrossRefGoogle Scholar
  20. 20.
    Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. Mach Learn ECML-98:137–142Google Scholar
  21. 21.
    Garla V, Taylor C, Brandt C (2013) Semi-supervised clinical text classification with Laplacian SVMs: an application to cancer case management. J Biomed Inform 46(5):869–875CrossRefGoogle Scholar
  22. 22.
    Bejan CA, Xia F, Vanderwende L, Wurfel MM, Yetisgen-Yildiz M (2012) Pneumonia identification using statistical feature selection. J Am Med Inform Assoc 19(5):817–823CrossRefGoogle Scholar
  23. 23.
    McCowan IA, Moore DC, Nguyen AN, Bowman RV, Clarke BE, Duhig EE, Fry MJ (2007) Collection of cancer stage data by classifying free-text medical reports. J Am Med Inform Assoc 14(6):736–745CrossRefGoogle Scholar
  24. 24.
    Z. Zeng et al., "Contralateral breast cancer event detection using Nature Language Processing," in AMIA Annual Symposium Proceedings, 2017, vol. 2017, pp. 1885–1892: American Medical Informatics AssociationGoogle Scholar
  25. 25.
    R. J. Carroll, A. E. Eyler, and J. C. Denny, "Naïve electronic health record phenotype identification for rheumatoid arthritis," in AMIA annual symposium proceedings, 2011, vol. 2011, p. 189: American Medical Informatics AssociationGoogle Scholar
  26. 26.
    Denny JC, Smithers JD, Miller RA, Spickard A III (2003) “Understanding” medical school curriculum content using KnowledgeMap. J Am Med Inform Assoc 10(4):351–362CrossRefGoogle Scholar
  27. 27.
    Y. Kim, "Convolutional neural networks for sentence classification," arXiv preprint arXiv:1408.5882, 2014Google Scholar
  28. 28.
    N. Kalchbrenner, E. Grefenstette, and P. Blunsom, "A convolutional neural network for modelling sentences," arXiv preprint arXiv:1404.2188, 2014Google Scholar
  29. 29.
    K. S. Tai, R. Socher, and C. D. Manning, "Improved semantic representations from tree-structured long short-term memory networks," arXiv preprint arXiv:1503.00075, 2015Google Scholar
  30. 30.
    Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, "Hierarchical attention networks for document classification," in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1480–1489Google Scholar
  31. 31.
    S. Gehrmann et al., "Comparing Rule-Based and Deep Learning Models for Patient Phenotyping," arXiv preprint arXiv:1703.08705, 2017Google Scholar
  32. 32.
    Luo Y (2017) Recurrent neural networks for classifying relations in clinical notes. J Biomed Inform 72:85–95CrossRefGoogle Scholar
  33. 33.
    Luo Y, Cheng Y, Uzuner Ö, Szolovits P, Starren J (2017) Segment convolutional neural networks (Seg-CNNs) for classifying relations in clinical notes. J Am Med Inform Assoc 25(1):93–98CrossRefGoogle Scholar
  34. 34.
    Wu Y, Jiang M, Lei J, Xu H (2015) Named entity recognition in Chinese clinical text using deep neural network. Studies in health technology and informatics 216:624Google Scholar
  35. 35.
    A. N. Jagannatha and H. Yu, "Structured prediction models for RNN based sequence labeling in clinical text," in Proceedings of the Conference on Empirical Methods in Natural Language Processing Conference on Empirical Methods in Natural Language Processing, 2016, vol. 2016, p. 856: NIH Public AccessGoogle Scholar
  36. 36.
    A. N. Jagannatha and H. Yu, "Bidirectional rnn for medical event detection in electronic health records," in Proceedings of the conference Association for Computational Linguistics North American Chapter Meeting, 2016, vol. 2016, p. 473: NIH Public AccessGoogle Scholar
  37. 37.
    DeLisle S, Kim B, Deepak J, Siddiqui T, Gundlapalli A, Samore M, D'Avolio L (2013) Using the electronic medical record to identify community-acquired pneumonia: toward a replicable automated strategy. PLoS One 8(8):e70944CrossRefGoogle Scholar
  38. 38.
    Lin C, Karlson EW, Dligach D, Ramirez MP, Miller TA, Mo H, Braggs NS, Cagan A, Gainer V, Denny JC, Savova GK (2014) Automatic identification of methotrexate-induced liver toxicity in patients with rheumatoid arthritis from the electronic medical record. J Am Med Inform Assoc 22(e1):e151–e161CrossRefGoogle Scholar
  39. 39.
    Liao KP, Cai T, Savova GK, Murphy SN, Karlson EW, Ananthakrishnan AN, Gainer VS, Shaw SY, Xia Z, Szolovits P, Churchill S, Kohane I (2015) Development of phenotype algorithms using electronic medical records and incorporating natural language processing. bmj 350:h1885CrossRefGoogle Scholar
  40. 40.
    F. Galton, Finger prints. Macmillan and Company, 1892Google Scholar
  41. 41.
    Leemans CR, Tiwari R, Nauta J, Van der Waal I, Snow GB (1993) Regional lymph node involvement and its significance in the development of distant metastases in head and neck carcinoma. Cancer 71(2):452–456CrossRefGoogle Scholar
  42. 42.
    A. R. Aronson, "Metamap: mapping text to the umls metathesaurus," Bethesda, MD: NLM, NIH, DHHS, pp. 1–26, 2006Google Scholar
  43. 43.
    Chapman WW et al (2013) Extending the NegEx lexicon for multiple languages. Stud Health Technol Inform 192:677Google Scholar
  44. 44.
    Pedregosa F et al (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830MathSciNetzbMATHGoogle Scholar
  45. 45.
    L. De Vine, G. Zuccon, B. Koopman, L. Sitbon, and P. Bruza, "Medical semantic similarity with a neural language model," in Proceedings of the 23rd ACM international conference on conference on information and knowledge management, 2014, pp. 1819–1822: ACMGoogle Scholar
  46. 46.
    M. Abadi et al, "Tensorflow: a system for large-scale machine learning," in OSDI, 2016, vol. 16, pp. 265–283Google Scholar
  47. 47.
    D. Kinga and J. B. Adam, "A method for stochastic optimization," in International Conference on Learning Representations (ICLR), 2015, vol. 5Google Scholar
  48. 48.
    Luo Y, Xin Y, Hochberg E, Joshi R, Uzuner O, Szolovits P (2015) Subgraph augmented non-negative tensor factorization (SANTF) for modeling clinical narrative text. J Am Med Inform Assoc:ocv016Google Scholar
  49. 49.
    Luo Y, Sohani AR, Hochberg EP, Szolovits P (2014) Automatic lymphoma classification with sentence subgraph mining from pathology reports. J Am Med Inform Assoc 21(5):824–832CrossRefGoogle Scholar
  50. 50.
    Boland MR, Hripcsak G, Shen Y, Chung WK, Weng C (2013) Defining a comprehensive verotype using electronic health records for personalized medicine. J Am Med Inform Assoc 20:e232–e238CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Preventive Medicine, Northwestern University Feinberg School of MedicineChicagoUSA
  2. 2.Department of Surgery, Northwestern University Feinberg School of MedicineChicagoUSA
  3. 3.Department of Medicine, Brigham and Women’s HospitalBostonUSA

Personalised recommendations