Skip to main content

Clinician perspectives on machine learning prognostic algorithms in the routine care of patients with cancer: a qualitative study

Abstract

Purpose

Oncologists may overestimate prognosis for patients with cancer, leading to delayed or missed conversations about patients’ goals and subsequent low-quality end-of-life care. Machine learning algorithms may accurately predict mortality risk in cancer, but it is unclear how oncology clinicians would use such algorithms in practice.

Methods

The purpose of this qualitative study was to assess oncology clinicians’ perceptions on the utility and barriers of machine learning prognostic algorithms to prompt advance care planning. Participants included medical oncology physicians and advanced practice providers (APPs) practicing in tertiary and community practices within a large academic healthcare system. Transcripts were coded and analyzed inductively using NVivo software.

Results

The study included 29 oncology clinicians (19 physicians, 10 APPs) across 6 practice sites (1 tertiary, 5 community) in the USA. Fourteen participants had previously had exposure to an automated machine learning-based prognostic algorithm as part of a pragmatic randomized trial. Clinicians believed that there was utility for algorithms in validating their own intuition about prognosis and prompting conversations about patient goals and preferences. However, this enthusiasm was tempered by concerns about algorithm accuracy, over-reliance on algorithm predictions, and the ethical implications around disclosure of an algorithm prediction. There was significant variation in tolerance for false positive vs. false negative predictions.

Conclusion

While oncologists believe there are applications for advanced prognostic algorithms in routine care of patients with cancer, they are concerned about algorithm accuracy, confirmation and automation biases, and ethical issues of prognostic disclosure.

This is a preview of subscription content, access via your institution.

Data availability

Not applicable.

References

  1. Gilligan T, Coyle N, Frankel RM et al (2017) Patient-clinician communication: American Society of Clinical Oncology consensus guideline. J Clin Oncol 35:3618–3632. https://doi.org/10.1200/JCO.2017.75.2311

    Article  PubMed  Google Scholar 

  2. Emanuel EJ, Young-Xu Y, Levinsky NG et al (2003) Chemotherapy use among Medicare beneficiaries at the end of life. Ann Intern Med 138:639–643

    Article  Google Scholar 

  3. Earle CC, Neville BA, Landrum MB et al (2004) Trends in the aggressiveness of cancer care near the end of life. J Clin Oncol 22:315–321. https://doi.org/10.1200/JCO.2004.08.136

    Article  PubMed  Google Scholar 

  4. Earle CC, Landrum MB, Souza JM et al (2008) Aggressiveness of cancer care near the end of life: is it a quality-of-care issue? J Clin Oncol 26:3860–3866. https://doi.org/10.1200/JCO.2007.15.8253

    Article  PubMed  PubMed Central  Google Scholar 

  5. Chastek B, Harley C, Kallich J et al (2012) Health care costs for patients with cancer at the end of life. J Oncol Pract 8:75s–80s. https://doi.org/10.1200/JOP.2011.000469

    Article  PubMed  PubMed Central  Google Scholar 

  6. Wen F-H, Chen J-S, Su P-J et al (2018) Terminally ill cancer patients’ concordance between preferred life-sustaining treatment states in their last six months of life and received life-sustaining treatment states in their last month: an observational study. J Pain Symptom Manage 56:509-518.e3. https://doi.org/10.1016/j.jpainsymman.2018.07.003

    Article  PubMed  Google Scholar 

  7. Christakis NA, Lamont EB (2000) Extent and determinants of error in doctors’ prognoses in terminally ill patients: prospective cohort study. BMJ 320:469–472. https://doi.org/10.1136/bmj.320.7233.469

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  8. Sborov K, Giaretta S, Koong A et al (2019) Impact of accuracy of survival predictions on quality of end-of-life care among patients with metastatic cancer who receive radiation therapy. J Oncol Pract 18:e262–e270. https://doi.org/10.1200/JOP.18.00516

    Article  Google Scholar 

  9. Manz CR, Parikh RB, Small DS et al (2020) Effect of integrating machine learning mortality estimates with behavioral nudges to clinicians on serious illness conversations among patients with cancer: a stepped-wedge cluster randomized clinical trial. JAMA Oncol 2020:e204759. https://doi.org/10.1001/jamaoncol.2020.4759

    Article  Google Scholar 

  10. Wright AA, Zhang B, Ray A et al (2008) Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA 300:1665–1673. https://doi.org/10.1001/jama.300.14.1665

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  11. Brinkman-Stoppelenburg A, Rietjens JAC, van der Heide A (2014) The effects of advance care planning on end-of-life care: a systematic review. Palliat Med 28:1000–1025. https://doi.org/10.1177/0269216314526272

    Article  PubMed  Google Scholar 

  12. Robbins R (2020) Hospitals tap AI to nudge clinicians toward end-of-life conversations. https://www.statnews.com/2020/07/01/end-of-life-artificial-intelligence/. Accessed 6 Oct 2020

  13. Huang S, Yang J, Fong S et al (2020) Artificial intelligence in cancer diagnosis and prognosis: Opportunities and challenges. Cancer Lett 471:61–71. https://doi.org/10.1016/j.canlet.2019.12.007

    CAS  Article  PubMed  Google Scholar 

  14. Machine learning applications in cancer prognosis and prediction (2015) Computational and Structural. Biotechnol J 13:8–17. https://doi.org/10.1016/j.csbj.2014.11.005

    CAS  Article  Google Scholar 

  15. Nagy M, Radakovich N, Nazha A (2020) Machine learning in oncology: what should clinicians know? JCO Clin Cancer Inform 4:799–810. https://doi.org/10.1200/CCI.20.00049

    Article  PubMed  Google Scholar 

  16. Elfiky AA, Pany MJ, Parikh RB et al (2018) Development and application of a machine learning approach to assess short-term mortality risk among patients with cancer starting chemotherapy. JAMA Netw Open 1:e180926–e180926. https://doi.org/10.1001/jamanetworkopen.2018.0926

    Article  PubMed  PubMed Central  Google Scholar 

  17. Thorsen-Meyer H-C, Nielsen AB, Nielsen AP et al (2020) Dynamic and explainable machine learning prediction of mortality in patients in the intensive care unit: a retrospective study of high-frequency data in electronic patient records. Lancet Digit Health 2(4):e179–e191. https://doi.org/10.1016/S2589-7500(20)30018-2

    Article  PubMed  Google Scholar 

  18. Brajer N, Cozzi B, Gao M et al (2020) Prospective and external evaluation of a machine learning model to predict in-hospital mortality of adults at time of admission. JAMA Netw Open 3:e1920733. https://doi.org/10.1001/jamanetworkopen.2019.20733

    Article  PubMed  Google Scholar 

  19. Desai RJ, Wang SV, Vaduganathan M et al (2020) Comparison of machine learning methods with traditional models for use of administrative claims with electronic medical records to predict heart failure outcomes. JAMA Netw Open 3:e1918962. https://doi.org/10.1001/jamanetworkopen.2019.18962

    Article  PubMed  PubMed Central  Google Scholar 

  20. Marafino BJ, Park M, Davies JM et al (2018) Validation of prediction models for critical care outcomes using natural language processing of electronic health record data. JAMA Netw Open 1:e185097. https://doi.org/10.1001/jamanetworkopen.2018.5097

    Article  PubMed  PubMed Central  Google Scholar 

  21. Sahni N, Simon G, Arora R (2018) Development and validation of machine learning models for prediction of 1-year mortality utilizing electronic medical record data available at the end of hospitalization in multicondition patients: a proof-of-concept study. J Gen Intern Med 33:921–928. https://doi.org/10.1007/s11606-018-4316-y

    Article  PubMed  PubMed Central  Google Scholar 

  22. Rajkomar A, Oren E, Chen K et al (2018) Scalable and accurate deep learning with electronic health records. NPJ Digit Med 1:18. https://doi.org/10.1038/s41746-018-0029-1

    Article  PubMed  PubMed Central  Google Scholar 

  23. Bertsimas D, Dunn J, Pawlowski C et al (2018) Applied informatics decision support tool for mortality predictions in patients with cancer. JCO Clin Cancer Inform 2:1–11. https://doi.org/10.1200/CCI.18.00003

    Article  PubMed  Google Scholar 

  24. Parikh RB, Manz C, Chivers C et al (2019) Machine learning approaches to predict 6-month mortality among patients with cancer. JAMA Netw Open 2:e1915997. https://doi.org/10.1001/jamanetworkopen.2019.15997

    Article  PubMed  PubMed Central  Google Scholar 

  25. Titano JJ, Badgeley M, Schefflein J et al (2018) Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat Med 24:1337–1341. https://doi.org/10.1038/s41591-018-0147-y

    CAS  Article  PubMed  Google Scholar 

  26. Gensheimer MF, Aggarwal S, Benson KRK et al (2020) Automated model versus treating physician for predicting survival time of patients with metastatic cancer. J Am Med Inform Assoc 2020:ocaa290. https://doi.org/10.1093/jamia/ocaa290

    Article  Google Scholar 

  27. Parikh RB, Gdowski A, Patt DA et al (2019) Using big data and predictive analytics to determine patient risk in oncology. Am Soc Clin Oncol Educ Book 39:e53–e58. https://doi.org/10.1200/EDBK_238891

    Article  PubMed  Google Scholar 

  28. Vollmer S, Mateen BA, Bohner G et al (2020) Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ 368:l6927. https://doi.org/10.1136/bmj.l6927

    Article  PubMed  Google Scholar 

  29. Hallen SAM, Hootsmans NAM, Blaisdell L, Gutheil CM, Han PKJ (2015) Physicians’ perceptions of the value of prognostic models: the benefits and risks of prognostic confidence. Health Expect 18(6):2266–2277. https://doi.org/10.1111/hex.12196

    Article  PubMed  Google Scholar 

  30. Adibi A, Sadatsafavi M, Ioannidis JPA (2020) Validation and utility testing of clinical prediction models: time to change the approach. JAMA 324:235–236. https://doi.org/10.1001/jama.2020.1230

    Article  PubMed  Google Scholar 

  31. Manz CR, Chen J, Liu M et al (2020) Validation of a machine learning algorithm to predict 180-day mortality for outpatients with cancer. JAMA oncology 6(11):1723–1730. https://doi.org/10.1001/jamaoncol.2020.4331

    Article  PubMed  PubMed Central  Google Scholar 

  32. Bernacki RE (2003) Block SD (2014) American College of Physicians High Value Care Task Force Communication about serious illness care goals: a review and synthesis of best practices. JAMA Int Med 174:1994. https://doi.org/10.1001/jamainternmed.2014.5271

    Article  Google Scholar 

  33. Dying in America: improving quality and honoring individual preferences near the end of life. Institute of Medicine. http://nationalacademies.org/hmd/Reports/2014/Dying-In-America-Improving-Quality-and-Honoring-Individual-Preferences-Near-the-End-of-Life.aspx. Accessed 23 Apr 2019

  34. Elston DM (2020) Confirmation bias in medical decision-making. J Am Acad Dermatol 82:572. https://doi.org/10.1016/j.jaad.2019.06.1286

    Article  PubMed  Google Scholar 

  35. Saposnik G, Redelmeier D, Ruff CC et al (2016) Cognitive biases associated with medical decisions: a systematic review. BMC Med Inform Decis Mak 16:138. https://doi.org/10.1186/s12911-016-0377-1

    Article  PubMed  PubMed Central  Google Scholar 

  36. Zerilli J, Knott A, Maclaurin J et al (2019) Algorithmic decision-making and the control problem. Mind Mach 29:555–578

    Article  Google Scholar 

  37. Chen C-H, Tang S-T (2014) Prognostic disclosure and its influence on cancer patients. J Cancer Res Pract 1:103–112. https://doi.org/10.6323/JCRP.2014.1.2.02

    Article  Google Scholar 

  38. van der Velden NCA, Meijers MC, Han PKJ et al (2020) The effect of prognostic communication on patient outcomes in palliative cancer care: a systematic review. Curr Treat Options Oncol 21:40. https://doi.org/10.1007/s11864-020-00742-y

    Article  PubMed  PubMed Central  Google Scholar 

  39. Walczak A, Henselmans I, Tattersall MHN et al (2015) A qualitative analysis of responses to a question prompt list and prognosis and end-of-life care discussion prompts delivered in a communication support program. Psychooncology 24:287–293. https://doi.org/10.1002/pon.3635

    Article  PubMed  Google Scholar 

  40. Obermeyer Z, Powers B, Vogeli C et al (2019) Dissecting racial bias in an algorithm used to manage the health of populations. Science 366:447–453. https://doi.org/10.1126/science.aax2342

    CAS  Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors would acknowledge Zoe Belardo for assistance in interview coding.

Funding

This study was supported by the National Cancer Institute K08CA263541 (to R.B.P.), Penn Center for Precision Medicine Accelerator Fund (to R.B.P. and C.R.M.), and the National Palliative Care Research Center (to R.B.P.). The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Author information

Authors and Affiliations

Authors

Contributions

Ravi Parikh (conceptualization, formal analysis, funding acquisition, methodology, resources, writing—original draft, writing—review and editing); Christopher Manz (conceptualization, formal analysis, funding acquisition, methodology, resources, writing—original draft, writing—review and editing); Maria Nelson (data curation, formal analysis, methodology, resources, software, writing—review and editing); Chalanda Evans (project administration); Susan Regli (writing—review and editing); Nina O’Connor (writing—review and editing); Lynn Schuchter (writing—review and editing); Lawrence Shulman (writing—review and editing); Mitesh Patel (writing—review and editing); Joanna Paladino (writing—review and editing); Judy Shea (conceptualization, methodology, resources, supervision, writing—review and editing).

Corresponding author

Correspondence to Ravi B. Parikh.

Ethics declarations

Ethics approval

This study was approved by the University of Pennsylvania Institutional Review Board.

Consent to participate

Informed consent was obtained from all individual participants included in this study.

Consent for publication

Participants signed informed consent regarding the publishing of their data in a de identified manner.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Dr. Ravi B. Parikh and Dr. Christopher R. Manz are co-first authors of this publication.

Supplementary Information

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Parikh, R.B., Manz, C.R., Nelson, M.N. et al. Clinician perspectives on machine learning prognostic algorithms in the routine care of patients with cancer: a qualitative study. Support Care Cancer 30, 4363–4372 (2022). https://doi.org/10.1007/s00520-021-06774-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00520-021-06774-w

Keywords

  • Predictive analytics
  • Machine learning
  • Advance care planning
  • Palliative care
  • Supportive oncology