Skip to main content
SpringerLink
Log in

Dr. GPT will see you now: the ability of large language model-linked chatbots to provide colorectal cancer screening recommendations

  • Original Paper
  • Published:
Health and Technology Aims and scope Submit manuscript

Abstract

Purpose

This study assessed the performance of LLM-linked chatbots in providing accurate advice for colorectal cancer screening to both clinicians and patients.

Methods

We created standardized prompts for nine patient cases varying by age and family history to query ChatGPT, Bing Chat, Google Bard, and Claude 2 for screening recommendations to clinicians. Chatbots were asked to specify which screening test was indicated and the frequency of interval screening. Separately, the chatbots were queried with lay terminology for screening advice to patients. Clinician and patient advice was compared to guidelines from the United States Preventive Services Task Force (USPSTF), Canadian Cancer Society (CCS), and the U.S. Multi-Society Task Force (USMSTF) on Colorectal Cancer.

Results

Based on USPSTF criteria, clinician advice aligned with 3/4 (75.0%), 2/4 (50.0%), 3/4 (75.0%), and 1/4 (25.0%) cases for ChatGPT, Bing Chat, Google Bard, and Claude 2, respectively. With CCS criteria, clinician advice corresponded to 2/4 (50.0%), 2/4 (50.0%), 2/4 (50.0%), and 1/4 (25.0%) cases for ChatGPT, Bing Chat, and Google Bard, respectively. For USMSTF guidelines, clinician advice aligned with 7/9 (77.8%), 5/9 (55.6%), 6/9 (66.7%), and 3/9 (33.3%) cases for ChatGPT, Bing Chat, Google Bard, and Claude 2, respectively. Discordant advice was given to clinicians and patients for 2/9 (22.2%), 3/9 (33.3%), 2/9 (22.2%), and 3/9 (33.3%) cases for ChatGPT, Bing Chat, Google Bard, and Claude 2, respectively. Clinical advice provided by the chatbots stemmed from a range of sources including the American Cancer Society (ACS), USPSTF, USMSTF, and the CCS.

Conclusion

LLM-linked chatbots provide colorectal cancer screening recommendations with inconsistent accuracy for both patients and clinicians. Clinicians must educate patients on the pitfalls of using these platforms for health advice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data availability

Not applicable.

Code availability

Not applicable.

References

  1. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29:1930–40.

    Article  Google Scholar 

  2. Bhattacharya K, Bhattacharya AS, Bhattacharya N, Yagnik VD, Garg P, Kumar S. ChatGPT in Surgical Practice—a New Kid on the Block. Indian J Surg. 2023. https://doi.org/10.1007/s12262-023-03727-x.

    Article  Google Scholar 

  3. Rudolph J, Tan S, Tan S. War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. JALT. 2023. https://doi.org/10.37074/jalt.2023.6.1.23.

    Article  Google Scholar 

  4. Ferdush J, Begum M, Hossain ST. ChatGPT and clinical decision support: scope, application, and limitations. Ann Biomed Eng. 2023. https://doi.org/10.1007/s10439-023-03329-4.

    Article  Google Scholar 

  5. Rahsepar AA, Tavakoli N, Kim GHJ, Hassani C, Abtin F, Bedayat A. How AI responds to common lung Cancer questions: ChatGPT vs Google Bard. Radiology. 2023;307:e230922.

    Article  Google Scholar 

  6. Ayers JW, Leas EC, Dredze M, Allem JP, Grabowski JG, Hill L. Clinicians’ perceptions of barriers to avoiding Inappropriate Imaging for LowBack Pain— Knowing is not enough. JAMA Intern Med. 2016;176:1865–6.

    Article  Google Scholar 

  7. Xie Y, Seth I, Hunter-Smith DJ, Rozen WM, Ross R, Lee M. Aesthetic surgery advice and counseling from Artificial Intelligence: a Rhinoplasty Consultation with ChatGPT. Aesthetic Plast Surg. 2023;47:1985–93.

    Article  Google Scholar 

  8. Haver HL, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J, Yi PH. Appropriateness of breast Cancer Prevention and Screening recommendations provided by ChatGPT. Radiology. 2023;307:e230424.

    Article  Google Scholar 

  9. Fu S, Wang L, Moon S, Zong N, He H, Pejaver V, et al. Recommended practices and ethical considerations for natural language processing-assisted observational research: a scoping review. Clin Transl Sci. 2023;16:398–411.

    Article  Google Scholar 

  10. Kalyta A, De Vera MA, Peacock S, Telford JJ, Brown CJ, Donnellan F, et al. Canadian colorectal cancer screening guidelines: do they need an update given changing incidence and global practice patterns? Curr Oncol. 2021;28:1558–70.

    Article  Google Scholar 

  11. Wolf AMD, Fontham ETH, Church TR, Flowers CR, Guerra CE, LaMonte SJ, et al. Colorectal cancer screening for average-risk adults: 2018 guideline update from the American Cancer Society. CA Cancer J Clin. 2018;68:250–81.

    Article  Google Scholar 

  12. Davidson KW, Barry MJ, Mangione CM, Cabana M, Caughey AB, Davis EM, et al. Screening for Colorectal Cancer: US Preventive Services Task Force Recommendation Statement. JAMA. 2021;325:1965–77.

    Article  Google Scholar 

  13. Rex DK, Boland CR, Dominitz JA, Giardiello FM, Johnson DA, Kaltenbach T, et al. Colorectal Cancer screening: recommendations for Physicians and patients from the U.S. Multi-society Task Force on Colorectal Cancer. Am J Gastroenterol. 2017;112:1016–30.

    Article  Google Scholar 

  14. Bacchus C, Dunfield L, Gorber S, Holmes N, Birtwhistle R, Dickinson J, et al. Recommendations on screening for colorectal cancer in primary care. Can Med Assoc J. 2016;188:340–8.

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This study was unfunded.

Author information

Authors and Affiliations

  1. Division of General Surgery, Department of Surgery, McMaster University, 112 King St E, Unit #713, Hamilton, ON, L8N1A8, Canada

    Bright Huo, Tyler McKechnie, Yung Lee, Vanessa Boudreau, Karim Ramji & Cagla Eskicioglu

  2. Department of General Surgery, Università Politecnica delle Marche, Ancona, Italy

    Monica Ortenzi

  3. Department of Surgery, Papageorgiou General Hospital, Thessaloniki, Greece

    Stavros Antoniou

  4. Hospital Clinico San Carlos, IdISSC, Universidad Complutense de Madrid, Madrid, Spain

    Julio Mayol

  5. Phelix AI, Hamilton, ON, Canada

    Hassaan Ahmed

Authors
  1. Bright Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tyler McKechnie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Monica Ortenzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stavros Antoniou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Julio Mayol
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hassaan Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Vanessa Boudreau
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Karim Ramji
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Cagla Eskicioglu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. SA, JM, HA, VB, KR, and CE provided expert guidance on the study methodology. Material preparation, data collection and analysis were performed by BH, TM, MO, and Yung Lee. The first draft of the manuscript was written by Bright Huo and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Bright Huo.

Ethics declarations

Ethical approval

Not applicable.

Consent to participate

Not applicable.

Conflicts of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huo, B., McKechnie, T., Ortenzi, M. et al. Dr. GPT will see you now: the ability of large language model-linked chatbots to provide colorectal cancer screening recommendations. Health Technol. 14, 463–469 (2024). https://doi.org/10.1007/s12553-024-00836-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12553-024-00836-9

Keywords

Search

Navigation