Skip to main content

Advertisement

SpringerLink
Log in

Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google Bard

  • Miscellaneous
  • Published:
European Archives of Oto-Rhino-Laryngology Aims and scope Submit manuscript

Abstract

Purpose

To perform the first head-to-head comparative evaluation of patient education material for obstructive sleep apnoea generated by two artificial intelligence chatbots, ChatGPT and its primary rival Google Bard.

Methods

Fifty frequently asked questions on obstructive sleep apnoea in English were extracted from the patient information webpages of four major sleep organizations and categorized as input prompts. ChatGPT and Google Bard responses were selected and independently rated using the Patient Education Materials Assessment Tool–Printable (PEMAT-P) Auto-Scoring Form by two otolaryngologists, with a Fellowship of the Royal College of Surgeons (FRCS) and a special interest in sleep medicine and surgery. Responses were subjectively screened for any incorrect or dangerous information as a secondary outcome. The Flesch-Kincaid Calculator was used to evaluate the readability of responses for both ChatGPT and Google Bard.

Results

A total of 46 questions were curated and categorized into three domains: condition (n = 14), investigation (n = 9) and treatment (n = 23). Understandability scores for ChatGPT versus Google Bard on the various domains were as follows: condition 90.86% vs.76.32% (p < 0.001); investigation 89.94% vs. 71.67% (p < 0.001); treatment 90.78% vs.73.74% (p < 0.001). Actionability scores for ChatGPT versus Google Bard on the various domains were as follows: condition 77.14% vs. 51.43% (p < 0.001); investigation 72.22% vs. 54.44% (p = 0.05); treatment 73.04% vs. 54.78% (p = 0.002). The mean Flesch–Kincaid Grade Level for ChatGPT was 9.0 and Google Bard was 5.9. No incorrect or dangerous information was identified in any of the generated responses from both ChatGPT and Google Bard.

Conclusion

Evaluation of ChatGPT and Google Bard patient education material for OSA indicates the former to offer superior information across several domains.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data, materials and/or code availability

Data are available on request from the corresponding author.

References

  1. OpenAI (2022) Introducing ChatGPT [Internet]. OpenAI.com. pp 1–11. https://openai.com/blog/chatgpt. Accessed 6 July 2023

  2. Google AI updates: Bard and new AI features in Search [Internet]. Google. (2023). https://blog.google/technology/ai/bard-google-ai-search-updates/. Accessed 6 July 2023

  3. Meskó B, Topol EJ (2023) The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. https://doi.org/10.1038/s41746-023-00873-0

    Article  PubMed  PubMed Central  Google Scholar 

  4. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C et al (2023) Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 2(2):e0000198

    Article  PubMed  PubMed Central  Google Scholar 

  5. Patel SB, Lam K (2023) ChatGPT: the future of discharge summaries? Lancet Digit Heal [Internet] 5(3):e107–e108. https://doi.org/10.1016/S2589-7500(23)00021-3

    Article  CAS  Google Scholar 

  6. Ali SR, Dobbs TD, Hutchings HA, Whitaker IS (2023) Using ChatGPT to write patient clinic letters. Lancet Digit Heal [Internet]. 5(4):e179–e181. https://doi.org/10.1016/S2589-7500(23)00048-1

    Article  CAS  Google Scholar 

  7. Ayoub NF, Lee YJ, Grimm D, Balakrishnan K (2023) Comparison between ChatGPT and google search as sources of postoperative patient instructions. JAMA Otolaryngol Head Neck Surg [Internet] 149(6):556–558

    Article  PubMed  Google Scholar 

  8. Benjafield AV, Ayas NT, Eastwood PR, Heinzer R, Ip MSM, Morrell MJ et al (2019) Estimation of the global prevalence and burden of obstructive sleep apnoea: a literature-based analysis. Lancet Respir Med 7(8):687–698

    Article  PubMed  PubMed Central  Google Scholar 

  9. Marin JM, Carrizo SJ, Vicente E, Agusti AGN (2005) Long-term cardiovascular outcomes in men with obstructive sleep apnoea-hypopnoea with or without treatment with continuous positive airway pressure: an observational study. Lancet 365(9464):1046–1053

    Article  PubMed  Google Scholar 

  10. Watson NF (2016) Health care savings: the economic value of diagnostic and therapeutic care for obstructive sleep apnea. J Clin Sleep Med 12(8):1075–1077

    Article  PubMed  PubMed Central  Google Scholar 

  11. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web—the Official Microsoft Blog [Internet]. https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/. Accessed 8 July 2023

  12. Patient-Friendly Guides—American Academy of Sleep Medicine—Association for Sleep Clinicians and Researchers [Internet]. https://aasm.org/clinical-resources/patient-info/patient-friendly-guidelines/. Accessed 12 July 2023

  13. Obstructive Sleep Apnea (OSA) [Internet]. https://www.sleephealthfoundation.org.au/obstructive-sleep-apnea.html. Accessed 12 July 2023

  14. Snoring and Sleep Apnoea | ENT UK [Internet]. https://www.entuk.org/patients/conditions/34/snoring_and_sleep_apnoea. Accessed 12 July 2023

  15. For Patients | International Surgical Sleep Society [Internet]. https://www.surgicalsleep.org/for-patients. Accessed 12 July 2023.

  16. Flesch Kincaid Calculator | Good Calculators [Internet]. https://goodcalculators.com/flesch-kincaid-calculator/. Accessed 20 July 2023

  17. Shoemaker SJ, Wolf MS, Brach C. The patient education materials assessment tool (PEMAT) and user’s guide. Rockville, MD: Agency for Healthcare Research and Quality. 2013 Nov;11.

  18. Introduction | Agency for Healthcare Research and Quality [Internet]. https://www.ahrq.gov/health-literacy/patient-education/pemat1.html. Accessed 15 July 2023

  19. Lyons MM, Bhatt NY, Pack AI, Magalang UJ (2020) Global burden of sleep-disordered breathing and its implications. Respirology 25(7):690–702

    Article  PubMed  Google Scholar 

  20. Bing vs Google: Search Engine Comparison 2023 | Impression [Internet]. https://www.impressiondigital.com/blog/bing-differ-google/. Accessed 15 July 2023

  21. Internet and social media users in the world 2023 | Statista [Internet]. https://www.statista.com/statistics/617136/digital-population-worldwide/. Accessed 15 July 2023

  22. Alkaissi H, McFarlane SI (2023) Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus 15(2):2–5

    Google Scholar 

  23. ChatGPT and LLMs: what’s the risk—NCSC.GOV.UK [Internet]. https://www.ncsc.gov.uk/blog-post/chatgpt-and-large-language-models-whats-the-risk. Accessed 16 July 2023

  24. Oliffe M, Thompson E, Johnston J, Freeman D, Bagga H, Wong PKK (2019) Assessing the readability and patient comprehension of rheumatology medicine information sheets: a cross-sectional Health Literacy Study. BMJ Open 9(2):1–10

    Article  Google Scholar 

  25. Epic and Microsoft Bring GPT-4 to EHRs [Internet]. https://www.epic.com/epic/post/epic-and-microsoft-bring-gpt-4-to-ehrs. Accessed 6 July 2023

  26. NHS Long Term Plan » The NHS Long Term Plan [Internet]. https://www.longtermplan.nhs.uk/publication/nhs-long-term-plan/. Accessed 7 July 2023

  27. Funding boost for artificial intelligence in NHS to speed up diagnosis of deadly diseases—GOV.UK [Internet]. https://www.gov.uk/government/news/funding-boost-for-artificial-intelligence-in-nhs-to-speed-up-diagnosis-of-deadly-diseases. Accessed 17 July 2023

  28. £21 million to roll out artificial intelligence across the NHS—GOV.UK [Internet]. https://www.gov.uk/government/news/21-million-to-roll-out-artificial-intelligence-across-the-nhs. Accessed 17 July 2023

Download references

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Otolaryngology-Head and Neck Surgery Department, The Royal Marsden NHS Foundation Trust, Fulham Road, London, SW3 6JJ, UK

    Ryan Chin Taw Cheong, Andrew Williamson & Vinidh Paleri

  2. Otolaryngology-Head and Neck Surgery Department, The Royal National ENT and Eastman Dental Hospitals, University College London Hospitals NHS Foundation Trust, London, UK

    Samit Unadkat, Venkata Mcneillis, Jonathan Joseph, Premjit Randhawa & Peter Andrews

Authors
  1. Ryan Chin Taw Cheong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samit Unadkat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Venkata Mcneillis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andrew Williamson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jonathan Joseph
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Premjit Randhawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Peter Andrews
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Vinidh Paleri
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study’s conception and design. Material preparation, data collection and analysis were performed by [RCTC], [SU], [AW] and [VP]. The first draft of the manuscript was written by [RCTC], [SU], [VM], [JJ], [PR], [PA] and [VP] all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ryan Chin Taw Cheong.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Ethics approval

No patient or clinical data was used in this study. No ethical approval required.

Consent

No patient or clinical data was used in this study. No consent is required.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheong, R.C.T., Unadkat, S., Mcneillis, V. et al. Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google Bard. Eur Arch Otorhinolaryngol 281, 985–993 (2024). https://doi.org/10.1007/s00405-023-08319-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00405-023-08319-9

Keywords

Search

Navigation