Skip to main content

Advertisement

SpringerLink
Log in

Progression of an Artificial Intelligence Chatbot (ChatGPT) for Pediatric Cardiology Educational Knowledge Assessment

  • Research
  • Published:
Pediatric Cardiology Aims and scope Submit manuscript

Abstract

Artificial intelligence chatbots, like ChatGPT, have become powerful tools that are disrupting how humans interact with technology. The potential uses within medicine are vast. In medical education, these chatbots have shown improvements, in a short time span, in generalized medical examinations. We evaluated the overall performance and improvement between ChatGPT 3.5 and 4.0 in a test of pediatric cardiology knowledge. ChatGPT 3.5 and ChatGPT 4.0 were used to answer text-based multiple-choice questions derived from a Pediatric Cardiology Board Review textbook. Each chatbot was given an 88 question test, subcategorized into 11 topics. We excluded questions with modalities other than text (sound clips or images). Statistical analysis was done using an unpaired two-tailed t-test. Of the same 88 questions, ChatGPT 4.0 answered 66% of the questions correctly (n = 58/88) which was significantly greater (p < 0.0001) than ChatGPT 3.5, which only answered 38% (33/88). The ChatGPT 4.0 version also did better on each subspeciality topic as compared to ChatGPT 3.5. While acknowledging that ChatGPT does not yet offer subspecialty level knowledge in pediatric cardiology, the performance in pediatric cardiology educational assessments showed a considerable improvement in a short period of time between ChatGPT 3.5 and 4.0.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Abbreviations

AI:

Artificial intelligence

GPT:

Generative Pre-trained Transformer

NLP:

Natural language processing

ECG:

Electrocardiogram

References

  1. Liu PR, Lu L, Zhang JY, Huo TT, Liu SX, Ye ZW (2021) Application of artificial intelligence in medicine: an overview. Curr Med Sci 41(6):1105–1115

    Article  PubMed  PubMed Central  Google Scholar 

  2. Bart NK, Pepe S, Gregory AT, Denniss AR (2023) Emerging roles of Artificial Intelligence (AI) in cardiology: benefits and barriers in a “Brave New World.” Heart Lung Circ 32(8):883–888

    Article  PubMed  Google Scholar 

  3. Dave T, Athaluri SA, Singh S (2023) ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell 6:1169595

    Article  PubMed  PubMed Central  Google Scholar 

  4. Sardana D, Fagan TR, Wright JT (2023) ChatGPT: a disruptive innovation or disrupting innovation in academia? J Am Dent Assoc 154(5):361–364

    Article  PubMed  Google Scholar 

  5. Gao CA, Howard FM, Markov NS et al (2023) Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digit Med 6(1):75

    Article  PubMed  PubMed Central  Google Scholar 

  6. Ruksakulpiwat S, Kumar A, Ajibade A (2023) Using ChatGPT in medical research: current status and future directions. J Multidiscip Healthc 16:1513–1520

    Article  PubMed  PubMed Central  Google Scholar 

  7. Teixeira da Silva JA (2023) Can ChatGPT rescue or assist with language barriers in healthcare communication? Patient Educ Couns 115:107940

    Article  PubMed  Google Scholar 

  8. Mihalache A, Popovic MM, Muni RH (2023) Performance of an Artificial Intelligence Chatbot in ophthalmic knowledge assessment. JAMA Ophthalmol 141(6):589–597

    Article  PubMed  Google Scholar 

  9. Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD (2023) Evaluating ChatGPT as an adjunct for radiologic decision-making. medRxiv. https://doi.org/10.1101/2023.02.02.23285399

    Article  PubMed  PubMed Central  Google Scholar 

  10. Patel SB, Lam K (2023) ChatGPT: the future of discharge summaries? Lancet Digit Health 5(3):e107–e108

    Article  CAS  PubMed  Google Scholar 

  11. Singh S, Djalilian A, Ali MJ (2023) ChatGPT and ophthalmology: exploring its potential with discharge summaries and operative notes. Semin Ophthalmol 38(5):503–507

    Article  PubMed  Google Scholar 

  12. Skalidis I, Cagnina A, Luangphiphat W et al (2023) ChatGPT takes on the European exam in core cardiology: an artificial intelligence success story? Eur Heart J Digit Health 4(3):279–281

    Article  PubMed  PubMed Central  Google Scholar 

  13. Gibson A (2022) How does ChatGPT perform on the medical licensing exams? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 9:e45312

    Google Scholar 

  14. Kung TH, Cheatham M, Medenilla A et al (2023) Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health 2(2):e0000198

    Article  PubMed  PubMed Central  Google Scholar 

  15. Mihalache A, Huang RS, Popovic MM, Muni RH (2023) Performance of an upgraded Artificial Intelligence Chatbot for ophthalmic knowledge assessment. JAMA Ophthalmol 141(8):798–800

    Article  PubMed  Google Scholar 

  16. Waisberg E, Ong J, Masalkhi M et al (2023) GPT-4: a new era of artificial intelligence in medicine. Ir J Med Sci. 192:3197–3200

    Article  PubMed  Google Scholar 

  17. Eidem B (2023) Pediatric cardiology board review, 2nd edn. Wolters Kluwer, Philadelphia, PA

    Google Scholar 

  18. Sallam M (2023) ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel) 11(6):887

    Article  PubMed  Google Scholar 

  19. Vaid A, Jiang J, Sawant A et al (2023) A foundational vision transformer improves diagnostic performance for electrocardiograms. NPJ Digit Med 6(1):108

    Article  PubMed  PubMed Central  Google Scholar 

  20. He B, Kwan AC, Cho JH et al (2023) Blinded, randomized trial of sonographer versus AI cardiac function assessment. Nature 616(7957):520–524

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. The Daily: ‘Hard Fork—An Interview with Sam Altman’ [Internet]. The New York Times, 2023 November 24, 2023, Podcast

Download references

Funding

None.

Author information

Authors and Affiliations

  1. Division of Cardiology, The Labatt Family Heart Centre, The Hospital for Sick Children, 555 University Ave, Toronto, ON, M5G 1X8, Canada

    Michael N. Gritti, Pedrom Farid & Conall T. Morgan

  2. Department of Pediatrics, University of Toronto, Toronto, ON, Canada

    Michael N. Gritti, Hussain AlTurki & Conall T. Morgan

  3. Department of Pediatrics, The Hospital for Sick Children, Toronto, ON, Canada

    Hussain AlTurki

  4. Schulich School of Medicine and Dentistry, University of Western Ontario, London, ON, Canada

    Pedrom Farid

Authors
  1. Michael N. Gritti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hussain AlTurki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pedrom Farid
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Conall T. Morgan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All author helped with writing and reviewing the manuscript. M.N.G. and C.M. came up with the original idea. P.F. helped with the tables and figures.

Corresponding author

Correspondence to Michael N. Gritti.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gritti, M.N., AlTurki, H., Farid, P. et al. Progression of an Artificial Intelligence Chatbot (ChatGPT) for Pediatric Cardiology Educational Knowledge Assessment. Pediatr Cardiol 45, 309–313 (2024). https://doi.org/10.1007/s00246-023-03385-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00246-023-03385-6

Keywords

Search

Navigation