Abstract
Objectives
To evaluate the ChatGPT-4 performance in oncological board decisions.
Methods
Twenty medical records of patients with head and neck cancer were evaluated by ChatGPT-4 for additional examinations, management, and therapeutic approaches. The ChatGPT-4 propositions were assessed with the Artificial Intelligence Performance Instrument. The stability of ChatGPT-4 was evaluated through regenerated answers at 1-day interval.
Results
ChatGPT-4 provided adequate explanations for cTNM staging in 19 cases (95%). ChatGPT-4 proposed a significant higher number of additional examinations than practitioners (72 versus 103; p = 0.001). ChatGPT-4 indications of endoscopy–biopsy, HPV research, ultrasonography, and PET–CT were consistent with the oncological board decisions. The therapeutic propositions of ChatGPT-4 were accurate in 13 cases (65%). Most additional examination and primary treatment propositions were consistent throughout regenerated response process.
Conclusions
ChatGPT-4 may be an adjunctive theoretical tool in oncological board simple decisions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Data are available on request.
References
Vaira LA, Lechien JR, Abbate V, Allevi F, Audino A, Beltramini GA et al (2023) Accuracy of ChatGPT-generated information on head and neck and oromaxillofacial surgery : a multicenter collaborative analysis. Otolaryngol head neck surg. https://doi.org/10.1002/ohn.489
Hill-Yardin EL, Hutchinson MR, Laycock R, Spencer SJ (2023) A Chat(GPT) about the future of scientific publishing. Brain Behav Immun 110:152–154. https://doi.org/10.1016/j.bbi.2023.02.022
Hoch CC, Wollenberg B, Lüers JC, Knoedler S, Knoedler L, Frank K, Cotofana S, Alfertshofer M (2023) ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-023-08051-4
Chiesa-Estomba CM, Lechien JR, Vaira LA, Brunet A, Cammaroto G, Mayo-Yanez M, Sanchez-Barrueco A, Saga-Gutierrez C (2023) Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support. Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-023-08104-8
Lechien JR, Maniaci A, Gengler I, Hans S, Chiesa-Estomba CM, Vaira LA (2023) Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the artificial intelligence performance instrument (AIPI). Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-023-08219-y
Lechien JR, Georgescu BM, Hans S, Chiesa-Estomba CM (2023) ChatGPT performance in laryngology and head & neck surgery: a clinical case-series. Eur Arch Otorhinolaryngol. https://doi.org/10.1007/s00405-023-08282-5
Cuny F, Babin E, Lacau-Saint-Guily J, Baujat B, Bensadoun R, Bozec A, Chevalier D, Choussy O, Deneuve S, Fakhry N, Guigay J, Makeieff M, Merol JC, Mouawad F, Pavillet J, Rebiere C, Righini C, Sostras MC, Tournaille M, Vergez S, SFORL work group (2015) French Society of ENT (SFORL) guidelines for care pathway organization in head and neck oncology (short version). Early management of head and neck cancer. Eur Ann Otorhinolaryngol Head Neck Dis 132(4):205–208. https://doi.org/10.1016/j.anorl.2015.06.007
Verdonck-de Leeuw I, Dawson C, Licitra L, Eriksen JG, Hosal S, Singer S, Laverty DP, Golusinski W, Machczynski P, Varges Gomes A, Girvalaki C, Simon C, Leemans CR (2022) European Head and Neck Society recommendations for head and neckcancer survivorship care. Oral Oncol 133:106047. https://doi.org/10.1016/j.oraloncology.2022.106047
Lydiatt WM et al (2017) Head and Neck cancers-major changes in the American Joint Committee on cancer eighth edition cancer staging manual. CA Cancer J Clin 67(2):122–137
Piazza C, Paderno A, Sjogren EV, Bradley PJ, Eckel HE, Mäkitie A, Matar N, Paleri V, Peretti G, Puxeddu R, Quer M, Remacle M, Vander Poorten V, Vilaseca I, Simo R (2021) Salvage carbon dioxide transoral laser microsurgery for laryngeal cancer after (chemo)radiotherapy: a European Laryngological Society consensusstatement. Eur Arch Otorhinolaryngol 278(11):4373–4381. https://doi.org/10.1007/s00405-021-06957-5
Radulesco T, Saibene AM, Michel J, Vaira LA, Lechien JR (2023) ChatGPT-4 Performance in Rhinology: A Clinical Case-Series. Revision. Int For Allerg Rhinol
Perlis RH (2023). MedRxiv. https://doi.org/10.1101/2023.04.14.23288595
Lechien JR, Briganti G, Vaira LA (2023) Accuracy of ChatGPT-3.5 and -4 in providing scientific references in otolaryngology-head and neck surgery. Eur Arch Otorhinolaryngol
Funding
None.
Author information
Authors and Affiliations
Contributions
Study concept and design: Lechien, Hans, Chiesa-Estomba. Acquisition, analysis, or interpretation of data: Lechien, Hans. Drafting of the manuscript: Lechien. Critical revision of the manuscript for important intellectual content: Hans, Chiesa-Estomba, Baudouin.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflict of interest.
Ethical declarations
The author Jerome R. Lechien is also guest editor of the special issue on ‘ChatGPT and Artificial Intelligence in Otolaryngology—Head and Neck Surgery’. He was not involved with the peer review process of this article.
Informed consent
Patients consented to the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1: Cases
N | G | Age | Symptoms | History/medication | Clinical examination | Oncological board-practitioners | ||
|---|---|---|---|---|---|---|---|---|
Additional examinations | Diagnosis | Oncological board treatments | ||||||
1 | F | 72 | Dysphagia, right otalgia, weight loss (>6kg in 3 months) | HT, DB2, CD, RTU Current ATC | Trismus, normal tongue mobility, Exophytic lesion of the lateral oropharyngeal wall (right) Right neck node | MRI, PET–CT, endoscopy Biopsies: SCC | cT4aN2bM0 OSCC | Primary: radiotherapy Alternative: surgery, free flap, postoperative RT |
2 | M | 45 | Dysphonia (2 months) | Radiotherapy for cT1N0 of the glottic region (4y), Past TC | Anterior commissure lesion, decreased movement of vocal folds | Neck CT, PET–CT Endoscopy, biopsy: LSCC | cT3N0M0 LSCC Thyroid cartilage invasion | Primary: CHEP only Alternative: CTh/immunotherapy |
3 | M | 75 | Dysphonia, neck pain, dysphagia, weight loss (> 5kg in 3 months) | CP, HT, DB2, current ATC, RT for cT1N0 glottic LSCC (4 years) | Fixed right hemilarynx No exophytic lesion | Neck CT, PET–CT, endoscopy Biopsies: SCC | cT3N0M0 LSCC | Primary: total laryngectomy Alternative: CTh/immunotherapy. |
4 | M | 55 | Dysphonia (2 months) | HT, current TC TLM for cT1a glottis LSCC (6 months) | Exophytic lesion of the right vocal cord Normal laryngeal mobility | Neck CT and PET–CT No biopsy regarding morphological lesion | cT1aN0M0 LSCC (recurrence) | Primary: TLM or RT Alternative: – |
5 | M | 56 | Dysphagia (6 months) | Current TC | Exophytic lesion of right ary—epiglottic fold and epiglottis Normal vocal cord exam | Neck CT, PET–CT Endoscopy, biopsy: LSCC | cT2N1M0 Supraglottic LSCC | Primary: TORS supraglottic laryngectomy and neck dissections or RT Alternative: – |
6 | F | 56 | Throat pain, globus (6 months) | Thyroidectomy (1 year) for medullar cancer, bilateral neck dissection | Right oropharyngeal wall mass | Neck MRI, PET–CT Calcitonin biology | Neck recurrence of medullar thyroid cancer (parapharyngeal space) | Primary: TORS node surgery, or targeted therapy Alternative: CTh |
7 | M | 75 | Throat pain, dysphagia, weight loss (>7kg–3 months) | HT, current ATC RT for cT2N2 OSCC (5 years) | Left oropharyngeal wall exophytic lesion, ipsilateral neck nodes | Neck MRI, PET–CT, endoscopy Biopsies: SCC | cT4aN2M1 OSCC spinal bone metastases | Primary: chemotherapy Alternative: immunotherapy bone radiation |
8 | F | 70 | Dysphagia, throat pain (3 months) | HT, no ATC | Right base of tongue ulcerative lesion | Neck MRI, PET–CT, endoscopy Biopsies: SCC | cT2N0M0 OSCC | Primary: TORS and ipsi-lateral neck dissection or RT Alternative: CTh |
9 | M | 70 | Dysphonia (4 months) | Current TC Radiotherapy for cT3 Supraglottic cancer | Exophytic lesion of the left vocal cord and anterior commissure of the larynx | Neck CT, PET–CT, endoscopy | cT2N0M0 Supraglottic LSCC (recurrence) | Primary: partial laryngectomy Alternative: CTh/immunotherapy. |
10 | M | 50 | Neck mass (6 months) | None | Right neck node 2cm Endoscopy: normal | Neck MRI, PET–CT, endoscopy Fine-needle aspiration biopsy | cTxN1M0 SCC | Primary: TORS tonsillectomy Neck dissection, no post-RT or RT Alternative: CTh |
11 | F | 49 | Throat pain (4 months) | None | Ulceration lesion in left tonsil and left neck adenopathy | Neck MRI, PET–CT Biopsies: SCC HPV detection | cT2N1M0 OSCC cT2N0M0 | Primary: TORS tonsillectomy neck dissection, no post-RT or RT Alternative: CTh |
12 | M | 62 | Left parotid gland nodules | HT, left superficial parotidectomy for pleomorphic adenoma (10 years) | Left parotid node, no facial palsy, no adenopathy | Neck MRI Fine-needle aspiration biopsy HPV detection | Parotid carcinoma | Primary: parotidectomy neck dissection, postoperative RT or RT Alternative: CRT |
13 | F | 55 | Right facial nerve paralysis, nodules (3 weeks) | Parotidectomy and RT for a right adeno-carcinoma (4 years) | Right facial nerve paralysis Parotid region nodules No lymph adenopathy | Neck MRI, PET–CT Chest CT Fine-needle aspiration biopsy | cT1N0M0 Recurrence of Adenocarcinoma | Primary: parotidectomy free flap, reinnervation Alternative: CTh/immunotherapy. |
14 | M | 55 | Right EU-Tirads 5 thyroid nodule (8 months) | None | Normal, endoscopy normal No vocal fold paralysis | Fine-needle aspiration biopsy Ultrasonography | cT1N0M0 papillary carcinoma | Primary: lobectomy Alternative: iodine |
15 | M | 55 | Right sublingual gland nodule (6 months) | HT, current TC | Right sublingual nodule, Examination: normal | Neck MRI Fine-needle aspiration | cT1N0M0 muco-epidermoid carcinoma | Primary: sublingual surgery, low grade cancer, no postoperative RT Alternative: RT |
16 | F | 51 | Oral cavity pain, and tongue ulceration (6 months) | Current TC | Right tongue ulceration of 1cm (latero-posterior) | Neck/oral MRI, PET–CT Endoscopy and biopsy | cT1N0M0 Oral SCC | Primary: partial glossectomy Sentinel node dissection or RT Alternative: CTh |
17 | M | 53 | Oral cavity pain, ulceration of inferior tongue part and oral cavity floor (7 months) | HT, current ATC | Left tongue and oral cavity floor lesion, left neck nodes | Neck MRI, CT, PET–CT Biopsies: SCC HPV detection | cT2N2aM0 Oral SCC | Primary: partial glossectomy, neck dissection, FAMM, or RT Alternative: CTh |
18 | F | 45 | Epistaxis, diplopia, Right deafness, neck nodes (5 months) | None | Right exophytic nasopharyngeal lesion and right chronic otitis Multiple cervical nodes | MRI, Neck CT, PET–CT Biopsy Audiometry, tympanometry | cT4N2cM0 UCNT | Primary: induction CTh and CRT Alternative: CRT |
19 | F | 58 | Aphagia, weight loss throat pain (6 months) | Past TC, CRT for a cT4N2M0 HSCC (2 years) | Laryngopharyngeal edema and saliva, no neck node | Neck CT, MRI, PET–CT Biopsies: SCC | pT2N0M0 HLSCC (recurrence) | Primary: salvage pharyngo-laryngectomy and neck dissections Alternative: CTh/immunotherapy. |
20 | M | 67 | Unilateral epistaxis and obstruction, diplopia | HT | Exophytic lesion in right nasal cavity, no neck node | Nasal CT, MRI, PET–CT Biopsy | cT2N0M0 Ethmoid Intestinal-type Adenocarcinoma | Primary: endoscopic nasal surgery, postoperative RT Alternative: CRT |
Appendix 2: Artificial intelligence performance instrument scores of ChatGPT-4
AIPI outcomes | Mean (SD) |
|---|---|
1. Medical and Surgical History (/2) | 1.55 ± 0.76 |
2. Symptoms (/2) | 1.95 ± 0.22 |
3. Physical examinations (/2) | 1.80 ± 0.41 |
Patient feature score (/6) | 5.30 ± 1.08 |
4. Differential diagnoses (/3) | 3.00 ± 0.01 |
5. Primary diagnosis (/3) | 3.00 ± 0.01 |
6. Management plan (/1) | 0.80 ± 0.41 |
Diagnostic score (/7) | 6.80 ± 0.41 |
7. Additional examinations (/3) | 2.05 ± 0.69 |
8. Most relevant additional examination (/1) | 0.90 ± 0.31 |
Additional examination score (/4) | 2.95 ± 0.83 |
9. Treatment (/3) | 2.30 ± 0.87 |
10. AIPI total score (/20) | 17.35 ± 2.32 |
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lechien, J.R., Chiesa-Estomba, CM., Baudouin, R. et al. Accuracy of ChatGPT in head and neck oncological board decisions: preliminary findings. Eur Arch Otorhinolaryngol 281, 2105–2114 (2024). https://doi.org/10.1007/s00405-023-08326-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00405-023-08326-w