Skip to main content

Advertisement

SpringerLink
Log in

Examining the Potential of ChatGPT on Biomedical Information Retrieval: Fact-Checking Drug-Disease Associations

  • Letter to the Editor
  • Published:
Annals of Biomedical Engineering Aims and scope Submit manuscript

Abstract

Large language models (LLMs) such as ChatGPT have recently attracted significant attention due to their impressive performance on many real‐world tasks. These models have also demonstrated the potential in facilitating various biomedical tasks. However, little is known of their potential in biomedical information retrieval, especially identifying drug-disease associations. This study aims to explore the potential of ChatGPT, a popular LLM, in discerning drug-disease associations. We collected 2694 true drug-disease associations and 5662 false drug-disease pairs. Our approach involved creating various prompts to instruct ChatGPT in identifying these associations. Under varying prompt designs, ChatGPT’s capability to identify drug-disease associations with an accuracy of 74.6–83.5% and 96.2–97.6% for the true and false pairs, respectively. This study shows that ChatGPT has the potential in identifying drug-disease associations and may serve as a helpful tool in searching pharmacy-related information. However, the accuracy of its insights warrants comprehensive examination before its implementation in medical practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Liu, J., C. Wang, and S. Liu. Utility of ChatGPT in clinical practice. J. Med. Internet Res.25:e48568, 2023. https://doi.org/10.2196/48568.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Takagi, S., T. Watari, A. Erabi, and K. Sakaguchi. Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study. JMIR Med. Educ.9:e48002, 2023. https://doi.org/10.2196/48002.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Zhao, W. X., et al. A survey of large language models. 2023. https://doi.org/10.48550/ARXIV.2303.18223

  4. Eggmann, F., R. Weiger, N. U. Zitzmann, and M. B. Blatz. Implications of large language models such as ChatGPT for dental medicine. J. Esthet. Restor. Dent. 2023. https://doi.org/10.1111/jerd.13046.

    Article  PubMed  Google Scholar 

  5. Thirunavukarasu, A. J., D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting. Large language models in medicine. Nat. Med. 29(8):1930–1940, 2023. https://doi.org/10.1038/s41591-023-02448-8.

    Article  CAS  PubMed  Google Scholar 

  6. OpenAI. Models. 2023. https://platform.openai.com/docs/models/gpt-3-5

  7. Gilson, A., et al. How does ChatGPT perform on the united states medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med. Educ.9:e45312, 2023. https://doi.org/10.2196/45312.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Walker, H. L., et al. Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument. J. Med. Internet Res.25:e47479, 2023. https://doi.org/10.2196/47479.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Gu, Y., et al. Distilling large language models for biomedical knowledge extraction: a case study on adverse drug events. 2023. https://doi.org/10.48550/ARXIV.2307.06439.

  10. Jahan, I., M. T. R. Laskar, C. Peng, and J. Huang. Evaluation of ChatGPT on biomedical tasks: a zero-shot comparison with fine-tuned generative transformers. 2023. https://doi.org/10.48550/ARXIV.2306.04504.

  11. Juhi, A., N. Pipil, S. Santra, S. Mondal, J. K. Behera, and H. Mondal. The capability of ChatGPT in predicting and explaining common drug-drug interactions. Cureus. 2023. https://doi.org/10.7759/cureus.36272.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Chen, S., et al. Use of artificial intelligence Chatbots for cancer treatment information. JAMA Oncol. 2023. https://doi.org/10.1001/jamaoncol.2023.2954.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Eysenbach, G. The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers. JMIR Med. Educ.9:e46885, 2023. https://doi.org/10.2196/46885.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Miao, H., and H. Ahn. Impact of ChatGPT on interdisciplinary nursing education and research. Asian Pac. Isl. Nurs. J.7:e48136, 2023. https://doi.org/10.2196/48136.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Wang, Q., Z. Gao, and R. Xu. Exploring the in-context learning ability of large language model for biomedical concept linking. 2023. https://doi.org/10.48550/ARXIV.2307.01137.

  16. Dave, T., S. A. Athaluri, and S. Singh. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front. Artif. Intell. 6:1169595, 2023. https://doi.org/10.3389/frai.2023.1169595.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Drees, J. Google receives more than 1 billion health questions every day. Mar. 11, 2019. https://www.beckershospitalreview.com/healthcare-information-technology/google-receives-more-than-1-billion-health-questions-every-day.html. Accessed 23 Aug 2023.

  18. Ayoub, N. F., Y.-J. Lee, D. Grimm, and K. Balakrishnan. Comparison between ChatGPT and google search as sources of postoperative patient instructions. JAMA Otolaryngol. Head Neck Surg. 149(6):556, 2023. https://doi.org/10.1001/jamaoto.2023.0704.

    Article  PubMed  Google Scholar 

  19. Xu, R., Y. Feng, and H. Chen. ChatGPT vs. Google: a comparative study of search performance and user experience. 2023. https://doi.org/10.48550/ARXIV.2307.01135.

  20. Dudley, J. T., T. Deshpande, and A. J. Butte. Exploiting drug-disease relationships for computational drug repositioning. Brief. Bioinform. 12(4):303–311, 2011. https://doi.org/10.1093/bib/bbr013.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Avram, S., et al. DrugCentral 2021 supports drug discovery and repositioning. Nucleic Acids Res. 49(D1):D1160–D1169, 2021. https://doi.org/10.1093/nar/gkaa997.

    Article  CAS  PubMed  Google Scholar 

  22. Wang, Q., and R. Xu. Drug repositioning for prostate cancer: using a data-driven approach to gain new insights. AMIA Annu. Symp. Proc. 2017:1724–1733, 2017.

    PubMed  Google Scholar 

  23. Wang, Q., and R. Xu. Disease comorbidity-guided drug repositioning: a case study in schizophrenia. AMIA Annu. Symp. Proc. 2018:1300–1309, 2018.

    PubMed  PubMed Central  Google Scholar 

  24. White, J., et al. A prompt pattern catalog to enhance prompt engineering with ChatGPT. 2023. https://doi.org/10.48550/ARXIV.2302.11382.

  25. Li, L., L. Fan, S. Atreja, and L. Hemphill. ‘HOT’ ChatGPT: the promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media. 2023. https://doi.org/10.48550/ARXIV.2304.10619.

  26. Saravia, E. Prompt engineering guide. https://github.com/dair-ai/Prompt-Engineering-Guide. Accessed 5 Aug 2023

  27. Wei, J., et al. Finetuned language models are zero-shot learners. 2021. https://doi.org/10.48550/ARXIV.2109.01652.

  28. Xian, Y., B. Schiele, and Z. Akata. Zero-shot learning—the good, the bad and the ugly. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI: IEEE, 2017, pp. 3077–3086. https://doi.org/10.1109/CVPR.2017.328.

  29. Brown, T. B., et al. Language models are few-shot learners. 2020. https://doi.org/10.48550/ARXIV.2005.14165.

  30. Wang, Y., Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from a few examples: a survey on few-shot learning. ACM Comput. Surv. 53(3):1–34, 2021. https://doi.org/10.1145/3386252.

    Article  Google Scholar 

  31. Google AI. Google AI PaLM 2. Google AI 2023. https://ai.google/discover/palm2/ Accessed 6 Aug. 2023.

  32. Meta, A.I. Introducing LLaMA: a foundational, 65-billion-parameter large language model. https://ai.facebook.com/blog/large-language-model-llama-meta-ai/. Accessed 24 Feb 2023

  33. Chen, L., M. Zaharia, and J. Zou. How is ChatGPT’s behavior changing over time? 2023. https://doi.org/10.48550/ARXIV.2307.09009.

  34. Natalie. What is ChatGPT?|OpenAI Help Center. 2023. https://help.openai.com/en/articles/6783457-what-is-chatgpt. Accessed 6 Aug 2023.

  35. Zuccon, G., and B. Koopman. Dr ChatGPT, tell me what I want to hear: how prompt knowledge impacts health answer correctness. 2023. https://doi.org/10.48550/ARXIV.2302.13793.

  36. Xu, R., and Q. Wang. Large-scale extraction of accurate drug-disease treatment pairs from biomedical literature for drug repurposing. BMC Bioinform. 14(1):1–11, 2013.

    Article  Google Scholar 

  37. Han, R., T. Peng, C. Yang, B. Wang, L. Liu, X. Wan. Is information extraction solved by ChatGPT? An analysis of performance, evaluation criteria, robustness and errors. 2023. https://doi.org/10.48550/arXiv.2305.14450.

  38. Brinkmann, A., R. Shraga, R. C. Der, C. Bizer. Product information extraction using ChatGPT. 2023. https://doi.org/10.48550/arXiv.2306.14921.

  39. Li, B., G. Fang, Y. Yang, Q. Wang, W. Ye, W. Zhao, S. Zhang. Evaluating ChatGPT's information extraction capabilities: an assessment of performance, explainability, calibration, and faithfulness. 2023. https://doi.org/10.48550/arXiv.2304.11633.

Download references

Funding

Funding was provided by National Institute on Alcohol Abuse and Alcoholism (AA029831), National Institute on Aging (AG057557, AG061388, AG062272, AG07664), National Eye Institute (EY029297), and National Institute on Drug Abuse (UG1DA049435, CTN-0114).

Author information

Author notes

  1. Zhenxiang Gao, Lingyao Li, and Siyuan Ma have contributed equally to this paper.

Authors and Affiliations

  1. Center for Artificial Intelligence in Drug Discovery, School of Medicine, Case Western Reserve University, Cleveland, OH, USA

    Zhenxiang Gao, Qinyong Wang & Rong Xu

  2. School of Information, University of Michigan, Ann Arbor, MI, USA

    Lingyao Li & Libby Hemphill

  3. Vanderbilt University Medical Center, Nashville, TN, USA

    Siyuan Ma

Authors
  1. Zhenxiang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lingyao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Siyuan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qinyong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Libby Hemphill
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rong Xu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Associate Editor Joel Stitzel oversaw the review of this article.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 269 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, Z., Li, L., Ma, S. et al. Examining the Potential of ChatGPT on Biomedical Information Retrieval: Fact-Checking Drug-Disease Associations. Ann Biomed Eng (2023). https://doi.org/10.1007/s10439-023-03385-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10439-023-03385-w

Keywords

Search

Navigation