Abstract
To assess ChatGPT 4.0 (ChatGPT) and Gemini Ultra 1.0 (Gemini) large language models on NONMEM coding tasks relevant to pharmacometrics and clinical pharmacology. ChatGPT and Gemini were assessed on tasks mimicking real-world applications of NONMEM. The tasks ranged from providing a curriculum for learning NONMEM, an overview of NONMEM code structure to generating code. Prompts in lay language to elicit NONMEM code for a linear pharmacokinetic (PK) model with oral administration and a more complex model with two parallel first-order absorption mechanisms were investigated. Reproducibility and the impact of “temperature” hyperparameter settings were assessed. The code was reviewed by two NONMEM experts. ChatGPT and Gemini provided NONMEM curriculum structures combining foundational knowledge with advanced concepts (e.g., covariate modeling and Bayesian approaches) and practical skills including NONMEM code structure and syntax. ChatGPT provided an informative summary of the NONMEM control stream structure and outlined the key NONMEM Translator (NM-TRAN) records needed. ChatGPT and Gemini were able to generate code blocks for the NONMEM control stream from the lay language prompts for the two coding tasks. The control streams contained focal structural and syntax errors that required revision before they could be executed without errors and warnings. The code output from ChatGPT and Gemini was not reproducible, and varying the temperature hyperparameter did not reduce the errors and omissions substantively. Large language models may be useful in pharmacometrics for efficiently generating an initial coding template for modeling projects. However, the output can contain errors and omissions that require correction.
Similar content being viewed by others
References
ChatGPT Version Jan 2024 (2024) https://chat.openai.com/. Accessed Mar 2024
Gemini Team, Anil R, Borgeaud S et al (2023) Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805. https://doi.org/10.48550/arXiv.2312.11805. Accessed Dec 01 2023. https://ui.adsabs.harvard.edu/abs/2023arXiv231211805G
Llama 2: open source, free for research and commercial use (2024) https://llama.meta.com/llama2/
Touvron H, Martin L, Stone K, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023: arXiv:2307.09288. https://doi.org/10.48550/arXiv.2307.09288. Accessed Jul 01 2023. https://ui.adsabs.harvard.edu/abs/2023arXiv230709288T
Meet Claude (2024) https://www.anthropic.com/claude
Orru G, Piarulli A, Conversano C, Gemignani A (2023) Human-like problem-solving abilities in large language models using ChatGPT. Front Artif Intell 6:1199350. https://doi.org/10.3389/frai.2023.1199350
Roumeliotis KI, Tselikas ND (2023) ChatGPT and Open-AI Models: A Preliminary Review. Future Internet 15(6):192
Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training
Owen JS, Fiedler-Kelly J (2014) Introduction to population pharmacokinetic/pharmacodynamic analysis with nonlinear mixed effects models. Wiley
Pétricoul O, Cosson V, Fuseau E, Marchand M (2007) Population models for drug absorption and enterohepatic recycling. Pharmacometrics: the science of quantitative pharmacology 345–382
Bauer RJ (2019) NONMEM tutorial part I: description of commands and options, with simple examples of population analysis. CPT Pharmacometrics Syst Pharmacol 8(8):525–537. https://doi.org/10.1002/psp4.12404
Sun H, Fadiran EO, Jones CD et al (1999) Population pharmacokinetics: a regulatory perspective. Clin Pharmacokinet 37:41–58
Cloesmeijer ME, Janssen A, Koopman SF, Cnossen MH, Mathôt RA, consortium S, (2024) ChatGPT in pharmacometrics? Potential opportunities and limitations. British J Clin Pharmacol 90(1):360–365
Shin E, Ramanathan M (2024) Evaluation of prompt engineering strategies for pharmacokinetic data analysis with the ChatGPT large language model. J Pharmacokinet Pharmacodyn 51(2):101–108
Bard Large language model. 2023. https://bard.google.com
Fidler M, Wilkins JJ, Hooijmaijers R et al (2019) Nonlinear Mixed-Effects Model Development and Simulation Using nlmixr and Related R Open-Source Packages. CPT Pharmacometrics Syst Pharmacol 8(9):621–633. https://doi.org/10.1002/psp4.12445
Bonate PL, Barrett JS, Ait-Oudhia S et al (2023) Training the next generation of pharmacometric modelers: a multisector perspective. J Pharmacokinet Pharmacodyn. https://doi.org/10.1007/s10928-023-09878-4
Shin E, Ramanathan M (2023) Evaluation of prompt engineering strategies for pharmacokinetic data analysis with the ChatGPT large language model. J Pharmacokinet Pharmacodyn. https://doi.org/10.1007/s10928-023-09892-6
mrgsolve: Simulate from ODE-Based Models. R package version 1.4.1. Metrum Research Group; 2024. https://github.com/metrumresearchgroup/mrgsolve
Fidler M, Hooijmaijers R, Schoemaker R, Wilkins JJ, Xiong Y, Wang W (2021) R and nlmixr as a gateway between statistics and pharmacometrics. CPT Pharmacometrics Syst Pharmacol 10(4):283–285. https://doi.org/10.1002/psp4.12618
nlmixr: an R package for population PKPD modeling. 2019. https://nlmixrdevelopment.github.io/nlmixr/index.html
Anonymous. Monolix documentation. Lixoft-SimulationsPlus. Accessed March 13, 2024, 2024. https://monolix.lixoft.com/single-page/
Stan Reference Manual. NumFOCUS; 2011. https://mc-stan.org/docs/reference-manual/
Cloesmeijer ME, Janssen A, Koopman SF, Cnossen MH, Mathot RAA, Symphony consortium (2024) ChatGPT in pharmacometrics? Potential opportunities and limitations. Br J Clin Pharmacol 90(1):360–365. https://doi.org/10.1111/bcp.15895
Frieder S, Pinchetti L, Chevalier A et al (2023) Mathematical capabilities of ChatGPT. arXiv. arXiv:2301.13867v2
Yuan Z, Yuan H, Tan C, Wang W, Huang S. How well do large language models perform in arithmetic tasks? arXiv. 2023:arXiv:2304.02015
Alkaissi H, McFarlane SI (2023) Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus J Med Sci 15(2).https://doi.org/10.7759/cureus.35179
Beutel G, Geerits E, Kielstein JT (2023) Artificial hallucination: GPT on LSD? Crit Care 27(1):148. https://doi.org/10.1186/s13054-023-04425-6
Funding
This is unfunded research. Support from Grant MS190096 from the Department of Defense Multiple Sclerosis Research Program for the Office of the Congressionally Directed Medical Research Programs (CDMRP) to the Ramanathan laboratory is gratefully acknowledged.
Author information
Authors and Affiliations
Contributions
Euibeom Shin – Data analysis, manuscript preparation.
Yifan Yu – Data analysis, manuscript preparation.
Robert Bies – Manuscript preparation.
Murali Ramanathan – Study concept and design, data analysis, manuscript preparation.
Corresponding author
Ethics declarations
Financial conflicts
Dr. Bies receives grant funding from NIDA, NIAID, NICHD, USAID, Bill and Melinda Gates Foundation and serves as a consultant for Advanced Biosciences Laboratories (NIAID). In the past he has served as a consultant for Lumos Biopharma through NGT Biopharma Consultants. Dr. Murali Ramanathan received research funding from the National Multiple Sclerosis Society, Department of Defense, National Science Foundation, and National Institute of Neurological Diseases and Stroke. He receives royalty from a self-published textbook.
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Shin, E., Yu, Y., Bies, R.R. et al. Evaluation of ChatGPT and Gemini large language models for pharmacometrics with NONMEM. J Pharmacokinet Pharmacodyn (2024). https://doi.org/10.1007/s10928-024-09921-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10928-024-09921-y