Skip to main content
SpringerLink
Log in

Developing Tools to Evaluate Non-linear Mixed Effect Models: 20 Years on the npde Adventure

  • Review Article
  • Theme: Celebrating Women in the Pharmaceutical Sciences
  • Published:
The AAPS Journal Aims and scope Submit manuscript

Abstract

This article revisits 20 years of our work in developing evaluation tools adapted to non-linear mixed effect models. These hierarchical models involve a large number of assumptions concerning the structural evolution of the outcomes, the link between different outcomes, the variabilities in the parameters and model evaluation aims at assessing these various components, both to help guide the model building and to communicate on model adequacy for a given purpose. During our career, we have developed and extended simulation-based evaluation tools called normalised prediction discrepancies (npd) and normalised prediction distribution errors (npde), providing informative diagnostics through graphs and tests.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Sheiner LB, Rosenberg B, Marathe VV. Estimation of population characteristics of pharmacokinetic parameters from routine clinical data. J Pharmacokin Biopharm. 1977;5(5):445–79.

    Article  CAS  Google Scholar 

  2. Mentré F. Lewis Sheiner ISoP/UCSF lecturer award: from drug use to statistical models and vice versa. CPT Pharmacometrics Syst Pharmacol. 2014;3:e154.

    Article  Google Scholar 

  3. Beal S, Sheiner L, Boeckmann A, Bauer R. NONMEM Version 7.2. Ellicott City; 1989-2011.

  4. Lavielle M. Mixed effects models for the population approach: models, tasks, methods and tools. Chapman & Hall/CRC Biostatistics Series; 2014.

  5. Carpenter B, Gelman A, Hoffman M, Lee D, Goodrich B, Betancourt M, et al. Stan: a probabilistic programming language. J Stat Softw. 2017;76:1–32.

    Article  Google Scholar 

  6. Pinheiro J, Bates D, DebRoy S, Sarkar D, R Core Team. nlme: linear and nonlinear mixed effects models; 2020. R package version 3.1–150. Available from: https://CRAN.R-project.org/package=nlme.

  7. Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw. 2015;67(1):1–48.

    Article  Google Scholar 

  8. Comets E, Lavenu A, Lavielle M. Parameter estimation in nonlinear mixed effect models using saemix, an R implementation of the SAEM algorithm. J Stat Softw. 2017;80:1–41.

    Article  Google Scholar 

  9. Fidler M, Xiong Y, Schoemaker R, Wilkins J, Trame M, Hooijmaijers R, et al.. nlmixr: nonlinear mixed effects models in population pharmacokinetics and pharmacodynamics; 2021. R package version 2.0.1. Available from: https://CRAN.R-project.org/package=nlmixr.

  10. Mentré F, Mallet A, Baccar D. Optimal design in random-effects regression models. Biometrika. 1984;84:429–42.

    Article  Google Scholar 

  11. Sheiner LB, Rosenberg B, Melmon KL. Modelling of individual pharmacokinetics for computer- aided drug dosage. Comput Biomed Res. 1972;5(5):441–59.

    Article  Google Scholar 

  12. Sheiner LB, Beal SL. Some suggestions for measuring predictive performance. J Pharmacokin Biopharm. 1981;9:503–12.

    Article  CAS  Google Scholar 

  13. European Medicines Agency. Guideline on reporting the results of population pharmacokinetic analysis (CHMP); 2007. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003067.pdf.

  14. Food and Drug Administration. Guidance for industry exposure-response relationships– study design, data analysis, and regulatory applications; 2019. Available from: https://www.fda.gov/media/128793/download.

  15. Karlsson M, Savic R. Diagnosing model diagnostics. Clin Pharmacol Ther. 2007;82:17–20.

    Article  CAS  Google Scholar 

  16. Mentré F, Escolano S. Prediction discrepancies for the evaluation of nonlinear mixed-effects models. J Pharmacokinet Pharmacodyn. 2006;33:345–67.

    Article  Google Scholar 

  17. Brendel K, Comets E, Laffont C, Laveille C, Mentré F. Metrics for external model evaluation with an application to the population pharmacokinetics of gliclazide. Pharm Res. 2006;23:2036–49.

    Article  CAS  Google Scholar 

  18. Mesnil F, Mentré F, Dubruc C, Thénot JP, Mallet A. Population pharmacokinetic analysis of mizolastine and validation from sparse data on patients using the nonparametric maximum likelihood method. J Pharmacokinet Pharmacodyn. 1998;26(2):133–61.

    Article  CAS  Google Scholar 

  19. Dunn PK, Smyth GK. Randomized quantile residuals. J Comput Graph Stat. 1996;5:236–44.

    Google Scholar 

  20. Brendel K, Comets E, Laffont C, Mentré F. Evaluation of different tests based on observations for external model evaluation of population analyses. J Pharmacokinet Pharmacodyn. 2010;37:49–65.

    Article  Google Scholar 

  21. Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J Pharmacokinet Pharmacodyn. 2001;28(2):171–92.

    Article  CAS  Google Scholar 

  22. Holford N. The visual predictive check—superiority to standard diagnostic (Rorschach) plots. PAGE 14. 2005;Abstr 738.

  23. Bergstrand M, Hooker A, Wallin J, Karlsson M. Prediction-corrected visual predictive checks for diagnosing nonlinear mixed-effects models. AAPS J. 2011;13:143–51.

    Article  Google Scholar 

  24. Hooker AC, Staatz CE, Karlsson MO. Conditional weighted residuals (CWRES): a model diagnostic for the FOCE method. Pharm Res. 2007;24(12):2187–97.

    Article  CAS  Google Scholar 

  25. Nyberg J, Bauer RJ, Hooker AC. Investigations of the weighted residuals in NONMEM 7. PAGE 10. 2010;Abstr 1883.

  26. Comets E, Brendel K, Mentré F. Computing normalised prediction distribution errors to evaluate nonlinear mixed-effect models: the npde add-on package for R. Comput Meth Prog Biomed. 2008;90:154–66.

    Article  Google Scholar 

  27. R Core Team. R: a language and environment for statistical computing. Vienna, Austria; 2015. Available from: https://www.R-project.org/.

  28. Keizer R, Karlsson M, Hooker A. Modeling and simulation workbench for NONMEM: tutorial on Pirana, PsN, and Xpose. CPT Pharmacometrics Syst Pharmacol. 2013;2:e50.

    Article  CAS  Google Scholar 

  29. Nguyen T, Mouksassi MS, Holford N, Al-Huniti N, Freedman I, Hooker A, et al. Model evaluation of continuous data pharmacometric models: metrics and graphics. CPT Pharmacometrics Syst Pharmacol. 2017;6(2):87–109.

    Article  CAS  Google Scholar 

  30. Bergstrand M, Karlsson M. Handling data below the limit of quantification in mixed effect models. AAPS J. 2009;11(2):371–80.

    Article  CAS  Google Scholar 

  31. Nguyen THT, Comets E, Mentré F. Extension of NPDE for evaluation of nonlinear mixed effect models in presence of data below the quantification limit with applications to HIV dynamic model. J Pharmacokinet Pharmacodyn. 2012;39(5):499–518.

    Article  Google Scholar 

  32. Cerou M, Lavielle M, Brendel K, Chenel M, Comets E. Development and performance of npde for the evaluation of time-to-event models. Pharm Res. 2018;35(2):30.

    Article  CAS  Google Scholar 

  33. Mallet A. A maximum likelihood estimation method for random coefficient regression models. Biometrika. 1986;73:645–56.

    Article  Google Scholar 

  34. Comets E, Brendel K, Mentré F. Model evaluation in nonlinear mixed effect models, with applications to pharmacokinetics. J Soc Fr Statistique. 2010;151:106–28.

    Google Scholar 

  35. Comets E, Nguyen THT, Mentré F. Additional features and graphs in the new npde library for R. PAGE 22. 2013;Abstr 2775.

  36. Cerou M, Peigné S, Chenel M, Comets E. Performance of npde for the evaluation of joint model with time to event data. PAGE 28. 2019;Abstr 8940.

  37. Wickham H. ggplot2: elegant graphics for data analysis. New York: Springer-Verlag; 2016. Available from: https://ggplot2.tidyverse.org

    Book  Google Scholar 

  38. Savic R, Barrail-Tran A, Duval X, Nembot G, Panhard X, Descamps D, et al. Effect of adherence as measured by MEMS, ritonavir boosting, and CYP3A5 genotype on atazanavir pharmacokinetics in treatment-naive HIV-infected patients. Clin Pharmacol Ther. 2012;92:575–83. [39] box G. science and statistics. J Am Stat Assoc. 1976;71:791–9.

    Article  Google Scholar 

  39. Brendel K, Dartois C, Comets E, Lemmenuel-Diot A, Laveille C, Tranchand B, et al. Are population PK and/or PD models adequately evaluated? A 2002 to 2004 literature survey. Clin Pharmacokin. 2007;46:221–34.

    Article  Google Scholar 

  40. Dartois C, Brendel K, Comets E, Laffont C, Laveille C, Tranchand B, et al. Overview of model building strategies in population PK/PD analyses: 2002 to 2004 literature survey. Br J Clin Pharmacol. 2007;64:603–12.

    Article  CAS  Google Scholar 

  41. Ioannidis J. Reproducible pharmacokinetics. J Pharmacokinet Pharmacodyn. 2019;46:111–6.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Université de Paris, INSERM, IAME, F-75006, Paris, France

    Emmanuelle Comets & France Mentré

  2. INSERM, CIC1414, Université Rennes-1, 35000, Rennes, France

    Emmanuelle Comets

Authors
  1. Emmanuelle Comets
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. France Mentré
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The two (female) authors of this article (F. Mentré and E. Comets) have been the main architects for the development of npde. The original idea of prediction discrepancies was proposed by France Mentré with her PhD student Florence Mesnil for non-parametric mixed effect models and extended to general parametric models with Sylvie Escolano. Emmanuelle Comets got involved in the definition of npde and co-supervised the PhD of Karl Brendel, who also conducted in collaboration with Céline Dartois a survey of how population PK and PD analyses were evaluated. The extension of npde to BLQ data was the methodological topic of Tram (Thi Huyen) Nguyen, another female PhD student supervised by France Mentré. Tram was also the first author for the collaborative white paper by the ISoP group (29). Emmanuelle Comets supervised the PhD of Marc Cerou, our second male PhD student on the topic, who extended npd to models involving time-to-event and categorical outcomes (manuscript under preparation). Finally, we also would like to acknowledge the contribution since January 2020 of our engineer, Romain Leroux, for the latest version of the npde library. France Mentré was awarded the prestigious Lewis Sheiner lecturer award from the University of California and the International Society of Pharmacometrics in 2013, recalling in her lecture the prominent part of model evaluation in her career (2).

Corresponding author

Correspondence to Emmanuelle Comets.

Additional information

Guest Editors: Diane Burgess, Marilyn Morris and Meena Subramanyam

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(PDF 806 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Comets, E., Mentré, F. Developing Tools to Evaluate Non-linear Mixed Effect Models: 20 Years on the npde Adventure. AAPS J 23, 75 (2021). https://doi.org/10.1208/s12248-021-00597-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1208/s12248-021-00597-7

KEY WORDS

Search

Navigation