Developing Tools to Evaluate Non-linear Mixed Effect Models: 20 Years on the npde Adventure


This article revisits 20 years of our work in developing evaluation tools adapted to non-linear mixed effect models. These hierarchical models involve a large number of assumptions concerning the structural evolution of the outcomes, the link between different outcomes, the variabilities in the parameters and model evaluation aims at assessing these various components, both to help guide the model building and to communicate on model adequacy for a given purpose. During our career, we have developed and extended simulation-based evaluation tools called normalised prediction discrepancies (npd) and normalised prediction distribution errors (npde), providing informative diagnostics through graphs and tests.

Graphical abstract

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6


  1. 1.

    Sheiner LB, Rosenberg B, Marathe VV. Estimation of population characteristics of pharmacokinetic parameters from routine clinical data. J Pharmacokin Biopharm. 1977;5(5):445–79.

    CAS  Article  Google Scholar 

  2. 2.

    Mentré F. Lewis Sheiner ISoP/UCSF lecturer award: from drug use to statistical models and vice versa. CPT Pharmacometrics Syst Pharmacol. 2014;3:e154.

    Article  Google Scholar 

  3. 3.

    Beal S, Sheiner L, Boeckmann A, Bauer R. NONMEM Version 7.2. Ellicott City; 1989-2011.

  4. 4.

    Lavielle M. Mixed effects models for the population approach: models, tasks, methods and tools. Chapman & Hall/CRC Biostatistics Series; 2014.

  5. 5.

    Carpenter B, Gelman A, Hoffman M, Lee D, Goodrich B, Betancourt M, et al. Stan: a probabilistic programming language. J Stat Softw. 2017;76:1–32.

    Article  Google Scholar 

  6. 6.

    Pinheiro J, Bates D, DebRoy S, Sarkar D, R Core Team. nlme: linear and nonlinear mixed effects models; 2020. R package version 3.1–150. Available from:

  7. 7.

    Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw. 2015;67(1):1–48.

    Article  Google Scholar 

  8. 8.

    Comets E, Lavenu A, Lavielle M. Parameter estimation in nonlinear mixed effect models using saemix, an R implementation of the SAEM algorithm. J Stat Softw. 2017;80:1–41.

    Article  Google Scholar 

  9. 9.

    Fidler M, Xiong Y, Schoemaker R, Wilkins J, Trame M, Hooijmaijers R, et al.. nlmixr: nonlinear mixed effects models in population pharmacokinetics and pharmacodynamics; 2021. R package version 2.0.1. Available from:

  10. 10.

    Mentré F, Mallet A, Baccar D. Optimal design in random-effects regression models. Biometrika. 1984;84:429–42.

    Article  Google Scholar 

  11. 11.

    Sheiner LB, Rosenberg B, Melmon KL. Modelling of individual pharmacokinetics for computer- aided drug dosage. Comput Biomed Res. 1972;5(5):441–59.

    Article  Google Scholar 

  12. 12.

    Sheiner LB, Beal SL. Some suggestions for measuring predictive performance. J Pharmacokin Biopharm. 1981;9:503–12.

    CAS  Article  Google Scholar 

  13. 13.

    European Medicines Agency. Guideline on reporting the results of population pharmacokinetic analysis (CHMP); 2007. Available from:

  14. 14.

    Food and Drug Administration. Guidance for industry exposure-response relationships– study design, data analysis, and regulatory applications; 2019. Available from:

  15. 15.

    Karlsson M, Savic R. Diagnosing model diagnostics. Clin Pharmacol Ther. 2007;82:17–20.

    CAS  Article  Google Scholar 

  16. 16.

    Mentré F, Escolano S. Prediction discrepancies for the evaluation of nonlinear mixed-effects models. J Pharmacokinet Pharmacodyn. 2006;33:345–67.

    Article  Google Scholar 

  17. 17.

    Brendel K, Comets E, Laffont C, Laveille C, Mentré F. Metrics for external model evaluation with an application to the population pharmacokinetics of gliclazide. Pharm Res. 2006;23:2036–49.

    CAS  Article  Google Scholar 

  18. 18.

    Mesnil F, Mentré F, Dubruc C, Thénot JP, Mallet A. Population pharmacokinetic analysis of mizolastine and validation from sparse data on patients using the nonparametric maximum likelihood method. J Pharmacokinet Pharmacodyn. 1998;26(2):133–61.

    CAS  Article  Google Scholar 

  19. 19.

    Dunn PK, Smyth GK. Randomized quantile residuals. J Comput Graph Stat. 1996;5:236–44.

    Google Scholar 

  20. 20.

    Brendel K, Comets E, Laffont C, Mentré F. Evaluation of different tests based on observations for external model evaluation of population analyses. J Pharmacokinet Pharmacodyn. 2010;37:49–65.

    Article  Google Scholar 

  21. 21.

    Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J Pharmacokinet Pharmacodyn. 2001;28(2):171–92.

    CAS  Article  Google Scholar 

  22. 22.

    Holford N. The visual predictive check—superiority to standard diagnostic (Rorschach) plots. PAGE 14. 2005;Abstr 738.

  23. 23.

    Bergstrand M, Hooker A, Wallin J, Karlsson M. Prediction-corrected visual predictive checks for diagnosing nonlinear mixed-effects models. AAPS J. 2011;13:143–51.

    Article  Google Scholar 

  24. 24.

    Hooker AC, Staatz CE, Karlsson MO. Conditional weighted residuals (CWRES): a model diagnostic for the FOCE method. Pharm Res. 2007;24(12):2187–97.

    CAS  Article  Google Scholar 

  25. 25.

    Nyberg J, Bauer RJ, Hooker AC. Investigations of the weighted residuals in NONMEM 7. PAGE 10. 2010;Abstr 1883.

  26. 26.

    Comets E, Brendel K, Mentré F. Computing normalised prediction distribution errors to evaluate nonlinear mixed-effect models: the npde add-on package for R. Comput Meth Prog Biomed. 2008;90:154–66.

    Article  Google Scholar 

  27. 27.

    R Core Team. R: a language and environment for statistical computing. Vienna, Austria; 2015. Available from:

  28. 28.

    Keizer R, Karlsson M, Hooker A. Modeling and simulation workbench for NONMEM: tutorial on Pirana, PsN, and Xpose. CPT Pharmacometrics Syst Pharmacol. 2013;2:e50.

    CAS  Article  Google Scholar 

  29. 29.

    Nguyen T, Mouksassi MS, Holford N, Al-Huniti N, Freedman I, Hooker A, et al. Model evaluation of continuous data pharmacometric models: metrics and graphics. CPT Pharmacometrics Syst Pharmacol. 2017;6(2):87–109.

    CAS  Article  Google Scholar 

  30. 30.

    Bergstrand M, Karlsson M. Handling data below the limit of quantification in mixed effect models. AAPS J. 2009;11(2):371–80.

    CAS  Article  Google Scholar 

  31. 31.

    Nguyen THT, Comets E, Mentré F. Extension of NPDE for evaluation of nonlinear mixed effect models in presence of data below the quantification limit with applications to HIV dynamic model. J Pharmacokinet Pharmacodyn. 2012;39(5):499–518.

    Article  Google Scholar 

  32. 32.

    Cerou M, Lavielle M, Brendel K, Chenel M, Comets E. Development and performance of npde for the evaluation of time-to-event models. Pharm Res. 2018;35(2):30.

    CAS  Article  Google Scholar 

  33. 33.

    Mallet A. A maximum likelihood estimation method for random coefficient regression models. Biometrika. 1986;73:645–56.

    Article  Google Scholar 

  34. 34.

    Comets E, Brendel K, Mentré F. Model evaluation in nonlinear mixed effect models, with applications to pharmacokinetics. J Soc Fr Statistique. 2010;151:106–28.

    Google Scholar 

  35. 35.

    Comets E, Nguyen THT, Mentré F. Additional features and graphs in the new npde library for R. PAGE 22. 2013;Abstr 2775.

  36. 36.

    Cerou M, Peigné S, Chenel M, Comets E. Performance of npde for the evaluation of joint model with time to event data. PAGE 28. 2019;Abstr 8940.

  37. 37.

    Wickham H. ggplot2: elegant graphics for data analysis. New York: Springer-Verlag; 2016. Available from:

    Book  Google Scholar 

  38. 38.

    Savic R, Barrail-Tran A, Duval X, Nembot G, Panhard X, Descamps D, et al. Effect of adherence as measured by MEMS, ritonavir boosting, and CYP3A5 genotype on atazanavir pharmacokinetics in treatment-naive HIV-infected patients. Clin Pharmacol Ther. 2012;92:575–83. [39] box G. science and statistics. J Am Stat Assoc. 1976;71:791–9.

    Article  Google Scholar 

  39. 39.

    Brendel K, Dartois C, Comets E, Lemmenuel-Diot A, Laveille C, Tranchand B, et al. Are population PK and/or PD models adequately evaluated? A 2002 to 2004 literature survey. Clin Pharmacokin. 2007;46:221–34.

    Article  Google Scholar 

  40. 40.

    Dartois C, Brendel K, Comets E, Laffont C, Laveille C, Tranchand B, et al. Overview of model building strategies in population PK/PD analyses: 2002 to 2004 literature survey. Br J Clin Pharmacol. 2007;64:603–12.

    CAS  Article  Google Scholar 

  41. 41.

    Ioannidis J. Reproducible pharmacokinetics. J Pharmacokinet Pharmacodyn. 2019;46:111–6.

    Article  Google Scholar 

Download references

Author information




The two (female) authors of this article (F. Mentré and E. Comets) have been the main architects for the development of npde. The original idea of prediction discrepancies was proposed by France Mentré with her PhD student Florence Mesnil for non-parametric mixed effect models and extended to general parametric models with Sylvie Escolano. Emmanuelle Comets got involved in the definition of npde and co-supervised the PhD of Karl Brendel, who also conducted in collaboration with Céline Dartois a survey of how population PK and PD analyses were evaluated. The extension of npde to BLQ data was the methodological topic of Tram (Thi Huyen) Nguyen, another female PhD student supervised by France Mentré. Tram was also the first author for the collaborative white paper by the ISoP group (29). Emmanuelle Comets supervised the PhD of Marc Cerou, our second male PhD student on the topic, who extended npd to models involving time-to-event and categorical outcomes (manuscript under preparation). Finally, we also would like to acknowledge the contribution since January 2020 of our engineer, Romain Leroux, for the latest version of the npde library. France Mentré was awarded the prestigious Lewis Sheiner lecturer award from the University of California and the International Society of Pharmacometrics in 2013, recalling in her lecture the prominent part of model evaluation in her career (2).

Corresponding author

Correspondence to Emmanuelle Comets.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Guest Editors: Diane Burgess, Marilyn Morris and Meena Subramanyam

Supplementary Information


(PDF 806 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Comets, E., Mentré, F. Developing Tools to Evaluate Non-linear Mixed Effect Models: 20 Years on the npde Adventure. AAPS J 23, 75 (2021).

Download citation


  • mixed effect models
  • model diagnostics
  • model evaluation
  • npde