Conditional Non-parametric Bootstrap for Non-linear Mixed Effect Models



Non-linear mixed effect models are widely used and increasingly integrated into decision-making processes. Propagating uncertainty is an important element of this process, and while standard errors (SE) on pa- rameters are most often computed using asymptotic approaches, alternative methods such as the bootstrap are also available. In this article, we propose a modified residual parametric bootstrap taking into account the different levels of variability involved in these models.


The proposed approach uses samples from the individual conditional distribution, and was implemented in R using the saemix algorithm. We performed a simulation study to assess its performance in different scenarios, comparing it to the asymptotic approximation and to standard bootstraps in terms of coverage, also looking at bias in the parameters and their SE.


Simulations with an Emax model with different designs and sigmoidicity factors showed a similar coverage rate to the parametric bootstrap, while requiring less hypotheses. Bootstrap improved coverage in several scenarios compared to the asymptotic method especially for the variance param-eters. However, all bootstraps were sensitive to estimation bias in the original datasets.


The conditional bootstrap provided better coverage rate than the traditional residual bootstrap, while preserving the structure of the data generating process.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3


  1. 1.

    Lalonde RL, Kowalski KG, Hutmacher MM, Ewy W, Nichols DJ, Mil-ligan PA, et al. Model-based drug development Clin Pharmacol Ther. 2007;82:21–32.

    CAS  PubMed  Google Scholar 

  2. 2.

    Wang Y, Zhu H, Madabushi R, Liu Q, Huang SM, Zineh I. Model-informed drug development: current US regulatory practice and future considerations. Clinical Pharmacology & Therapeutics. 2019;105(4):899–911.

    Article  Google Scholar 

  3. 3.

    Desḿee S, Mentŕe F, Veyrat-Follet C, Śebastien B, Guedj J. Nonlinear joint models for individual dynamic prediction of risk of death using Hamiltonian Monte Carlo: application to metastatic prostate cancer. BMC Med Res Methodol. 2017;17:105.

    Article  Google Scholar 

  4. 4.

    Retout S, Mentŕe F, Bruno R. Fisher information matrix for nonlinear mixed-effects models: evaluation and application for optimal de- sign of enoxaparin population pharmacokinetics. Stat Med. 2002;21(18):2623–39.

    Article  Google Scholar 

  5. 5.

    Gelman A, Carlin J, Stern HS, Dunson DB, Vehtari A, Rubin DB. Bayesian Data Analysis: Chapman & Hall /CRC Press; 2013.

  6. 6.

    Dartois C, Lemenuel-Diot A, Laveille C, Tranchand B, Tod M, Girard P. Evaluation of uncertainty parameters estimated by different population PK software and methods. J Pharmacokinet Pharmacodyn. 2007;34:289–31.

    Article  Google Scholar 

  7. 7.

    Lindstrom MJ, Bates DM. Nonlinear mixed effects models for repeated measures data. Biometrics. 1990;46:673–87.

    CAS  Article  Google Scholar 

  8. 8.

    Dosne AG, Niebecker R, Karlsson MO. dOFV distributions: a new diagnostic for the adequacy of parameter uncertainty in nonlinear mixed effects models applied to the bootstrap. J Pharmacokinet Pharmacodyn. 2016;43:597–608.

    Article  Google Scholar 

  9. 9.

    Thai H, Mentŕe F, Holford NH, Veyrat-Follet C, Comets E. A comparison of bootstrap approaches for estimating uncertainty of parameters in linear mixed-effects models. Pharm Stat. 2013;12:129–40.

    Article  Google Scholar 

  10. 10.

    Efron B. Bootstrap methods: another look at the jacknife. Ann Stat. 1979;7:1–26.

    Article  Google Scholar 

  11. 11.

    Kümmel A, Peter L, Bonate JD, Krause A. Confidence and prediction intervals for Pharmacometric models. CPT Pharmacometrics Syst Pharmacol. 2018;7:360–73.

    Article  Google Scholar 

  12. 12.

    Efron B, Tibshirani RJ. An introduction to the bootstrap: Chapman & Hall; 1994.

  13. 13.

    Das S, Krishen A. Some bootstrap methods in non-linear mixed-effects models. Journal of Statistical Planning and Inference. 1999;75:237–45.

    Article  Google Scholar 

  14. 14.

    Carpenter JR, Goldstein H, Rasbash J. A novel bootstrap procedure for assessing the relationship between class size and achievement. Appl Stat. 2003;52:431–43.

    Google Scholar 

  15. 15.

    Thai H, Mentŕe F, Holford NH, Veyrat-Follet C, Comets E. Evaluation of bootstrap methods for estimating uncertainty of parameters in nonlinear mixed-effects models: a simulation study in population pharmacokinetics. J Pharmacokinet Pharmacodyn. 2014;41:15–33.

    Article  Google Scholar 

  16. 16.

    Lavielle M, Ribba B. Enhanced method for diagnosing pharmacometric models: random sampling from conditional distributions. Pharm Res. 2016;33:2979–88.

    CAS  Article  Google Scholar 

  17. 17.

    Plan E, Maloney A, Mentŕe F, Karlsson M, Bertrand J. Performance comparison of various maximum likelihood nonlinear mixed-effects estimation methods for dose-response models. AAPS J. 2012;14:420–32.

    Article  Google Scholar 

  18. 18.

    Lavielle M. Mixed effects models for the population approach: models, tasks, methods and tools: Chapman & Hall CRC Bio- statistics Series; 2014.

  19. 19.

    Kuhn E, Lavielle M. Maximum likelihood estimation in nonlinear mixed effects models. Computational Statistics and Data Analysis. 2005;49:1020–38.

    Article  Google Scholar 

  20. 20.

    R Development Core Team. R: A Language and Environment for Statis- tical Computing. Vienna, Austria; 2006. ISBN 3–900051–07-0. Available from:

  21. 21.

    Comets E, Lavenu A, Lavielle M. Parameter estimation in nonlinear mixed effect models using saemix, an R implementation of the SAEM algorithm. J Stat Softw. 2017;80:1–41.

    Article  Google Scholar 

  22. 22.

    Karlsson M, Savic R. Diagnosing model diagnostics. Clin Pharmacol Ther. 2007;82:17–20.

    CAS  Article  Google Scholar 

  23. 23.

    Combes FP, Retout S, Frey N, Mentŕe F. Powers of the likelihood ratio test and the correlation test using empirical bayes estimates for various shrinkages in population pharmacokinetics. CPT: Pharmacometrics and Systems Pharmacology. 2014;3:e109.

    CAS  PubMed  Google Scholar 

  24. 24.

    Broeker A, Wicha SG. Assessing parameter uncertainty in small-n pharmacometric analyses: value of the log-likelihood profiling-based sampling importance resampling (LLP-SIR) technique. Journal of Pharmacokinetics and Pharmacodynamics. 2020;47:219–28.

    Article  Google Scholar 

  25. 25.

    Dumont C, Lestini G, Le Nagard H, Mentŕe F, Comets E, Nguyen T, et al. PFIM 4.0, an extended R program for design evaluation and optimization in nonlinear mixed-effect models. Computer Methods and Programs Biomedicine. 2018;156:217–29.

    Article  Google Scholar 

  26. 26.

    Buatois S, Ueckert S, Frey N, Retout S, Mentŕe F. Comparison of model averaging and model selection in dose finding trials analyzed by nonlinear mixed effect models. AAPS J. 2018;20:56.

    Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Emmanuelle Comets.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information


(PDF 1305 kb)

Acknowledgments and Disclosures

Conceptualisation, E.C.; formal analysis and simulations, E.C. and C.R.; methodology, M.U. and E.C.; software: E.C. and C.R.; visualisation, E.C. and M.U.; writing, original draft, C.R. and E.C.; writing, review and editing, E.C., M.U. C.R. and V.J. This work was performed during the authors’ research time and no other funding was used. All data generated or analysed during this study are included in this published article and its supplementary information files (see Appendix A) and are available on Zenodo (

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Comets, E., Rodrigues, C., Jullien, V. et al. Conditional Non-parametric Bootstrap for Non-linear Mixed Effect Models. Pharm Res (2021).

Download citation

Key Words

  • bootstrap
  • conditional distribution
  • non-linear mixed effect models
  • uncertainty of parameter estimates