Evaluation of Uncertainty Parameters Estimated by Different Population PK Software and Methods

  • Céline DartoisEmail author
  • Annabelle Lemenuel-Diot
  • Christian Laveille
  • Brigitte Tranchand
  • Michel Tod
  • Pascal Girard

The uncertainty associated with parameter estimations is essential for population model building, evaluation, and simulation. Summarized by the standard error (SE), its estimation is sometimes questionable. Herein, we evaluate SEs provided by different non linear mixed-effect estimation methods associated with their estimation performances. Methods based on maximum likelihood (FO and FOCE in NONMEMTM, nlme in SplusTM, and SAEM in MONOLIX) and Bayesian theory (WinBUGS) were evaluated on datasets obtained by simulations of a one-compartment PK model using 9 different designs. Bootstrap techniques were applied to FO, FOCE, and nlme. We compared SE estimations, parameter estimations, convergence, and computation time. Regarding SE estimations, methods provided concordant results for fixed effects. On random effects, SAEM and WinBUGS, tended respectively to under or over-estimate them. With sparse data, FO provided biased estimations of SE and discordant results between bootstrapped and original datasets. Regarding parameter estimations, FO showed a systematic bias on fixed and random effects. WinBUGS provided biased estimations, but only with sparse data. SAEM and WinBUGS converged systematically while FOCE failed in half of the cases. Applying bootstrap with FOCE yielded CPU times too large for routine application and bootstrap with nlme resulted in frequent crashes. In conclusion, FO provided bias on parameter estimations and on SE estimations of random effects. Methods like FOCE provided unbiased results but convergence was the biggest issue. Bootstrap did not improve SEs for FOCE methods, except when confidence interval of random effects is needed. WinBUGS gave consistent results but required long computation times. SAEM was in-between, showing few under-estimated SE but unbiased parameter estimations.


Uncertainty Standard error non-linear mixed effect model pharmaco-kinetics estimation method 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Meibohm B. and Derendorf H. (2002) Pharmacokinetic/pharmacodynamic studies in drug product development. J. Pharm. Sci. 91:18–31PubMedCrossRefGoogle Scholar
  2. 2.
    Sheiner L.B. and Steimer J.L. (2000) Pharmacokinetic/pharmacodynamic modeling in drug development. Annu. Rev. Pharmacol. Toxicol. 40:67–95PubMedCrossRefGoogle Scholar
  3. 3.
    Evans W.E., Relling M.V., Rodman J.H., Crom W.R., Boyett J.M., Pui C.H. (1998) Conventional compared with individualized chemotherapy for childhood acute lymphoblastic leukemia. N. Engl. J. Med. 338:499–505PubMedCrossRefGoogle Scholar
  4. 4.
    Jelliffe R.W. (1987) Clinical applications of pharmacokinetics and adaptive control. IEEE Trans. Biomed. Eng. 34:624–632PubMedCrossRefGoogle Scholar
  5. 5.
    Jelliffe R. W., A. Schumitzky, Van Liu G. M. M., Hu L., Maire P., Gomis P., Barbaut X., and Tahani B. (1993) Individualizing drug dosage regimens: roles of population pharmacokinetic and dynamic models, Bayesian fitting, and adaptive control. Ther. Drug Monit. 15:380–393.Google Scholar
  6. 6.
    Sheiner L.B., Rosenberg B., Melmon K.L. (1972) Modelling of individual pharmacokinetics for computer-aided drug dosage. Comput. Biomed. Res. 5:411–459PubMedCrossRefGoogle Scholar
  7. 7.
    Karlsson M.O. and Sheiner L.B. (1993) The importance of modeling interoccasion variability in population pharmacokinetic analyses. J. Pharmacokinet. Biopharm. 21:735–750PubMedCrossRefGoogle Scholar
  8. 8.
    S. Retout, S. Duffull, and F. Mentré. (2001) Development and implementation of the population Fisher information matrix for the evaluation of population pharmacokinetic designs. Comput. Methods Programs Biomed. 65:141–151.Google Scholar
  9. 9.
    Retout S. and Mentré F. (2003) Further developments of the Fisher information matrix in nonlinear mixed effects models with evaluation in population pharmacokinetics. J. Biopharm. Stat. 13:209–227PubMedCrossRefGoogle Scholar
  10. 10.
    Yano Y., Beal S.L., and Sheiner L.B. (2001) Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J. Pharmacokinet. Pharmacodyn. 28:171–192PubMedCrossRefGoogle Scholar
  11. 11.
    Food and Drug Administration. Guidance for industry – Population pharmacokinetics. Available from: Last Accessed: July 4 2006.Google Scholar
  12. 12.
    F. Mentré and M. E. Ebelin. Validation of population pharmacokinetic/pharmacodynamic analyses: review of proposed approaches. In: COST B1. European cooperation in the field of scientific and technical research. The population approach: measuring and managing variability in response, concentration and dose, L. Aarons, L. P. Balant, M. Danhof, M. Gex-Fabry, U. Gundert-Remy, M. Karlsson, F. Mentre, P. Morselli, M. Rowland, J. L. Steimer, S. Vozeh, and F. Rombout, eds. pp. 148–160, Office for official publications of the European Communities, Brussels.Google Scholar
  13. 13.
    Ette E.I., Williams P.J., Kim Y.H., Lane J.R., Liu M.J., and Capparelli E.V. (2003) Model appropriateness and population pharmacokinetic modeling. J. Clin. Pharmacol. 43:610–623PubMedGoogle Scholar
  14. 14.
    Tod M. and Rocchisani J.M. (1996) Implementation of OSPOP, an algorithm for the estimation of optimal sampling times in pharmacokinetics by the ED, EID and API criteria. Comput. Methods Programs Biomed. 50:13–22PubMedCrossRefGoogle Scholar
  15. 15.
    Tod M. and Rocchisani J.M. (1997) Comparison of ED, EID, and API criteria for the robust optimization of sampling times in pharmacokinetics. J Pharmacokinet. Biopharm. 25:515–537PubMedCrossRefGoogle Scholar
  16. 16.
    Matthews I., Kirkpatrick C., and Holford N. (2004) Quantitative justification for target concentration intervention–parameter variability and predictive performance using population pharmacokinetic models for aminoglycosides. Br. J. Clin. Pharmacol. 58:8–19PubMedCrossRefGoogle Scholar
  17. 17.
    Bonate P.L., Craig A., Gaynon P., Gandhi V., Jeha S., Kadota R., Lam G.N., Plunkett W., Razzouk B., Rytting M., Steinherz P., and Weitman S. (2004) Population pharmacokinetics of clofarabine, a second-generation nucleoside analog, in pediatric patients with acute leukemia. J. Clin. Pharmacol. 44:1309–1322PubMedCrossRefGoogle Scholar
  18. 18.
    Jolling K., Ruixo J.J., Hemeryck A., Piotrovskij V., and Greway T. (2004) Population pharmacokinetic analysis of pegylated human erythropoietin in rats. J. Pharm. Sci. 93:3027–3038PubMedCrossRefGoogle Scholar
  19. 19.
    Bruno R., Washington C. B., Lu J. F., Lieberman G., Banken L., and Klein P. (2005) Population pharmacokinetics of trastuzumab in patients With HER2+ metastatic breast cancer. Cancer Chemother. Pharmacol.Google Scholar
  20. 20.
    Lindbom L., Pihlgren P., and Jonsson E.N. (2005) PsN-Toolkit–a collection of computer intensive statistical methods for non-linear mixed effect modeling using NONMEM. Comput. Methods Programs Biomed. 79:241–257PubMedCrossRefGoogle Scholar
  21. 21.
    Beal S. L. and Sheiner L. B. (1989). NONMEM User’s Guide - Part I. Users Basic Guide. University of California, San FranciscoGoogle Scholar
  22. 22.
    Pinheiro J. C. and Bates D. M. Mixed-Effects Models in S and S-PLUS. Springer-Verlag, New York, 2001.Google Scholar
  23. 23.
    Delyon B., Lavielle M., Moulines E. (1999) Convergence of a stochastic approximation version of the EM algorithm. Ann. Stat. 27:94–128CrossRefGoogle Scholar
  24. 24.
    Kuhn E. and Lavielle M. (2004) Coupling a stochastic approximation version of EM with a MCMC procedure. ESAIM P&S: 115–131Google Scholar
  25. 25.
    D. Spiegelhalter, A. Thomas, N. Best, and D. J. Lunn. WinBUGS User Manual Version 1.4. Available from: Last Accessed: July 4 2006.Google Scholar
  26. 26.
    Ette E.I. (1996) Comparing non-hierarchical models: application to non-linear mixed effects modeling. Comput. Biol. Med. 26:505–512PubMedCrossRefGoogle Scholar
  27. 27.
    M. Davidian and D. M. Giltinan. Nonlinear Models for Repeated Measurement Data. Chapman & Hall, London, 1995.Google Scholar
  28. 28.
    Davidian M. and Giltinan D.M. (2003) Nonlinear Models for Repeated Measurement Data: An Overview and Update. J. Agr. Biol. Envir. St. 8:387–419CrossRefGoogle Scholar
  29. 29.
    Lindstrom M.J. and Bates D.M. (1990) Nonlinear mixed effects models for repeated measures data. Biometrics 46:673–687PubMedCrossRefGoogle Scholar
  30. 30.
    J. I. Myung and D. J. Navarro. Information matrix. In: Encyclopedia of behavioral Statistics, B. Everitt and D. Howel, eds., 2004.Google Scholar
  31. 31.
    Retout S., Mentré F., Bruno R. (2002) Fisher information matrix for non-linear mixed-effects models: evaluation and application for optimal design of enoxaparin population pharmacokinetics. Stat. Med. 21:2623–2639PubMedCrossRefGoogle Scholar
  32. 32.
    S. L. Beal and L. B. Sheiner. NONMEM User’s Guide – Part II. Users Supplemental Guide. University of California, San Francisco, 1988.Google Scholar
  33. 33.
    White H. (1982) Maximum likelihood estimation of misspecified model. Econometrica 50:1–25CrossRefGoogle Scholar
  34. 34.
    Kuhn E. and Lavielle M. (2005) Maximum likelihood estimation in nonlinear mixed effects model. Comput. Stat. Data Anal. 49:1020–1038CrossRefGoogle Scholar
  35. 35.
    B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, New York, 1993.Google Scholar
  36. 36.
    Ette E.I. (1997) Stability and performance of a population pharmacokinetic model. J. Clin. Pharmacol. 37:486–495PubMedGoogle Scholar
  37. 37.
    Das S. and Krishen A. (1999) Some bootstrap methods in nonlinear mixed-effect models. J. Stat. Plan. Inference 75:237–245CrossRefGoogle Scholar
  38. 38.
    Wu C.F.J. (2006) Comment on jackknife, bootstrap and resampling methods in regression analysis. Ann. Stat. 14:1261–1350Google Scholar
  39. 39.
    Gelman A., Carlin J. B., Stern H. S., and Rubin D. B. Bayesian Data Analysis, 2nd ed. Chapman & Hall, London, 2004.Google Scholar
  40. 40.
    Gelman A. and Rubin D.B. (1992) Inference from iterative simulation using multiple sequences (with discussion). Stat. Sci. 7:457–511Google Scholar
  41. 41.
    Brooks S.P. and Gelman A. (1998) Alternative methods for monitoring convergence of iterative simulations. J. Comput. Graph. Stat. 7:434–455CrossRefGoogle Scholar
  42. 42.
    Best N.G., Tan K.K., Gilks W.R., and Spiegelhalter D.J. (1995) Estimation of population pharmacokinetics using the Gibbs sampler. J. Pharmacokinet. Biopharm. 23:407–435PubMedCrossRefGoogle Scholar
  43. 43.
    Lunn D.J., Best N., Thomas A., Wakefield J., and Spiegelhalter D. (2002) Bayesian analysis of population PK/PD models: general concepts and software. J. Pharmacokinet. Pharmacodyn 29:271–307PubMedCrossRefGoogle Scholar
  44. 44.
    Pouillot R., Albert I., Cornu M., and Denis J.-B. (2003) Estimation of uncertainty and variability in bacterial growth using Bayesian inference. Application to Listeria monocytogenes. Int. J. Food. Microbiol. 81:87–104CrossRefGoogle Scholar
  45. 45.
    Roe D.J. (1997) Comparison of population pharmacokinetic modeling methods using simulated data: results from the population modeling workgroup. Stat. Med. 16:1241–1262PubMedCrossRefGoogle Scholar
  46. 46.
    A. J. Boeckmann, L. Sheiner, and S. L. Beal. NONMEM User’s Guide – Part VIII. Help Guide. University of California, San Francisco, 1996.Google Scholar
  47. 47.
    Tod M., Mentré F., Merlé Y., and Mallet A. (1998) Robust optimal design for the estimation of hyperparameters in population pharmacokinetics. J. Pharmacokinet. Biopharm. 26:689–716PubMedCrossRefGoogle Scholar
  48. 48.
    M. Tod, F. Mentré, Y. Merlé, and A. Mallet. Introduction to POPOS (optimization of sampling times for population parameter estimation in pharmacokinetics). In POPOS 1.0. 2000, pp. 4–5.Google Scholar
  49. 49.
    Gisleskog P.O., Karlsson M.O., Beal S.L. (2005) Use of prior information to stabilize a population data analysis. J. Pharmacokinet. Biopharm. 29:473–505Google Scholar
  50. 50.
    White D.B., Walawander C.A., Tung Y., and Grasela T.H. (1991) An evaluation of point and interval estimates in population pharmacokinetics using NONMEM analysis. J. Pharmacokinet. Biopharm. 19:87–112PubMedGoogle Scholar
  51. 51.
    Laird N.M. and Louis T.A. (1987) Empirical Bayes confidence intervals based on bootstrap samples. J. Am. Stat. Assoc. 32:739–757CrossRefGoogle Scholar
  52. 52.
    Wade J.R., Kelman A.W., Howie C.A. and Whiting B. (1993) Effect of misspecification of the absorption process on subsequent parameter estimation in population analysis. J. Pharmacokinet. Biopharm. 21:209–222PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  • Céline Dartois
    • 1
    • 2
    Email author
  • Annabelle Lemenuel-Diot
    • 3
  • Christian Laveille
    • 4
  • Brigitte Tranchand
    • 1
    • 2
    • 5
  • Michel Tod
    • 1
    • 6
    • 7
  • Pascal Girard
    • 1
    • 2
  1. 1.Université de LyonLyonFrance
  2. 2.Université Lyon 1OullinsFrance
  3. 3.Servier Research GroupCourbevoieFrance
  4. 4.Exprimo NVLummenBelgium
  5. 5.Centre Léon BérardLyonFrance
  6. 6.Hôpital CochinParisFrance
  7. 7.Université de Lyon, Lyon, F-69003, France;ISPB, Université Lyon1LyonFrance

Personalised recommendations