Prediction Discrepancies for the Evaluation of Nonlinear Mixed-Effects Models

  • France MentréEmail author
  • Sylvie Escolano


Reliable estimation methods for non-linear mixed-effects models are now available and, although these models are increasingly used, only a limited number of statistical developments for their evaluation have been reported. We develop a criterion and a test to evaluate nonlinear mixed-effects models based on the whole predictive distribution. For each observation, we define the prediction discrepancy (pd) as the percentile of the observation in the whole marginal predictive distribution under H0. We propose to compute prediction discrepancies using Monte Carlo integration which does not require model approximation. If the model is valid, these pd should be uniformly distributed over [0, 1] which can be tested by a Kolmogorov–Smirnov test. In a simulation study based on a standard population pharmacokinetic model, we compare and show the interest of this criterion with respect to the one most frequently used to evaluate nonlinear mixed-effects models: standardized prediction errors (spe) which are evaluated using a first order approximation of the model. Trends in pd can also be evaluated via several plots to check for specific departures from the model


model evaluation population pharmacokinetics predictive distribution prediction errors 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    L.B. Sheiner and J.L. Steimer, Pharmacokinetic/pharmacodynamic modeling in drug development. Annu. Rev. Pharmacol. Toxicol. 40 (2000) 67-95CrossRefPubMedGoogle Scholar
  2. 2.
    Aarons L., Karlsson M.O., Mentré F., Rombout F., Steimer J.L., van Peer A., and Cost B15 experts. Role of modelling and simulation in phase I drug development. Eur. J. Pharm. Sci. 13:115–122 (2001).Google Scholar
  3. 3.
    N.H. Holford, H.C. Kimko, J.P. Monteleone and C.C. Peck, Simulation of clinical trials. Annu. Rev. Pharmacol. Toxicol. 40 (2000) 209-234CrossRefPubMedGoogle Scholar
  4. 4.
    L.J. Lesko, M. Rowland, C.C. Peck and T.F. Blaschke, Optimizing the science of drug development: opportunities for better candidate selection and accelerated evaluations in humans. J. Clin. Pharmacol. 40 (2000) 803-814CrossRefPubMedGoogle Scholar
  5. 5.
    H.C. Kimko and S.B. Duffull, Simulation for Designing Clinical Trials: A Pharmacokinetic—Pharmacodynamic Modeling Prospective. New York: Marcel Dekker (2003).Google Scholar
  6. 6.
    Y. Yano, S.L. Beal and L.B. Sheiner, Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J. Pharmacokinet. Pharmacodyn. 28 (2001) 171-192PubMedCrossRefGoogle Scholar
  7. 7.
    M. Davidian and D.M. Giltinan, Nonlinear Models for Repeated Measurement Data. London: Chapman and Hall (1995).Google Scholar
  8. 8.
    E.F. Vonesh and V.M. Chinchilli, Linear and Nonlinear Models for the Analysis of Repeated Measurements. New York: Marcel Dekker (1997).Google Scholar
  9. 9.
    J.C. Pinheiro and D.M. Bates, Mixed-Effect Models in S and Splus. New York: Springer Verlag (2000).Google Scholar
  10. 10.
    L. Aarons, Software for population pharmacokinetics and pharmacodynamics. Clin. Pharmacokinet. 36 (1999) 255-264PubMedCrossRefGoogle Scholar
  11. 11.
    A.J. Boeckmann, L.B. Sheiner and S.L. Beal, NONMEM users guide. San Francisco: NONMEM Users group at University of California (1994).Google Scholar
  12. 12.
    F. Mentré and R. Gomeni, A two-step iterative algorithm for estimation in nonlinear mixed-effect models with an evaluation in population pharmacokinetics. J. Biopharm. Stat. 5 (1995) 141-158PubMedCrossRefGoogle Scholar
  13. 13.
    R.D. Wolfinger and X. Lin, Two Taylor-series approximation methods for estimation in nonlinear mixed-effects models with an evaluation in population pharmacokinetics. Comput. Stat. Data Anal. 25 (1997) 465-490CrossRefGoogle Scholar
  14. 14.
    A.E. Gelfan, Model determination using sampling-based methods. In: Markov Chain Monte Carlo in Practice.. Boca Raton: Chapman & Hall (1996) pp. 145-161Google Scholar
  15. 15.
    F. Mentré and M. E. Ebelin. Validation of population pharmacokinetic/pharmacodynamic analyses: review of proposed approaches. In The Population Approach: Measuring and Managing Variability in Response Concentration and Dose. Office for official publications of the European Communities, Brussels, 1997, pp. 141–158.Google Scholar
  16. 16.
    Food and Drug Administration. Guidance for Industry: Population Pharmacokinetics (available at 1999).Google Scholar
  17. 17.
    A. Gelman, J.B. Carlin, H.S. Stern and D.B. Rubin, Bayesian Data Analysis. London: Chapman and Hall (1995).Google Scholar
  18. 18.
    P.J. Williams and I.E. Ette, Determination of model appropriateness. In: In Simulation for Designing Clinical Trials: A Pharmacokinetic–Pharmacodynamic Modeling Prospective.. New York: Marcel Dekker (2003) pp. 73-104Google Scholar
  19. 19.
    M. Evans, Comments of asymptotic distribution of P values in composite null models by J. M. Robins, A. van der Vaart and V. Ventura.. J. Am. Stat. Assoc. 95 (2000) 1160-1163CrossRefGoogle Scholar
  20. 20.
    M.J. Bayarri and J.O. Berger, P values for composite null models. J. Am. Stat. Assoc. 95 (2000) 1127-1142CrossRefGoogle Scholar
  21. 21.
    J.M. Robins, A. Vaart van der and V. Ventura, Asymptotic distribution of P values in composite null models (with discussion). J. Am. Stat. Assoc. 95 (2000) 1143-1172CrossRefGoogle Scholar
  22. 22.
    N. Lange and L. Ryan, Assessing normality in random effects models. Ann. Statist. 17 (1989) 624-642CrossRefGoogle Scholar
  23. 23.
    A.E. Gelfand, D.K. Det and H. Chang, Model determination using predictive distributions with implementation via sampling-based methods. In: In Bayesian Statistics 4.. Oxford: University Press (1992) pp. 147-167Google Scholar
  24. 24.
    N.G. Best, K.K.C. Tan, W.R. Gilks and D.J. Spiegelhalter, Estimation of Population Pharmacokinetics using the Gibbs sampler. J. Pharmacokinet. Biopharm. 23 (1995) 407-435PubMedCrossRefGoogle Scholar
  25. 25.
    F. Mesnil, F. Mentré, C. Dubruc, J.P. Thénot and A. Mallet, Population pharmacokinetic analysis of mizolastine and validation from sparse data on patients using the nonparametric maximum likelihood method. J. Pharmacokinet. Biopharm. 26 (1998) 133-161CrossRefPubMedGoogle Scholar
  26. 26.
    A. Mallet, A maximum likelihood estimation method for random coefficient regression models. Biometrika 73 (1996) 645-646CrossRefGoogle Scholar
  27. 27.
    E. Comets, K. Ikeda, P. Hoff, P. Fumoleau, J. Wanders and Y. Tanigawara, Comparison of the pharmacokinetics of S-1, an oral anticancer agent, in Western and Japanese patients. J. Pharmacokinet. Pharmacodyn. 30 (2003) 257-283CrossRefPubMedGoogle Scholar
  28. 28.
    A. Gelman and X.L. Meng, Model checking and model improvement. In: In Markov Chain Monte Carlo in Practice.. Boca Raton: Chapman & Hall (1996) pp. 189-201Google Scholar
  29. 29.
    R.B. D’Agostino and M.A. Stephens, Goodness-of-Fit Techniques. New York: Marcel Dekker (1986).Google Scholar
  30. 30.
    P. McCullagh and J.A. Nelder, Generalized Linear Models. London: Chapman & Hall (1989).Google Scholar

Copyright information

© Springer Science+Business Media, Inc. 2005

Authors and Affiliations

  1. 1.INSERMParisFrance
  2. 2.Department of Epidemiology, Biostatistics and Clinical ResearchUniversity Paris 7ParisFrance
  3. 3.University Hospital Bichat – Claude BernardParisFrance
  4. 4.INSERMVillejuif CedexFrance

Personalised recommendations