Skip to main content

Advertisement

SpringerLink
Log in

Metrics for External Model Evaluation with an Application to the Population Pharmacokinetics of Gliclazide

  • Research Paper
  • Published:
Pharmaceutical Research Aims and scope Submit manuscript

Abstract

Purpose

The aim of this study is to define and illustrate metrics for the external evaluation of a population model.

Materials and Methods

In this paper, several types of metrics are defined: based on observations (standardized prediction error with or without simulation and normalized prediction distribution error); based on hyperparameters (with or without simulation); based on the likelihood of the model. All the metrics described above are applied to evaluate a model built from two phase II studies of gliclazide. A real phase I dataset and two datasets simulated with the real dataset design are used as external validation datasets to show and compare how metrics are able to detect and explain potential adequacies or inadequacies of the model.

Results

Normalized prediction errors calculated without any approximation, and metrics based on hyperparameters or on objective function have good theoretical properties to be used for external model evaluation and showed satisfactory behaviour in the simulation study.

Conclusions

For external model evaluation, prediction distribution errors are recommended when the aim is to use the model to simulate data. Metrics through hyperparameters should be preferred when the aim is to compare two populations and metrics based on the objective function are useful during the model building process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. L. B. Sheiner and J. L. Steimer. Pharmacokinetic/pharmacodynamic modeling in drug development. Annu. Rev. Pharmacol. Toxicol. 40:67–95 (2000).

    Article  PubMed  CAS  Google Scholar 

  2. L. Aarons, M. O. Karlsson, F. Mentre, F. Rombout, J. L. Steimer, and A. van Peer. Role of modelling and simulation in Phase I drug development. Eur. J. Pharm. Sci. 13:115–122 (2001).

    Article  PubMed  CAS  Google Scholar 

  3. R. Jochemsen, C. Laveille, and D. D. Breimer. Application of pharmacokinetic/pharmacodynamic modelling and population approaches to drug development. Int. J. Pharm. Med. 13:243–251 (1999).

    Google Scholar 

  4. N. H. Holford, H. C. Kimko, J. P. Monteleone, and C. C. Peck. Simulation of clinical trials. Annu. Rev. Pharmacol. Toxicol. 40:209–234 (2000).

    Article  PubMed  CAS  Google Scholar 

  5. L. J. Lesko, M. Rowland, C. C. Peck, and T. F. Blaschke. Optimizing the science of drug development: opportunities for better candidate selection and accelerated evaluation in humans. Pharm. Res. 17:1335–1344 (2000).

    Article  PubMed  CAS  Google Scholar 

  6. H. C. Kimko and S. B. Duffull. Simulation for Designing Clinical Trials: A Pharmacokinetic–Pharmacodynamic Modeling Prospective. Marcel Dekker, New York, 2003.

    Google Scholar 

  7. Food and Drug Administration. Guidance for Industry: population pharmacokinetics (available at http://www.fda.gov/cder/guidance/index.html,1999).

  8. E. I. Ette. Stability and performance of a population pharmacokinetic model. J. Clin. Pharmacol. 37:486–495 (1997).

    PubMed  CAS  Google Scholar 

  9. P. J. Williams and E. I. Ette. Determination of Model Appropriateness. Marcel Dekker, New York, 2003.

    Google Scholar 

  10. Y. Yano, S. L. Beal, and L. B. Sheiner. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J. Pharmacokinet. Pharmacodyn. 28:171–192 (2001).

    Article  PubMed  CAS  Google Scholar 

  11. E. H. Cox, C. Veyrat-Follet, S. L. Beal, E. Fuseau, S. Kenkare, and L. B. Sheiner. A population pharmacokinetic–pharmacodynamic analysis of repeated measures time-to-event pharmacodynamic responses: the antiemetic effect of ondansetron. J. Pharmacokinet. Biopharm. 27:625–644 (1999).

    Article  PubMed  CAS  Google Scholar 

  12. P. Girard, T. F. Blaschke, H. Kastrissios, and L. B. Sheiner. A Markov mixed effect regression model for drug compliance. Stat Med. 17:2313–2333 (1998).

    Article  PubMed  CAS  Google Scholar 

  13. S. Vozeh, T. Uematsu, G. F. Hauf, and F. Follath. Performance of Bayesian feedback to forecast lidocaine serum concentration: evaluation of the prediction error and the prediction interval. J. Pharmacokinet. Biopharm. 13:203–212 (1985).

    Article  PubMed  CAS  Google Scholar 

  14. J. W. Mandema, R. F. Kaiko, B. Oshlack, R. F. Reder, and D. R. Stanski. Characterization and validation of a pharmacokinetic model for controlled-release oxycodone. Br. J. Clin. Pharmacol. 42:747–756 (1996).

    Article  PubMed  CAS  Google Scholar 

  15. K. Fattinger, S. Vozeh, H. R. Ha, M. Borner, and F. Follath. Population pharmacokinetics of quinidine. Br. J. Clin. Pharmacol. 31:279–286 (1991).

    PubMed  CAS  Google Scholar 

  16. T. H. Grasela, J. B. Fiedler-Kelly, C. Salvadori, C. Marey, R. Jochemsen, and H. Loo. Predictive performance of population pharmacokinetic parameters of tianeptine as applied to plasma concentrations from a post-marketing study. Eur. J. Clin. Pharmacol. 45:123–128 (1993).

    Article  PubMed  CAS  Google Scholar 

  17. L. Aarons, S. Vozeh, M. Wenk, P. Weiss, and F. Follath. Population pharmacokinetics of tobramycin. Br. J. Clin. Pharmacol. 28:305–314 (1989).

    PubMed  CAS  Google Scholar 

  18. L. B. Sheiner and S. L. Beal. Some suggestions for measuring predictive performance. J Pharmacokinet. Biopharm. 9:503–512 (1981).

    Article  PubMed  CAS  Google Scholar 

  19. F. Mentré and S. Escolano. Prediction discrepancies for the evaluation of nonlinear mixed-effects Models. J. Pharmacokinet. Pharmacodyn. 33:345–367 (2006).

    Article  PubMed  Google Scholar 

  20. E. Comets, K. Ikeda, P. Hoff, P. Fumoleau, J. Wanders, and Y. Tanigawara. Comparison of the pharmacokinetics of S-1, an oral anticancer agent, in Western and Japanese patients. J. Pharmacokinet. Pharmacodyn. 30:257–283 (2003).

    Article  PubMed  CAS  Google Scholar 

  21. N. Frey, C. Laveille, M. Paraire, M. Francillard, N. H. Holford, and R. Jochemsen. Population PKPD modelling of the long-term hypoglycaemic effect of gliclazide given as a once-a-day modified release (MR) formulation. Br. J. Clin. Pharmacol. 55:147–157 (2003).

    Article  PubMed  CAS  Google Scholar 

  22. S. L. Beal. Ways to fit a PK model with some data below the quantification limit. J. Pharmacokinet. Pharmacodyn. 28:481–504 (2001).

    Article  PubMed  CAS  Google Scholar 

  23. P. Delrat, M. Paraire, and R. Jochemsen. Complete bioavailability and lack of food-effect on pharmacokinetics of gliclazide 30 mg modified release in healthy volunteers. Biopharm. Drug Dispos. 23:151–157 (2002).

    Article  PubMed  CAS  Google Scholar 

  24. R. J. Simes. An improved Bonferroni procedure for multiple tests of significance. Biometrika 73:751–764 (1986).

    Article  Google Scholar 

  25. S. L. Beal and L. B. Sheiner. NONMEM Users Guides (I–VIII). Globomax LLC; Hanover, maryland. (1989–1998).

  26. F. Mesnil, F. Mentré, C. Dubruc, J. P. Thenot, and A. Mallet. Population pharmacokinetic analysis of mizolastine and validation from sparse data on patients using the nonparametric maximum likelihood method. J. Pharmacokinet. Biopharm. 26:133–161 (1998).

    Article  PubMed  CAS  Google Scholar 

  27. E. Comets and F. Mentré. Evaluation of tests based on individual versus population modeling to compare dissolution curves. J. Biopharm. Stat. 11:107–123 (2001).

    Article  PubMed  CAS  Google Scholar 

  28. F. Mentré and M. E. Ebelin. Validation of population pharmacokinetic/pharmacodynamic analyses: review of proposed approaches. The Population Approach: Measuring and Managing Variability in Response Concentration and Dose. Office for official publications of the European Communities, Brussels, 1997, pp. 141–158.

  29. A. Gelman, J. B. Carlin, H. S. Stern, and R. D.B. Bayesian Data Analysis. Chapman and Hall, London, 1995.

    Google Scholar 

  30. A. Hooker and M. O. Karlsson. Conditional weighted residuals. A diagnostic to improve population PK/PD model building and evaluation. AAPS J. 7(S2), Abstract W5321 (2005).

  31. M. J. Bayarri and P. Berger. P values for composite null models. JASA 95:1143–1172 (2000).

    Google Scholar 

  32. J. M. Robins, A. van der Vaart, and V. Ventura. Assymptotic distribution of P values in composite null models. JASA 95: 1143–1172 (2000).

    Google Scholar 

  33. G. Verbecke, and E. Lesaffre. A linear mixed-effects models with heterogeneity in random effects population. JASA 91: 217–221 (1996).

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. INSERM U738, Paris, France

    Karl Brendel, Emmanuelle Comets & France Mentré

  2. University Paris 7, Paris, France

    Karl Brendel & France Mentré

  3. AP-HP, Bichat Hospital, Paris, France

    Karl Brendel & France Mentré

  4. Institut de Recherches Internationales Servier, Courbevoie, France

    Céline Laffont & Christian Laveille

  5. Exprimo NV, Lummen, Belgium

    Christian Laveille

Authors
  1. Karl Brendel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emmanuelle Comets
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Céline Laffont
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christian Laveille
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. France Mentré
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Karl Brendel.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Brendel, K., Comets, E., Laffont, C. et al. Metrics for External Model Evaluation with an Application to the Population Pharmacokinetics of Gliclazide. Pharm Res 23, 2036–2049 (2006). https://doi.org/10.1007/s11095-006-9067-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11095-006-9067-5

Key words

Search

Navigation