Abstract
Purpose
The aim of this study is to define and illustrate metrics for the external evaluation of a population model.
Materials and Methods
In this paper, several types of metrics are defined: based on observations (standardized prediction error with or without simulation and normalized prediction distribution error); based on hyperparameters (with or without simulation); based on the likelihood of the model. All the metrics described above are applied to evaluate a model built from two phase II studies of gliclazide. A real phase I dataset and two datasets simulated with the real dataset design are used as external validation datasets to show and compare how metrics are able to detect and explain potential adequacies or inadequacies of the model.
Results
Normalized prediction errors calculated without any approximation, and metrics based on hyperparameters or on objective function have good theoretical properties to be used for external model evaluation and showed satisfactory behaviour in the simulation study.
Conclusions
For external model evaluation, prediction distribution errors are recommended when the aim is to use the model to simulate data. Metrics through hyperparameters should be preferred when the aim is to compare two populations and metrics based on the objective function are useful during the model building process.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
L. B. Sheiner and J. L. Steimer. Pharmacokinetic/pharmacodynamic modeling in drug development. Annu. Rev. Pharmacol. Toxicol. 40:67–95 (2000).
L. Aarons, M. O. Karlsson, F. Mentre, F. Rombout, J. L. Steimer, and A. van Peer. Role of modelling and simulation in Phase I drug development. Eur. J. Pharm. Sci. 13:115–122 (2001).
R. Jochemsen, C. Laveille, and D. D. Breimer. Application of pharmacokinetic/pharmacodynamic modelling and population approaches to drug development. Int. J. Pharm. Med. 13:243–251 (1999).
N. H. Holford, H. C. Kimko, J. P. Monteleone, and C. C. Peck. Simulation of clinical trials. Annu. Rev. Pharmacol. Toxicol. 40:209–234 (2000).
L. J. Lesko, M. Rowland, C. C. Peck, and T. F. Blaschke. Optimizing the science of drug development: opportunities for better candidate selection and accelerated evaluation in humans. Pharm. Res. 17:1335–1344 (2000).
H. C. Kimko and S. B. Duffull. Simulation for Designing Clinical Trials: A Pharmacokinetic–Pharmacodynamic Modeling Prospective. Marcel Dekker, New York, 2003.
Food and Drug Administration. Guidance for Industry: population pharmacokinetics (available at http://www.fda.gov/cder/guidance/index.html,1999).
E. I. Ette. Stability and performance of a population pharmacokinetic model. J. Clin. Pharmacol. 37:486–495 (1997).
P. J. Williams and E. I. Ette. Determination of Model Appropriateness. Marcel Dekker, New York, 2003.
Y. Yano, S. L. Beal, and L. B. Sheiner. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J. Pharmacokinet. Pharmacodyn. 28:171–192 (2001).
E. H. Cox, C. Veyrat-Follet, S. L. Beal, E. Fuseau, S. Kenkare, and L. B. Sheiner. A population pharmacokinetic–pharmacodynamic analysis of repeated measures time-to-event pharmacodynamic responses: the antiemetic effect of ondansetron. J. Pharmacokinet. Biopharm. 27:625–644 (1999).
P. Girard, T. F. Blaschke, H. Kastrissios, and L. B. Sheiner. A Markov mixed effect regression model for drug compliance. Stat Med. 17:2313–2333 (1998).
S. Vozeh, T. Uematsu, G. F. Hauf, and F. Follath. Performance of Bayesian feedback to forecast lidocaine serum concentration: evaluation of the prediction error and the prediction interval. J. Pharmacokinet. Biopharm. 13:203–212 (1985).
J. W. Mandema, R. F. Kaiko, B. Oshlack, R. F. Reder, and D. R. Stanski. Characterization and validation of a pharmacokinetic model for controlled-release oxycodone. Br. J. Clin. Pharmacol. 42:747–756 (1996).
K. Fattinger, S. Vozeh, H. R. Ha, M. Borner, and F. Follath. Population pharmacokinetics of quinidine. Br. J. Clin. Pharmacol. 31:279–286 (1991).
T. H. Grasela, J. B. Fiedler-Kelly, C. Salvadori, C. Marey, R. Jochemsen, and H. Loo. Predictive performance of population pharmacokinetic parameters of tianeptine as applied to plasma concentrations from a post-marketing study. Eur. J. Clin. Pharmacol. 45:123–128 (1993).
L. Aarons, S. Vozeh, M. Wenk, P. Weiss, and F. Follath. Population pharmacokinetics of tobramycin. Br. J. Clin. Pharmacol. 28:305–314 (1989).
L. B. Sheiner and S. L. Beal. Some suggestions for measuring predictive performance. J Pharmacokinet. Biopharm. 9:503–512 (1981).
F. Mentré and S. Escolano. Prediction discrepancies for the evaluation of nonlinear mixed-effects Models. J. Pharmacokinet. Pharmacodyn. 33:345–367 (2006).
E. Comets, K. Ikeda, P. Hoff, P. Fumoleau, J. Wanders, and Y. Tanigawara. Comparison of the pharmacokinetics of S-1, an oral anticancer agent, in Western and Japanese patients. J. Pharmacokinet. Pharmacodyn. 30:257–283 (2003).
N. Frey, C. Laveille, M. Paraire, M. Francillard, N. H. Holford, and R. Jochemsen. Population PKPD modelling of the long-term hypoglycaemic effect of gliclazide given as a once-a-day modified release (MR) formulation. Br. J. Clin. Pharmacol. 55:147–157 (2003).
S. L. Beal. Ways to fit a PK model with some data below the quantification limit. J. Pharmacokinet. Pharmacodyn. 28:481–504 (2001).
P. Delrat, M. Paraire, and R. Jochemsen. Complete bioavailability and lack of food-effect on pharmacokinetics of gliclazide 30 mg modified release in healthy volunteers. Biopharm. Drug Dispos. 23:151–157 (2002).
R. J. Simes. An improved Bonferroni procedure for multiple tests of significance. Biometrika 73:751–764 (1986).
S. L. Beal and L. B. Sheiner. NONMEM Users Guides (I–VIII). Globomax LLC; Hanover, maryland. (1989–1998).
F. Mesnil, F. Mentré, C. Dubruc, J. P. Thenot, and A. Mallet. Population pharmacokinetic analysis of mizolastine and validation from sparse data on patients using the nonparametric maximum likelihood method. J. Pharmacokinet. Biopharm. 26:133–161 (1998).
E. Comets and F. Mentré. Evaluation of tests based on individual versus population modeling to compare dissolution curves. J. Biopharm. Stat. 11:107–123 (2001).
F. Mentré and M. E. Ebelin. Validation of population pharmacokinetic/pharmacodynamic analyses: review of proposed approaches. The Population Approach: Measuring and Managing Variability in Response Concentration and Dose. Office for official publications of the European Communities, Brussels, 1997, pp. 141–158.
A. Gelman, J. B. Carlin, H. S. Stern, and R. D.B. Bayesian Data Analysis. Chapman and Hall, London, 1995.
A. Hooker and M. O. Karlsson. Conditional weighted residuals. A diagnostic to improve population PK/PD model building and evaluation. AAPS J. 7(S2), Abstract W5321 (2005).
M. J. Bayarri and P. Berger. P values for composite null models. JASA 95:1143–1172 (2000).
J. M. Robins, A. van der Vaart, and V. Ventura. Assymptotic distribution of P values in composite null models. JASA 95: 1143–1172 (2000).
G. Verbecke, and E. Lesaffre. A linear mixed-effects models with heterogeneity in random effects population. JASA 91: 217–221 (1996).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Brendel, K., Comets, E., Laffont, C. et al. Metrics for External Model Evaluation with an Application to the Population Pharmacokinetics of Gliclazide. Pharm Res 23, 2036–2049 (2006). https://doi.org/10.1007/s11095-006-9067-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11095-006-9067-5