Power, Selection Bias and Predictive Performance of the Population Pharmacokinetic Covariate Model

  • Jakob Ribbing
  • E. Niclas Jonsson


Identification and quantification of covariate relations is often an important part of population pharmacokinetic/pharmacodynamic (PK/PD) modelling. The covariate model is regularly built in a stepwise manner. With such methods, selection bias may be a problem if only statistically significant covariates are accepted into the model. Competition between multiple covariates may further increase selection bias, especially when there is a moderate to high correlation between the covariates. This can also result in a loss of power to find the true covariates. The aim of this simulation study was to investigate the effect on power, selection bias and predictive performance of the covariate model, when altering study design and system-related quantities. Data sets with 20–1000 subjects were investigated. Five covariates were created by sampling from a multivariate standard normal distribution. The true covariate was set up to have no, low, moderate and high correlation to the other four covariates, respectively. Data sets, in which each individual had two or three PK observations, were simulated using a one-compartment i.v. bolus model. The true covariate influenced clearance according to one of several magnitudes. Different magnitudes of residual error and inter-individual variability in the structural model parameters were also introduced to the simulation model. A total of 7400 replicate data sets were simulated independently for each combination of the above conditions. Models with one of the five simulated covariates influencing clearance and the model without any covariate were fitted to the data. The probability of selecting ( according to a pre-specified P-value ) the different covariates, along with the estimated covariate coefficient, was recorded. The results show that selection bias is very high for small data sets ( ≤ 50 subjects ) simulated with a weak covariate effect. If selected under these circumstances, the covariate coefficient is on average estimated to be more than twice its true value, making the covariate model useless for predictive purposes. Surprisingly, even though competition from false covariates caused substantial loss in the power of selecting the true covariate, the already high selection bias increased only marginally. This means that the bias due to competition is negligible if statistical significance is also required for covariate selection. Bias and predictive performance are direct functions of power, only indirectly affected by study design and system-related quantities. Mainly because of selection bias, low-powered covariates can be expected to harm the predictive performance when selected. For the same reason these low-powered covariates may falsely appear to be clinically relevant when selected. If the aim of an analysis is predictive modelling, we do not recommend stepwise selection or significance testing of covariates to be performed on small or moderately sized data sets ( <50–100 subjects)

NONMEM stepwise model building statistical significance stopping-rule bias competition bias 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    E. N. Jonsson and M. O. Karlsson. Automated covariate model building within NONMEM. Pharm. Res. 15:1463-1468 (1998).Google Scholar
  2. 2.
    A. J. Miller. Selection of subsets of regression variables. J. R. Stat. Soc. Series A (General). 147:389-425 (1984).Google Scholar
  3. 3.
    U. Wahlby, E. N. Jonsson, and M. O. Karlsson. Comparison of stepwise covariate model building strategies in population pharmacokinetic-pharmacodynamic analysis. AAPS Pharm. Sci. 4:27(2002).Google Scholar
  4. 4.
    P. L. Bonate. The effect of collinearity on parameter estimates in nonlinear mixed effect models. Pharm. Res. 16:709-717 (1999).Google Scholar
  5. 5.
    R. O'Neill and G. B. Wetherill. The present state of multiple comparison methods. J. R. Stat. Soc. Series B (Methodological). 33:218-250 (1971).Google Scholar
  6. 6.
    E. W. Steyerberg, M. J. Eijkemans, and J. D. Habbema. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J. Clin. Epidemiol. 52:935-942 (1999).Google Scholar
  7. 7.
    G. E. P. Box. Use and abuse of regression. Technometrics. 8:629-629 (1966).Google Scholar
  8. 8.
    D. W. Cockroft and M. H. Gault. Prediction of creatinine clearance from serum creatinine. Nephron. 16:31-41 (1976).Google Scholar
  9. 9.
    R. Bruno, N. Vivler, J. C. Vergniol, S. L. De Phillips, G. Montay, and L. B. Sheiner. A population pharmacokinetic model for docetaxel (Taxotere): model building and validation. J. Pharmacokinet. Biopharm.. 24:153-172 (1996).Google Scholar
  10. 10.
    I. The MathWorks. MATLAB. The Math Works, Natick, MA.Google Scholar
  11. 11.
    S. L. Beal. NONMEM Users Guides. NONMEM Project Group, University of California at San Francisco, San Francisco, CA, 1994.Google Scholar
  12. 12.
    U. Wahlby, E. N. Jonsson, and M. O. Karlsson. Assessment of actual significance levels for covariate effects in NONMEM. J. Pharmacokinet. Pharmacodyn. 28:231-252 (2001).Google Scholar
  13. 13.
    L. B. Sheiner and S. L. Beal. Some suggestions for measuring predictive performance. J. Pharmacokinet. Biopharm. 9:503-512 (1981).Google Scholar
  14. 14.
    J. S. U. Hjorth. Computer Intensive Statistical Methods. Chapman & Hall,London; 1994.Google Scholar
  15. 15.
    P. I. Lee. Design and power of a population pharmacokinetic study. Pharm. Res. 18:75-82 (2001).Google Scholar
  16. 16.
    E. N. Jonsson, J. R. Wade, and M. O. Karlsson. Comparison of some practical sampling strategies for population pharmacokinetic studies. J. Pharmacokinet. Biopharm. 24:245-263 (1996).Google Scholar
  17. 17.
    S. Retout, S. Duffull, and F. Mentre. Development and implementation of the population Fisher information matrix for the evaluation of population pharmacokinetic designs. Comput. Meth. Prog. Biomed. 65:141-151 (2001).Google Scholar
  18. 18.
    i. MathSoft.S-Plus 2000 Math Soft, Seattle, WA.Google Scholar

Copyright information

© Plenum Publishing Corporation 2004

Authors and Affiliations

  • Jakob Ribbing
    • 1
  • E. Niclas Jonsson
    • 1
  1. 1.Department of Pharmaceutical Biosciences, Division of Pharmacokinetics and Drug TherapyUppsala UniversityUppsalaSweden

Personalised recommendations