Updating for a new setting
A prediction model ideally provides valid predictions of outcome for individual patients at another setting than where the model was developed, e.g. differing in time and place. The validity of predictions can be assessed by comparing observed outcomes and predictions when empirical data from this setting are available. Various patterns of invalidity may however be observed as we have seen in the previous chapter. Detection of calibration-in-the-large problems should have top priority since miscalibration can cause systematically wrong decision making with the model (negative net benefit). Obviously, we may subsequently aim to update the model to improve predictions for future patients from the new setting. We discuss several approaches for updating a previously developed model. The risk is that simply re-estimating all regression coefficients in a model might replace reliable but slightly biased estimates by unbiased but very unreliable ones, particularly if the validation data set is relatively small.
We start with considering updating methods that focus on re-calibration (re-estimation of the intercept and/or updating of the slope of the linear predictor). Next, we turn to more structural model revisions (re-estimation of some or all regression coefficients, model extension with more predictors). For illustration we consider case studies with updating of a previously developed logistic regression model, a regression tree, and a previously developed Cox regression model. We conclude that parsimonious updating methods may often be preferable to more extensive model revisions, which should only be attempted with relatively large validation samples, in combination with shrinkage of differences between the updated model and the previously developed model.