Resampling for Assessing Parameter Estimates

Abstract

We often collect a random sample of cases from a population, let this sample interact with a model in some way, and then examine with interest a number (or several numbers) that result from the interaction. For example, we may use the random sample to train a model and then examine one or more of the model’s learned parameters. More often, we apply a previously trained model to the cases and compute a measure of the model’s performance so that we may judge the model’s worth and perhaps even extrapolate its future performance. On page 53 we saw an excellent method for computing confidence bounds for individual prediction errors. On page 121 we saw that this same method could be used to compute confidence bounds for clusters of future gains or costs obtained from a classification scheme. The subject of this chapter is somewhat different, though nevertheless related. Here, we are not concerned with performance on individual or small groups of future cases. Rather, we compute a single measure that describes some aspect of the model, and then we judge the quality of this measurement.