Synergized Bootstrapping: The Whole is Faster than the Sum of Its Parts

Abstract

Re-sampling methods are popular for assessing uncertainty, for testing hypotheses, or for cross-validation because of their simplicity. They all rely on a similar scheme: generating replicated datasets by sampling data points from an original dataset, fitting a model or conducting a statistical test on each of these, and aggregating the results. However, when fitting the model or conducting the statistical test becomes time-consuming, re-sampling methods become impractical because of the many replications. Many methods have been proposed to alleviate the computational burden, but they generally do not incorporate two key features of re-sampled datasets. One, re-sampled datasets all stem from the same origin and therefore have similar characteristics. Two, there is a large class of cost functions for which the cost of a parameter set given data can be computed by summing its costs across the individual data points. As a consequence, once the costs of the individual data points are known, the parameter set’s cost can be obtained for any of the cost functions related to one of the replicated datasets. The synergized bootstrap method put forward in this paper exploits these two features to accelerate the optimization procedures for re-sampling methods. It is applied to the non-parametric bootstrapping of the parameters of a univariate mixture model, of which the min-log-likelihood function can be shown to have multiple local minima, using the differential evolution heuristic as global optimizer. It is demonstrated that the synergized method can lead to incredible accelerations (up to 100-500 times faster) while being more accurate than the standard DE method.