Abstract
A new technique for generating regression ensembles is introduced in the present paper. The technique is based on earlier work on promoting model diversity through injection of noise into the outputs; it differs from the earlier methods in its rigorous requirement that the mean displacements applied to any data points output value be exactly zero.
It is illustrated how even the introduction of extremely large displacements may lead to prediction accuracy superior to that achieved by bagging.
It is demonstrated how ensembles of models with very high bias may have much better prediction accuracy than single models of the same bias-defying the conventional belief that ensembling high bias models is not purposeful.
Finally is outlined how the technique may be applied to classification.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Krogh, A. and Vedelsby, J. Neural network ensembles, Cross Validation, and Active Learning. In: G. Tesauro, D. S. Touretzky and T. K. Leen, eds. Advances in Neural Information Processing Systems 7, p. 231–238, MIT Press, Cambridge, MA, 1995.
Sollich, P. and Krogh, A. Learning with ensembles: How over-fitting can be useful. In: D. S. Touretzky, M. C. Mozer and M. E. Hasselmo, eds. Advances in Neural Information Processing Systems 8, p. 190–196, MIT Press, 1996.
Breiman, L. Bagging predictors. Machine Learning 24(2):123–140, 1996.
Breiman, L. Randomizing outputs to increase prediction accuracy. Machine Learning, 40(3): 229–242, September 2000.
Raviv, Y. and Intrator, N. Bootstrapping with noise: An effective regularization technique. Connection Science, Special issue on Combining Estimators, 8:356–372, 1996.
Raviv, Y. and Intrator, N. Variance reduction via noise and bias constraints. In: Sharkey, A. J. C. (Ed.) Combining Artificial Neural Nets. Springer Verlag. 1999.
Murphy, P. M. & Aha, D. W. UCI Repository of machine learning databases. University of California, Department of Information and Computer Science. Irvine, CA 1994.
Schwefel, H. Numerical Optimization of Computer Models. Wiley, New York, 1981.
Nelder, J. A and Mead, R. Computer Journal, 7, p. 308. 1965.
Press, W. H., Flannery, B. P., Teukolsky, S. A., Vetterling, W. T. Numerical Recipes in Pascal. Cambridge University Press, Cambridge, 1989.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Christensen, S.W. (2003). Ensemble Construction via Designed Output Distortion. In: Windeatt, T., Roli, F. (eds) Multiple Classifier Systems. MCS 2003. Lecture Notes in Computer Science, vol 2709. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44938-8_29
Download citation
DOI: https://doi.org/10.1007/3-540-44938-8_29
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40369-2
Online ISBN: 978-3-540-44938-6
eBook Packages: Springer Book Archive