Twin Boosting: improved feature selection and prediction
- 366 Downloads
We propose Twin Boosting which has much better feature selection behavior than boosting, particularly with respect to reducing the number of false positives (falsely selected features). In addition, for cases with a few important effective and many noise features, Twin Boosting also substantially improves the predictive accuracy of boosting. Twin Boosting is as general and generic as (gradient-based) boosting. It can be used with general weak learners and in a wide variety of situations, including generalized regression, classification or survival modeling. Furthermore, it is computationally feasible for large problems with potentially many more features than observed samples. Finally, for the special case of orthonormal linear models, we prove equivalence of Twin Boosting to the adaptive Lasso which provides some theoretical aspects on feature selection with Twin Boosting.
KeywordsClassification Gradient descent High-dimensional data Regression Regularization
Unable to display preview. Download preview PDF.
- Freund, Y., Schapire, R.: Experiments with a new boosting algorithm. In: Proceedings of the Thirteenth International Conference on Machine Learning. Morgan Kaufmann, San Francisco (1996) Google Scholar
- Guyon, I., Gunn, S., Nikravesh, M., Zadeh, L.: Feature Extraction, Foundations and Applications, Studies in Fuzziness and Soft Computing. Springer, Heidelberg (2006) Google Scholar
- Lutz, R.: Logitboost with trees applied to the WCCI 2006 performance prediction challenge datasets. In: Proceedings of the IJCNN 2006 Google Scholar
- Meir, R., Rätsch, G.: An introduction to boosting and leveraging. In: Mendelson, S., Smola, A. (eds.) Advanced Lectures on Machine Learning. Lecture Notes in Computer Science. Springer, Berlin (2003) Google Scholar
- Ridgeway, G.: The state of boosting. Comput. Sci. Stat. 31, 172–181 (1999) Google Scholar
- Schapire, R.: The boosting approach to machine learning: an overview. In: Denison, D., Hansen, M., Holmes, C., Mallick, B., Yu, B. (eds.) MSRI Workshop on Nonlinear Estimation and Classification. Springer, Berlin (2002) Google Scholar