Random projections for linear SVM ensembles
This paper presents an experimental study using different projection strategies and techniques to improve the performance of Support Vector Machine (SVM) ensembles. The study has been made over 62 UCI datasets using Principal Component Analysis (PCA) and three types of Random Projections (RP), taking into account the size of the projected space and using linear SVMs as base classifiers. Random Projections are also combined with the sparse matrix strategy used by Rotation Forests, which is a method based in projections too. Experiments show that for SVMs ensembles (i) sparse matrix strategy leads to the best results, (ii) results improve when projected space dimension is bigger than the original one, and (iii) Random Projections also contribute to the results enhancement when used instead of PCA. Finally, random projected SVMs are tested as base classifiers of some state of the art ensembles, improving their performance.
KeywordsEnsembles Random projections Rotation forests Diversity Kappa-error relative movement diagrams SVMs
Unable to display preview. Download preview PDF.
- 2.Bingham E, Mannila H (2001) Random projection in dimensionality reduction: applications to image and text data. In: Knowledge Discovery and Data Mining. ACM, New York, pp 245–250 Google Scholar
- 8.Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ (2008) Liblinear: a library for large linear classification. J Mach Learn Res 9:1871–1874 Google Scholar
- 10.Frank A, Asuncion A (2010) UCI machine learning repository. URL http://archive.ics.uci.edu/ml
- 14.Johnson W, Lindenstrauss J (1984) Extensions of Lipschitz maps into a Hilbert space. In: Conference in modern analysis and probability (1982, Yale University). Contemporary mathematics, vol 26. AMS, New York, pp 189–206 Google Scholar
- 16.Kuncheva LI, Rodríguez JJ (2007) An experimental study on rotation forest ensembles. In: 7th international workshop on multiple classifier systems, MCS 2007. LNCS, vol 4472. Springer, Berlin, pp 459–468 Google Scholar
- 17.Margineantu DD, Dietterich TG (1997) Pruning adaptive boosting. In: Proc. 14th international conference on machine learning. Morgan Kaufmann, San Mateo, pp 211–218 Google Scholar
- 20.Schclar A, Rokach L (2009) Random projection ensemble classifiers. In: Enterprise information systems 11th international conference proceedings. Lecture notes in business information processing, pp 309–316 Google Scholar
- 21.Vapnik VN (1999) The nature of statistical learning theory. Information science and statistics. Springer, Berlin Google Scholar
- 22.GI Webb (2000) Multiboosting: a technique for combining boosting and wagging. Mach Learn 40(2) Google Scholar