Applied Intelligence

, Volume 34, Issue 3, pp 347–359 | Cite as

Random projections for linear SVM ensembles

  • Jesús Maudes
  • Juan J. Rodríguez
  • César García-Osorio
  • Carlos Pardo
Original Paper


This paper presents an experimental study using different projection strategies and techniques to improve the performance of Support Vector Machine (SVM) ensembles. The study has been made over 62 UCI datasets using Principal Component Analysis (PCA) and three types of Random Projections (RP), taking into account the size of the projected space and using linear SVMs as base classifiers. Random Projections are also combined with the sparse matrix strategy used by Rotation Forests, which is a method based in projections too. Experiments show that for SVMs ensembles (i) sparse matrix strategy leads to the best results, (ii) results improve when projected space dimension is bigger than the original one, and (iii) Random Projections also contribute to the results enhancement when used instead of PCA. Finally, random projected SVMs are tested as base classifiers of some state of the art ensembles, improving their performance.


Ensembles Random projections Rotation forests Diversity Kappa-error relative movement diagrams SVMs 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Achlioptas D (2003) Database-friendly random projections: Johnson-Lindenstrauss with binary coins. J Comput Syst Sci 66(4):671–687 MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Bingham E, Mannila H (2001) Random projection in dimensionality reduction: applications to image and text data. In: Knowledge Discovery and Data Mining. ACM, New York, pp 245–250 Google Scholar
  3. 3.
    Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140 MathSciNetzbMATHGoogle Scholar
  4. 4.
    Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46 CrossRefGoogle Scholar
  5. 5.
    Dasgupta S, Freund Y (2008) Random projection trees and low dimensional manifolds. In: STOC ’08: proceedings of the 40th annual ACM symposium on theory of computing. ACM, New York, pp 537–546 CrossRefGoogle Scholar
  6. 6.
    Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30 MathSciNetGoogle Scholar
  7. 7.
    Dietterich TG (1998) Approximate statistical test for comparing supervised classification learning algorithms. Neural Comput 10(7):1895–1923 CrossRefGoogle Scholar
  8. 8.
    Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ (2008) Liblinear: a library for large linear classification. J Mach Learn Res 9:1871–1874 Google Scholar
  9. 9.
    Fradkin D, Madigan D (2003) Experiments with random projections for machine learning. In: KDD ’03: proceedings of the ninth ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, pp 517–522 CrossRefGoogle Scholar
  10. 10.
    Frank A, Asuncion A (2010) UCI machine learning repository. URL
  11. 11.
    Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139 MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. SIGKDD Explor 11(1):10–18 CrossRefGoogle Scholar
  13. 13.
    Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8):832–844 CrossRefGoogle Scholar
  14. 14.
    Johnson W, Lindenstrauss J (1984) Extensions of Lipschitz maps into a Hilbert space. In: Conference in modern analysis and probability (1982, Yale University). Contemporary mathematics, vol 26. AMS, New York, pp 189–206 Google Scholar
  15. 15.
    Kuncheva LI (2004) Combining pattern classifiers: methods and algorithms. Wiley-Interscience, New York zbMATHCrossRefGoogle Scholar
  16. 16.
    Kuncheva LI, Rodríguez JJ (2007) An experimental study on rotation forest ensembles. In: 7th international workshop on multiple classifier systems, MCS 2007. LNCS, vol 4472. Springer, Berlin, pp 459–468 Google Scholar
  17. 17.
    Margineantu DD, Dietterich TG (1997) Pruning adaptive boosting. In: Proc. 14th international conference on machine learning. Morgan Kaufmann, San Mateo, pp 211–218 Google Scholar
  18. 18.
    Maudes J, Rodríguez JJ, García-Osorio C (2009) Disturbing neighbors diversity for decision forests. In: Okun O, Valentini G (eds) Applications of supervised and unsupervised ensemble methods. Studies in computational intelligence, vol 245. Springer, Berlin, pp 113–133 CrossRefGoogle Scholar
  19. 19.
    Rodríguez JJ, Kuncheva LI, Alonso CJ (2006) Rotation forest: A new classifier ensemble method. IEEE Trans Pattern Anal Mach Intell 28(10):1619–1630 CrossRefGoogle Scholar
  20. 20.
    Schclar A, Rokach L (2009) Random projection ensemble classifiers. In: Enterprise information systems 11th international conference proceedings. Lecture notes in business information processing, pp 309–316 Google Scholar
  21. 21.
    Vapnik VN (1999) The nature of statistical learning theory. Information science and statistics. Springer, Berlin Google Scholar
  22. 22.
    GI Webb (2000) Multiboosting: a technique for combining boosting and wagging. Mach Learn 40(2) Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Jesús Maudes
    • 1
  • Juan J. Rodríguez
    • 1
  • César García-Osorio
    • 1
  • Carlos Pardo
    • 1
  1. 1.University of BurgosVitoria s/nSpain

Personalised recommendations