The Minimum Redundancy – Maximum Relevance Approach to Building Sparse Support Vector Machines
Recently, building sparse SVMs becomes an active research topic due to its potential applications in large scale data mining tasks. One of the most popular approaches to building sparse SVMs is to select a small subset of training samples and employ them as the support vectors. In this paper, we explain that selecting the support vectors is equivalent to selecting a number of columns from the kernel matrix, and is equivalent to selecting a subset of features in the feature selection domain. Hence, we propose to use an effective feature selection algorithm, namely the Minimum Redundancy – Maximum Relevance (MRMR) algorithm to solve the support vector selection problem. MRMR algorithm was then compared to two existing methods, namely back-fitting (BF) and pre-fitting (PF) algorithms. Preliminary results showed that MRMR generally outperformed BF algorithm while it was inferior to PF algorithm, in terms of generalization performance. However, the MRMR approach was extremely efficient and significantly faster than the two compared algorithms.
KeywordsRelevance Redundancy Sparse design SVMs Machine learning
Unable to display preview. Download preview PDF.
- 4.Fung, G., Mangasarian, O.L.: Proximal support vector machine classifiers. In: Proceedings of Knowledge Discovery and Data Mining, San Francisco, CA, New York, pp. 77–86 (2001)Google Scholar
- 6.Burges, C.J.C.: Simplified support vector decision rules. In: Proceedings of the 13th International Conference on Machine Learning, Bari, Italy, pp. 71–77 (1996)Google Scholar
- 7.Burges, C.J.C., Schoelkopf, B.: Improving speed and accuracy of support vector learning machines. In: Advances in Neural Information Processing Systems, vol. 9, pp. 375–381. MIT Press, Cambridge (1997)Google Scholar
- 9.Lee, Y., Mangasarian, O.L.: RSVM: reduced support vector machines. In: CD Proceedings of the First SIAM International Conference on Data Mining, Chicago (2001)Google Scholar
- 10.Lee, Y., Mangasarian, O.L.: SSVM: A smooth support vector machine. In: Computational Optimization and applications, pp. 5–22 (2001)Google Scholar
- 14.Sun, P., Yao, X.: Greedy forward selection algorithms to sparse Gaussian process regression. In: Proceedings of the 2006 International Joint Conference on Neural Networks (IJCNN 2006), Vancouver, Canada, pp. 159–165 (2006)Google Scholar
- 15.Ding, C., Peng, H.: Minimum redundancy feature selection from microarray gene expression data. In: Proceedings of the Computational Systems Bioinformatics, pp. 523–528 (2003)Google Scholar
- 16.UCI Machine Learning Repository, http://www.ics.uci.edu/~mlearn/MLRepository.html