Advertisement

Annals of Operations Research

, Volume 216, Issue 1, pp 327–342 | Cite as

Robust generalized eigenvalue classifier with ellipsoidal uncertainty

  • Petros Xanthopoulos
  • Mario R. Guarracino
  • Panos M. PardalosEmail author
Article

Abstract

Uncertainty is a concept associated with data acquisition and analysis, usually appearing in the form of noise or measure error, often due to some technological constraint. In supervised learning, uncertainty affects classification accuracy and yields low quality solutions. For this reason, it is essential to develop machine learning algorithms able to handle efficiently data with imprecision. In this paper we study this problem from a robust optimization perspective. We consider a supervised learning algorithm based on generalized eigenvalues and we provide a robust counterpart formulation and solution in case of ellipsoidal uncertainty sets. We demonstrate the performance of the proposed robust scheme on artificial and benchmark datasets from University of California Irvine (UCI) machine learning repository and we compare results against a robust implementation of Support Vector Machines.

Keywords

Robust optimization Generalized eigenvalue classification Uncertainty 

Notes

Acknowledgements

This project was partially funded by National Science Foundation (N.S.F.) grants and Italian Flagship Project Interomics funded by MIUR and CNR.

References

  1. Aeberhard, S., Coomans, D., & De Vel, O. (1992a). Comparison of classifiers in high dimensional settings. Tech. Rep. Dept. Math. Statist., James Cook Univ. North Queensland, Australia Google Scholar
  2. Aeberhard, S., Coomans, D., & De Vel, O. (1992b). The classification performance of RDA, Cambridge. Tech. Rep. (pp. 92–01). Dept. of Computer Science/Dept. of Mathematics and Statistics, James Cook University of North Queensland Google Scholar
  3. Andersen, M. S., Dahl, J., Liu, Z., & Vandenberghe, L. (2011). Interior-point methods for large-scale cone programming. Optimization for machine learning. Cambridge: MIT Press. Google Scholar
  4. Ben-Tal, A., El Ghaoui, L., & Nemirovski, A. S. (2009). Robust optimization. Princeton: Princeton University Press. Google Scholar
  5. Ben-Tal, A., & Nemirovski, A. (1998). Robust convex optimization. Mathematics of Operations Research, 23(4), 769–805. CrossRefGoogle Scholar
  6. Ben-Tal, A., & Nemirovski, A. (1999). Robust solutions of uncertain linear programs. Operations Research Letters, 25(1), 1–14. CrossRefGoogle Scholar
  7. Ben-Tal, A., & Nemirovski, A. (2000). Robust solutions of linear programming problems contaminated with uncertain data. Mathematical Programming, 88(3), 411–424. CrossRefGoogle Scholar
  8. Ben-Tal, A., & Nemirovski, A. S. (2002). Robust optimization—methodology and applications. Mathematical Programming, 92(3), 453–480. CrossRefGoogle Scholar
  9. Bertsimas, D., Brown, D. B., & Caramanis, C. (2011). Theory and applications of robust optimization. SIAM Review 53(3), 464–501. CrossRefGoogle Scholar
  10. Bertsimas, D., & Sim, M. (2004). The price of robustness. Operations Research, 52(1), 35–53. CrossRefGoogle Scholar
  11. Caramanis, C., Mannor, S., & Xu, H. (2011). Robust optimization in machine learning. In S. Sra, S. Nowozin, & S. J. Wright (Eds.), Optimization for machine learning (pp. 369–402). Cambridge: MIT Press. Google Scholar
  12. Cifarelli, C., Guarracino, M. R., Seref, O., Cuciniello, S., & Pardalos, P. M. (2007). Incremental classification with generalized eigenvalues. Journal of Classification, 24(2), 205–219. CrossRefGoogle Scholar
  13. Cortes, C., & Vapnik, V. N. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. Google Scholar
  14. D’Aspremont, A., Ghaoui, L., Jordan, M., & Lanckriet, G. (2004). A direct formulation for sparse PCA using semidefinite programming. SIAM Review, 49(3), 434–448. CrossRefGoogle Scholar
  15. El Ghaoui, L., & Lebret, H. (1997). Robust solutions to least-squares problems with uncertain data. SIAM Journal on Matrix Analysis and Applications, 18, 1035–1064. CrossRefGoogle Scholar
  16. El Ghaoui, L., Oustry, F., Lebret, H., et al. (1998). Robust solutions to uncertain semidefinite programs. SIAM Journal on Optimization, 9, 33–52. CrossRefGoogle Scholar
  17. Evgeniou, T., Pontil, M., & Poggio, T. (2000). Regularization networks and support vector machines. Advances in Computational Mathematics, 13(1), 1–50. CrossRefGoogle Scholar
  18. Fisher, R., et al. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7, 179–188. CrossRefGoogle Scholar
  19. Frank, A., & Asuncion, A. (2010). UCI machine learning repository. http://archive.ics.uci.edu/ml.
  20. Guarracino, M. R., Cifarelli, C., Seref, O., & Pardalos, P. M. (2007). A classification method based on generalized eigenvalue problems. Optimization Methods & Software, 22(1), 73–81. CrossRefGoogle Scholar
  21. Huber, P., & Ronchetti, E. (1981). MyiLibrary: robust statistics (Vol. 1). Hoboken: Wiley Online Library. CrossRefGoogle Scholar
  22. Hubert, M., Rousseeuw, P., & Vanden Branden, K. (2005). ROBPCA: a new approach to robust principal component analysis. Technometrics, 47(1), 64–79. CrossRefGoogle Scholar
  23. Irpino, A., Guarracino, M. R., & Verde, R. (2010). Multiclass generalized eigenvalue proximal support vector machines. In 4th IEEE conference on complex, intelligent and software intensive systems (CISIS 2010). (pp. 25–32). Los Alamitos: IEEE Computer Society. Google Scholar
  24. Kim, S. J., & Boyd, S. (2008). A minimax theorem with applications to machine learning, signal processing, and finance. SIAM Journal on Optimization, 19(3), 1344–1367. CrossRefGoogle Scholar
  25. Kim, S. J., Magnani, A., & Boyd, S. (2006). Robust fisher discriminant analysis. Advances in Neural Information Processing Systems, 18, 659. Google Scholar
  26. Mangasarian, O. L., & Wild, E. W. (2006). Multisurface proximal support vector machine classification via generalized eigenvalues. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(1), 69–74. CrossRefGoogle Scholar
  27. Musicant, D. R. (1998). NDC: normally distributed clustered datasets. http://www.cs.wisc.edu/dmi/svm/ndc/.
  28. Pothin, J., & Richard, C. (2006). Incorporating prior information into support vector machines in the form of ellipsoidal knowledge sets. Citeseer. Google Scholar
  29. Shahbazpanahi, S., Gershman, A., Luo, Z., & Wong, K. (2003). Robust adaptive beamforming using worst-case SINR optimization: a new diagonal loading-type solution for general-rank signal models. In 2003 IEEE international conference on acoustics, speech, and signal processing. Proceedings (ICASSP’03) (Vol. 5). New York: IEEE Press. Google Scholar
  30. Shivaswamy, P., Bhattacharyya, C., & Smola, A. (2006). Second order cone programming approaches for handling missing and uncertain data. The Journal of Machine Learning Research, 7, 1283–1314. Google Scholar
  31. Smith, J. W., Everhart, J. E., Dickson, W. C., Knowler, W. C., & Johannes, R. S. (1988). Using the adap learning algorithm to forecast the onset of diabetes mellitus. Johns Hopkins APL Technical Digest, 10, 262–266. Google Scholar
  32. Smola, A. J., Schölkopf, B., & Müller, K. R. (1998). The connection between regularization operators and support vector kernels. Neural Networks, 11(4), 637–649. CrossRefGoogle Scholar
  33. Song, Q., Hu, W., & Xie, W. (2002). Robust support vector machine with bullet hole image classification. IEEE Transactions on Systems, Man, and Cybernetics, Part C, 32(4), 440–448. CrossRefGoogle Scholar
  34. Trafalis, T. B., & Gilbert, R. C. (2006). Robust classification and regression using support vector machines. European Journal of Operational Research, 173(3), 893–909. CrossRefGoogle Scholar
  35. Trafalis, T. B., & Gilbert, R. C. (2007). Robust support vector machines for classification and computational issues. Optimization Methods & Software, 22(1), 187–198. CrossRefGoogle Scholar
  36. Vandenberghe, L. (2010). The CVXOPT linear and quadratic cone program solvers. Google Scholar
  37. Vapnik, V. N. (1999). The nature of statistical learning theory. Information science and statistics. Berlin: Springer. Google Scholar
  38. Verdú, S., & Poor, H. (2002). On minimax robustness: a general approach and applications. IEEE Transactions on Information Theory, 30(2), 328–340. CrossRefGoogle Scholar
  39. Xanthopoulos, P., Pardalos, P. M., & Trafalis, T. B. (2012). Robust data mining. New York: Springer. Google Scholar
  40. Xu, H., Caramanis, C., & Mannor, S. (2009). Robustness and regularization of support vector machines. Journal of Machine Learning Research, 10, 1485–1510. Google Scholar
  41. Xu, H., Caramanis, C., & Mannor, S. (2010). Robust regression and lasso. IEEE Transactions on Information Theory, 56(7), 3561–3574. CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Petros Xanthopoulos
    • 1
  • Mario R. Guarracino
    • 3
  • Panos M. Pardalos
    • 2
    Email author
  1. 1.Industrial Engineering and Management Systems DepartmentUniversity of Central FloridaOrlandUSA
  2. 2.Center for Applied Optimization, Department of Industrial and Systems EngineeringUniversity of FloridaGainesvilleUSA
  3. 3.High Performance Computing and Networking InstituteNational Research Council of ItalyNaplesItaly

Personalised recommendations