Comparison of a Novel Combined ECOC Strategy with Different Multiclass Algorithms Together with Parameter Optimization Methods

  • Marco Hülsmann
  • Christoph M. Friedrich
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4571)


In this paper we consider multiclass learning tasks based on Support Vector Machines (SVMs). In this regard, currently used methods are One-Against-All or One-Against-One, but there is much need for improvements in the field of multiclass learning. We developed a novel combination algorithm called Comb-ECOC, which is based on posterior class probabilities. It assigns, according to the Bayesian rule, the respective instance to the class with the highest posterior probability. A problem with the usage of a multiclass method is the proper choice of parameters. Many users only take the default parameters of the respective learning algorithms (e.g. the regularization parameter C and the kernel parameter γ). We tested different parameter optimization methods on different learning algorithms and confirmed the better performance of One-Against-One versus One-Against-All, which can be explained by the maximum margin approach of SVMs.


Support Vector Machine Support Vector Decision Boundary Bootstrap Replication Binary Support Vector Machine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bose, R.C., Ray-Chaudhuri, D.K.: On A Class of Error Correcting Binary Group Codes. Information and Control 3 (1960)Google Scholar
  2. 2.
    Breiman, L.: Bagging Predictors. Machine Learning 24, 123–140 (1996)zbMATHMathSciNetGoogle Scholar
  3. 3.
    Christianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000)Google Scholar
  4. 4.
    Crammer, K., Singer, Y.: On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines. Journal of Machine Learning Reseach 2, 265–292 (2001)CrossRefGoogle Scholar
  5. 5.
    Dietterich, T., Bakiri, G.: Solving Multiclass Learning Problems via Error-Correcting Output Codes. Journal of Artificial Intelligence Research 2, 263–286 (1995)zbMATHGoogle Scholar
  6. 6.
    Dimitriadou, E., Hornik, K., Leisch, F., Meyer, D., Weingessel, A.: The e1071 package. Manual (2006)Google Scholar
  7. 7.
    Efron, B., Tibshirani, R.: An Introduction to the Bootstrap. Chapman & Hall/CRC (1993)Google Scholar
  8. 8.
    Friedrich, C.: Kombinationen evolutionär optimierter Klassifikatoren. PhD thesis, Universität Witten/Herdecke (2005)Google Scholar
  9. 9.
    García-Pedrajas, N., Ortiz-Boyer, D.: Improving Multiclass Pattern Recognition by the Combination of Two Strategies IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (2006)Google Scholar
  10. 10.
    Hastie, T., Rosset, S., Tibshirani, R., Zhu, J.: The Entire Regularization Path for the Support Vector Machine. Technical Report, Statistics Department, Stanford University (2004)Google Scholar
  11. 11.
    Hsu, C.-W., Chang, C.-C., Lin, C.-J.: A Practical Guide to Support Vector Classification. Department of Computer Science and Information Engineering, National Taiwan University (2006)Google Scholar
  12. 12.
    Hsu, C.-W., Lin, C.-J.: A comparison of methods for multi-class Support Vector Machines. IEEE Transactions on Neural Networks 13, 415–425 (2002)CrossRefGoogle Scholar
  13. 13.
    Huang, T.-J., Weng, R.C., Lin, C.-J.: Generalized Bradley-Terry Models and Multi-class Probability Estimates. Journal of Machine Learning Research 7, 85–115 (2006)MathSciNetGoogle Scholar
  14. 14.
    Hülsmann, M.: Vergleich verschiedener kernbasierter Methoden zur Realisierung eines effizienten Multiclass-Algorithmus des Maschinellen Lernens. Master’s thesis, Universität zu Köln (2006)Google Scholar
  15. 15.
    Joachims, T.: Making large-Scale SVM learning practical. In: Advances in Kernel Methods – Support Vector Learning, pp. 41–56. MIT Press, Cambridge (1999)Google Scholar
  16. 16.
    Mencía, E.L.: Paarweises Lernen von Multilabel-Klassifikatoren mit dem Perzeptron-Algorithmus. Master’s thesis, Technische Universität Darmstadt (2006)Google Scholar
  17. 17.
    Meyer, D.: Support Vector Machines, the Interface to libsvm in package e1071. Vignette (2006)Google Scholar
  18. 18.
    Newman, D.J., Hettich, S., Blake, C.L., Merz, C.J.: UCI Repository of machine learning databases (1998),
  19. 19.
    Platt, J.C.: Probabilistic Ouputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In: Proceedings of Advances in Large-Margin Classifiers, pp. 61–74. MIT Press, Cambridge (1999)Google Scholar
  20. 20.
    Roever, C., Raabe, N., Luebke, K., Ligges, U., Szepanek, G., Zentgraf, M.: The klaR package. Manual (2006)Google Scholar
  21. 21.
    Ihaka, R., Gentleman, R.R.: A Language for Data Analysis and Graphics. Journal of Computational and Graphical Statistics 5, 299–314 (1996)CrossRefGoogle Scholar
  22. 22.
    Schölkopf, B., Smola, A.: Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond. MIT Press, Cambridge (2002)Google Scholar
  23. 23.
    Szedmak, S., Shawe-Taylor, J.: Multiclass Learning at One-Class Complexity. Information-Signals, Images, Systems (ISIS Group), Electronics and Computer Science. Technical Report (2005)Google Scholar
  24. 24.
    Tsochantaridis, I., Hofmann, T., Joachims, T., Altun, Y.: Support Vector Machine Learning for Interdependent and Structured Output Spaces. In: Proceedings of the 21th International Conference on Machine Learning. Banff, Canada (2004)Google Scholar
  25. 25.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995)zbMATHGoogle Scholar
  26. 26.
    Wolpert, D.H.: No Free Lunch Theorems for Optimization. In: Proceedings of IEEE Transactions on Evolutionary Computation 1, pp. 67–82 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Marco Hülsmann
    • 1
    • 2
  • Christoph M. Friedrich
    • 2
  1. 1.Universität zu KölnGermany
  2. 2.Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Schloß, Birlinghoven, 53754 Sankt AugustinGermany

Personalised recommendations