evolutionary Design of Code-matrices for Multiclass Problems

  • Ana Carolina Lorena
  • André C. P. L. F. de Carvalho

Several real problems involve the classification of data into categories or classes. Given a dataset containing data whose classes are known, Machine Learning algorithms can be employed for the induction of a classifier able to predict the class of new data from the same domain, performing the desired discrimination. Several machine learning techniques are originally conceived for the solution of problems with only two classes. In multiclass applications, an alternative frequently employed is to divide the original problem into binary subtasks, whose results are then combined. The decomposition can be generally represented by a code-matrix, where each row corresponds to a codeword assigned for one class and the columns represent the binary classifiers employed. This chapter presents a survey on techniques for multiclass problems code-matrix design. It also shows how evolutionary techniques can be employed to solve this problem.


Decomposition Strategy Evolutionary Design Strength Pareto Evolutionary Algorithm Multiclass Problem Multiclass Support Vector Machine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Alba, E., Cotta, C., Chicano, F., Nebro, A.J., (2002), Parallel evolutionary algorithms in telecommunications: two case studies. In: Proceedings of Congresso Argentino de Ciências de la Computación.Google Scholar
  2. Alba, E., Chicano, J.F., (2004), Solving the error correcting code problem with parallel hybrid heuristics. In: Proceedings of 2004 ACM Symposium on Applied Computing. Volume 2. 985-989.Google Scholar
  3. Allwein, E.L., Shapire, R.E., Singer, Y., (2000), Reducing multiclass to binary: a unifying approach for magin classifiers. In: Proceedings of the 17th International Conference on Machine Learning, Morgan Kaufmann 9-16.Google Scholar
  4. Alpaydin, E., Mayoraz, E., (1999), Learning error-correcting output codes from data. In: Proceedings of the 9th International Conference on Neural Networks. 743-748.Google Scholar
  5. Beasley, D. (2000), (Bäck et al., 2000) 4-18Google Scholar
  6. Berger, A., (1999), Error-correcting output coding for text classification.Google Scholar
  7. Blake, C.L., Merz, C.J., (1998), UCI repository of machine learning databases. Available at:∼mlearn/MLRepository.html.
  8. Boser, R.C., Ray-Chaudhuri, D.K., 1960, On a class of error-correcting binary group codes. Information and Control 3 68-79.CrossRefMathSciNetGoogle Scholar
  9. Bäck, T., Fogel, D.B., Michalewicz, T., (2000), Evolutionary Computation 1: Basic Algorithms and Operators. Institute of Physics Publishing.Google Scholar
  10. Bäck, T. (2000), (Bäck et al., 2000) 132-135Google Scholar
  11. Collins, M., Shapire, R.E., Singer, Y., 2002, Logistic regression, adaboost and bregman distances. Machine Learning 47(2/3) 253-285.CrossRefGoogle Scholar
  12. Crammer, K., Singer, Y., 2002, On the learnability and design of output codes for multiclass problems. Machine Learning 47(2-3) 201-233.zbMATHCrossRefGoogle Scholar
  13. Cristianini, N., Shawe-Taylor, J., (2000), An introduction to Support Vector Machines and other kernel-based learning methods. Cambridge University Press.Google Scholar
  14. Darwin, C., 1859, On the origin of species by means of natural selection. John Murray, London.Google Scholar
  15. Deb, K., 2000, An efficient constraint handling method for genetic algorithms. Computer Methods in Applied Mechanics and Engineering 186 311-338.zbMATHCrossRefGoogle Scholar
  16. Dekel, O., Singer, Y., (2003), Multiclass learning by probabilistic embeddings. In: Advances in Neural Information Processing Systems. Volume 15., MIT Press 945-952.Google Scholar
  17. Dietterich, T.G., Bariki, G., 1995, Solving multiclass learning problems via errorcorrecting output codes. Journal of Artificial Intelligence Research 2 263-286.zbMATHGoogle Scholar
  18. Dontas, K., Jong, K.D., (1990), Discovery of maximal distance codes using genetic algorithms. In: Proceedings of the 2nd International IEEE Conference on Tools for Artificial Intelligence, IEEE Computer Society Press 905-811.Google Scholar
  19. Eiben, A.E., Smith, J.E., (2003), Introduction to Evolutionary Computing. Springer.Google Scholar
  20. Escalera, S., Pujol, O., Radeva, R., (2006), Decoding of ternary error correcting output codes. In: Proceedings of the 11th Iberoamerican Congress on Pattern Recognition. Volume 4225 of Lecture Notes in Computer Science., SpringerVerlag 753-763.Google Scholar
  21. Freund, Y., Schapire, R.E., 1997, A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 1(55) 119-139.CrossRefMathSciNetGoogle Scholar
  22. Fürnkranz, J., 2002, Round robin classification. Journal of Machine Learning Research 2 721-747.zbMATHCrossRefGoogle Scholar
  23. Ghani, R., (2000), Using error correcting output codes for text classification. In: Proceedings of the 17th International Conference on Machine Learning, Morgan Kaufmann 303-310.Google Scholar
  24. Hastie, T., Tibshirani, R., 1998, Classification by pairwise coupling. The Annals of Statistics 2 451-471.MathSciNetGoogle Scholar
  25. Haykin, S., 1999, Neural Networks - A Compreensive Foundation. 2nd edn. Prentice-Hall, New Jersey.Google Scholar
  26. Holland, J.H., (1975), Adaptation in Natural and Artificial Systems. University of Michigan Press.Google Scholar
  27. Hsu, C.W., Lin, C.J., 2002, A comparison of methods for multi-class support vector machines. IEEE Transactions on Neural Networks 13(2) 415-425.Google Scholar
  28. Klautau, A., Jevtić, N., Orlistky, A., 2003, On nearest-neighbor error-correcting output codes with application to all-pairs multiclass support vector machines. Journal of Machine Learning Research 4 1-15.CrossRefGoogle Scholar
  29. Knerr, S., Personnaz, L., Dreyfus, G., 1992, Handwritten digit recognition by neural networks with single-layer training. IEEE Transactions on Neural Networks 3(6) 962-968.CrossRefGoogle Scholar
  30. Knerr, S., Personnaz, L., Dreyfus, G., (1990), In: Single-layer learning revisited: a stepwise procedure for building and training a neural network. Springer-Verlag, pp. 41-50Google Scholar
  31. Kreβel, U., (1999), Pairwise classification and support vector machines. In Schölkopf, B., Burges, C.J.C., Smola, A.J., eds.: Advances in Kernel Methods - Support Vector Learning, MIT Press 185-208.Google Scholar
  32. Kuncheva, L.I., 2005, Using diversity measures for generating error-correcting output codes in classifier ensembles. Pattern Recognition Letters 26 83-90.CrossRefGoogle Scholar
  33. Lorena, A.C., Carvalho, A.C.P.L.F., (2006), Evolutionary design of multiclass support vector machines. Journal of Intelligent and Fuzzy Systems . Accepted, to be published..Google Scholar
  34. Lorena, A.C., (2006), Investigação de estratégias para a geração de máquinas de vetores de suporte multiclasses [in portuguese], Ph.D. thesis, Departamento de Ciências de Computação, Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, São Carlos, Brazil, teses/disponiveis/55/55134/tde- 26052006- 111406.
  35. ı, R., Laguna, M., Campos, V., (2005), Scatter search vs. genetic algorithms: An experimental evaluation with permutation problems. In Rego, C., Alidaee, B., eds.: Metaheuristic Optimization Via Adaptive Memory and Evolution: Tabu Search and Scatter Search. Kluwer Academic Publishers 263-282.Google Scholar
  36. Masulli, F., Valentini, G., (2000), Effectiveness of error correcting output codes in multiclass learning problems. In: Proceedings of the 1st International Workshop on Multiple Classifier Systems. Volume 1857 of Lecture Notes in Computer Science., Springer-Verlag 107-116.Google Scholar
  37. Mayoraz, E., Alpaydim, E., 1998, Support vector machines for multi-class classification. Research Report IDIAP-RR-98-06, Dalle Molle Institute for Perceptual Artificial Intelligence, Martigny, Switzerland.Google Scholar
  38. Mayoraz, E., Moreira, M., 1996, On the decomposition of polychotomies into dichotomies. Research Report 96-08, IDIAP, Dalle Molle Institute for Perceptive Artificial Intelligence, Martigny, Valais, Switzerland.Google Scholar
  39. Michalewicz, Z., Fogel, D.B., (2004), How to solve it: modern heuristics. Springer. Mitchell, T., (1997), Machine Learning. McGraw Hill.Google Scholar
  40. Mitchell, M., (1999), An introduction to Genetic Algorithms. MIT Press.Google Scholar
  41. Passerini, A., Pontil, M., Frasconi, P., 2004, New results on error correcting output codes of kernel machines. IEEE Transactions on Neural Networks 15 45-54.CrossRefGoogle Scholar
  42. Pimenta, E., Gama, J., (2005), A study on error correcting output codes. In: Proceedings of the 2005 Portuguese Conference on Artificial Intelligence, IEEE Computer Society Press 218-223.Google Scholar
  43. Pimenta, E.M.C., 2005, Abordagens para decomposição de problemas multiıda (in portuguese). Master’s thesis, Departamento de Ciências de Computadores, Faculdade de Ciências da Universidade do Porto, Portugal.Google Scholar
  44. Pujol, O., Tadeva, P., Vitrià, J., 2006, Discriminant ECOC: a heuristic method for application dependetn design of error correcting output codes. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(6) 1007-1012.CrossRefGoogle Scholar
  45. ıguez, A., (2002), Puncturing multi-class support vector machines. In: Proceedings of the 12th International Conference on Neural Networks (ICANN). Volume 2415 of Lecture Notes in Computer Science., Springer-Verlag 751-756.Google Scholar
  46. Quinlan, J.R., 1986, Induction of decision trees. Machine Learning 1(1) 81-106. Rifkin, R., Klautau, A., (2004), In defense of one-vs-all classification. Journal of Machine Learning Research 5 1533-7928.Google Scholar
  47. Rätsch, G., Smola, A.J., Mika, S., (2003), Adapting codes and embeddings for polychotomies. In: Advances in Neural Information Processing Systems. Volume 15., MIT Press 513-520.Google Scholar
  48. Shen, L., Tan, E.C., (2005), Seeking better output-codes with genetic algorithm for multiclass cancer classification. Submitted to Bioinformatics.Google Scholar
  49. Simn, M.D.J., Pulido, J.A.G., Rodrguez, M.A.V., (2006), Prez, J.M.S., Criado, J.M.G., A genetic algorithm to design error correcting codes. In: Proceedings of the 13th IEEE Mediterranean Eletrotechnical Conference 2006, IEEE Computer Society Press 807-810.Google Scholar
  50. Statnikov, A., Aliferis, C.F., Tsamardinos, I., 2005, Hardin, D., Levy, S., A comprehensive evaluation of multicategory methods for microarray gene expression cancer diagnosis. Bioinformatics 21(5) 631-643.Google Scholar
  51. ıa-Villalba, J., Villena, J., (2001), Recursive adaptive ECOC models. In: Proceedings of the 10th Portuguese Conference on Artificial Intelligence. Volume 2258 of Lecture Notes in Artificial Intelligence., Springer-Verlag 96-103.Google Scholar
  52. ıa-Villalba, J., (2003), Good error correcting output codes for adaptive multiclass learning. In: Proceedings of the 4th International Workshop on Multiple Classifier Systems 2003. Volume 2709 of Lecture Notes in Computer Science., Springer-Verlag 156-165.Google Scholar
  53. Wallet, B.C., Marchette, D.J., Solka, J.L., (1996), A matrix representation for genetic algorithms. In: Automatic object recognition VI, Proceedings of the International Society for Optical Engineering. 206-214.Google Scholar
  54. Wallis, J.L., Houghten, S.K., (2002), A comparative study of search techniques applied to the minimum distance problem of BCH codes. Technical Report CS-02-08, Department of Computer Science, Brock University.Google Scholar
  55. Windeatt, T., Ghaderi, R., 2003, Coding and decoding strategies for multi-class learning problems. Information Fusion 4(1) 11-21.CrossRefGoogle Scholar
  56. Zhang, A., Wu, Z.L., Li, C.H., Fang, K.T., (2003), On hadamard-type output coding in multiclass learning. In: Proceedings of IDEAL. Volume 2690 of Lecture Notes in Computer Science., Springer-Verlag 397-404.Google Scholar
  57. Zitzler, E., Laumanns, M., Thiele, L., (2002), SPEA2: Improving the strength pareto evolutionary algorithm. In: Evolutionary Methods for Design, Optimisation, and Control, CIMNE, Barcelona, Spain. 95-100.Google Scholar
  58. Zitzler, E., Laumanns, M., Bleuler, S., (2004), A tutorial on evolutionary multiobjective optimization. In Gandibleux, X., Sevaux, M., Srensen, K., T’kindt, V., eds.: Metaheuristics for Multiobjective Optimisation. Volume 535 of Lecture Notes in Economics and Mathematical Systems., Springer-Verlag 3-37.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Ana Carolina Lorena
    • 1
  • André C. P. L. F. de Carvalho
    • 2
  1. 1.Centro de Matemática, Computação e CogniçãoUniversidade Federal do ABCBrazil
  2. 2.Ciências de ComputaçãoInstituto de Ciências Matemáticas e de ComputaçãoBrazil

Personalised recommendations