Error-correcting output codes for local learners

  • Francesco Ricci
  • David W. Aha
Instance Based Learning
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1398)


Error-correcting output codes (ECOCs) represent classes with a set of output bits, where each bit encodes a binary classification task corresponding to a unique partition of the classes. Algorithms that use ECOCs learn the function corresponding to each bit, and combine them to generate class predictions. ECOCs can reduce both variance and bias errors for multiclass classification tasks when the errors made at the output bits are not correlated. They work well with algorithms that eagerly induce global classifiers (e.g., C4.5) but do not assist simple local classifiers (e.g., nearest neighbor), which yield correlated predictions across the output bits. We show that the output bit predictions of local learners can be decorrelated by selecting different features for each bit. We present promising empirical results for this combination of ECOCs, nearest neighbor, and feature selection.


  1. [Aha and Bankert, 1997]
    D. W. Aha and R. L. Bankert. Cloud classification using error-correcting output codes. Artificial Intelligence Applications: Natural Science, Agriculture, and Environmental Science, 11:13–28, 1997.Google Scholar
  2. [Aha et al., 1991]
    D. W. Aha, D. Kibler, and M. K. Albert. Instance-based learning algorithms. Machine Learning, 6:37–66, 1991.Google Scholar
  3. [Aha, 1992]
    D. W. Aha. Tolerating noisy, irrelevant, and novel attributes in instancebased learning algorithms. International Journal of Man-Machine Studies, 36:267–287, 1992.CrossRefGoogle Scholar
  4. [Bottou and Vapnik, 1992]
    L. Bottou and V. Vapnik. Local learning algorithms. Neural Computation, 4:888–900, 1992.Google Scholar
  5. [Breiman, 1996]
    L. Breiman. Bias, variance, and arcing classifiers. Technical Report 460, University of California, Berkeley, April 1996.Google Scholar
  6. [Dietterich and Bakiri, 1995]
    T. G. Dietterich and G. Bakiri. Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2:263–286, 1995.Google Scholar
  7. [Kohavi and Wolpert, 1996]
    R. Kohavi and D. H. Wolpert. Bias plus variance decomposition for zero-one loss function. In Proceeding of the Thirteenth International Conference on Machine Learning, pages 275–283, Bari, Italy, 1996. Morgan Kaufmann.Google Scholar
  8. [Kong and Dietterich, 1995]
    E. B. Kong and T. G Dietterich. Error-correcting output coding corrects bias and variance. In Proceedings of the Twelfth International Conference on Machine Learning, pages 313–321, Tahoe City, CA, 1995. Morgan Kaufmann.Google Scholar
  9. [Maron and Moore, 1997]
    O. Maron and A. W. Moore. The racing algorithm: Model selection for lazy learners. Artificial Intelligence Review, pages 193–225, 1997.Google Scholar
  10. [Merz and Murphy, 1996]
    C. Merz and P. M. Murphy.UCI repository of machine learning databases. [≈mlearn/MLRepository.html], 1996.Google Scholar
  11. [Perrone and Cooper, 1993]
    M. P. Perrone and L. N. Cooper. When networks disagree: Ensemble methods for hybrid neural networks. In R. J. Mammone, editor, Neural Networks for Speech and Image Processing. Chapman and Hall, Philadelphia, PA, 1993.Google Scholar
  12. [Quinlan, 1993]
    J.R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA, 1993.Google Scholar
  13. [Ricci and Aha, 1997a]
    F. Ricci and D. W. Aha. Bias, variance, and error correcting output codes for local learners. Technical Report 9711-10, IRST, 1997.Google Scholar
  14. [Ricci and Aha, 1997b]
    F. Ricci and D. W. Aha. Extending local learners with error-correcting output codes. Technical Report 9701-08, IRST, 1997.Google Scholar
  15. [Rumelhart et al., 1986]
    D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing: Explorations in the Microstructures of Cognition. MIT Press, Cambridge, MA, 1986.Google Scholar
  16. [Stanfill and Waltz, 1986]
    C. Stanfill and D. Waltz. Toward memory-based reasoning. Communication of ACM, 29:1213–1229, 1986.CrossRefGoogle Scholar
  17. [Zhang et al., 1992]
    X. Zhang, J. Mesirov, and D. Waltz. Hybrid system for protein structure prediction. Journal of Molecular Biology, 225:1049–1063, 1992.CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1998

Authors and Affiliations

  • Francesco Ricci
    • 1
  • David W. Aha
    • 2
  1. 1.Istituto per la Ricerca Scientifica e TecnologicaPovo (TN)Italy
  2. 2.Naval Research Laboratory, Code 5510Navy Center for Applied Research in Artificial IntelligenceWashington, DCUSA

Personalised recommendations