Ensembles of Nested Dichotomies with Multiple Subset Evaluation

  • Tim LeathartEmail author
  • Eibe Frank
  • Bernhard Pfahringer
  • Geoffrey Holmes
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11439)


A system of nested dichotomies (NDs) is a method of decomposing a multiclass problem into a collection of binary problems. Such a system recursively applies binary splits to divide the set of classes into two subsets, and trains a binary classifier for each split. Many methods have been proposed to perform this split, each with various advantages and disadvantages. In this paper, we present a simple, general method for improving the predictive performance of NDs produced by any subset selection techniques that employ randomness to construct the subsets. We provide a theoretical expectation for performance improvements, as well as empirical results showing that our method improves the root mean squared error of NDs, regardless of whether they are employed as an individual model or in an ensemble setting.

Supplementary material

482290_1_En_7_MOESM1_ESM.pdf (40 kb)
Supplementary material 1 (pdf 39 KB)


  1. 1.
    Bengio, S., Weston, J., Grangier, D.: Label embedding trees for large multi-class tasks. In: NIPS, pp. 163–171 (2010)Google Scholar
  2. 2.
    Beygelzimer, A., Langford, J., Lifshits, Y., Sorkin, G., Strehl, A.: Conditional probability tree estimation analysis and algorithms. In: UAI, pp. 51–58 (2009)Google Scholar
  3. 3.
    Beygelzimer, A., Langford, J., Ravikumar, P.: Error-correcting tournaments. In: Gavaldà, R., Lugosi, G., Zeugmann, T., Zilles, S. (eds.) ALT 2009. LNCS (LNAI), vol. 5809, pp. 247–262. Springer, Heidelberg (2009). Scholar
  4. 4.
    Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)zbMATHGoogle Scholar
  5. 5.
    Brier, G.: Verification of forecasts expressed in term of probabilities. Mon. Weather Rev. 78, 1–3 (1950)CrossRefGoogle Scholar
  6. 6.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. JMLR 7(Jan), 1–30 (2006)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Dietterich, T.G., Bakiri, G.: Solving multiclass learning problems via error-correcting output codes. JAIR 2, 263–286 (1995)CrossRefGoogle Scholar
  8. 8.
    Dong, L., Frank, E., Kramer, S.: Ensembles of balanced nested dichotomies for multi-class problems. In: Jorge, A.M., Torgo, L., Brazdil, P., Camacho, R., Gama, J. (eds.) PKDD 2005. LNCS (LNAI), vol. 3721, pp. 84–95. Springer, Heidelberg (2005). Scholar
  9. 9.
    Duarte-Villaseñor, M.M., Carrasco-Ochoa, J.A., Martínez-Trinidad, J.F., Flores-Garrido, M.: Nested dichotomies based on clustering. In: Alvarez, L., Mejail, M., Gomez, L., Jacobo, J. (eds.) CIARP 2012. LNCS, vol. 7441, pp. 162–169. Springer, Heidelberg (2012). Scholar
  10. 10.
    Fox, J.: Applied Regression Analysis, Linear Models, and Related Methods. Sage, Thousand Oaks (1997)Google Scholar
  11. 11.
    Frank, E., Kramer, S.: Ensembles of nested dichotomies for multi-class problems. In: ICML, p. 39. ACM (2004)Google Scholar
  12. 12.
    Freund, Y., Schapire, R.E.: Game theory, on-line prediction and boosting. In: COLT, pp. 325–332 (1996)Google Scholar
  13. 13.
    Fürnkranz, J.: Round robin classification. JMLR 2(Mar), 721–747 (2002)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. ACM SIGKDD Explor. Newsl. 11(1), 10–18 (2009)CrossRefGoogle Scholar
  15. 15.
    Harter, H.L.: Expected values of normal order statistics. Biometrika 48(1/2), 151–165 (1961)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Hastie, T., Tibshirani, R., et al.: Classification by pairwise coupling. Ann. Stat. 26(2), 451–471 (1998)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51(2), 181–207 (2003)CrossRefGoogle Scholar
  18. 18.
    Leathart, T., Frank, E., Holmes, G., Pfahringer, B.: On calibration of nested dichotomies. In: Yang, Q., et al. (eds.) Advances in Knowledge Discovery and Data Mining. LNAI, vol. 11439, pp. 69–80. Springer, Heidelberg (2019)Google Scholar
  19. 19.
    Leathart, T., Pfahringer, B., Frank, E.: Building ensembles of adaptive nested dichotomies with random-pair selection. In: Frasconi, P., Landwehr, N., Manco, G., Vreeken, J. (eds.) ECML PKDD 2016. LNCS (LNAI), vol. 9852, pp. 179–194. Springer, Cham (2016). Scholar
  20. 20.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  21. 21.
    Lichman, M.: UCI machine learning repository (2013)Google Scholar
  22. 22.
    Meilă, M.: Comparing clusterings by the variation of information. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT-Kernel 2003. LNCS (LNAI), vol. 2777, pp. 173–187. Springer, Heidelberg (2003). Scholar
  23. 23.
    Melnikov, V., Hüllermeier, E.: On the effectiveness of heuristics for learning nested dichotomies: an empirical analysis. Mach. Learn. 107(8–10), 1–24 (2018)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Niculescu-Mizil, A., Caruana, R.: Predicting good probabilities with supervised learning. In: ICML, pp. 625–632. ACM (2005)Google Scholar
  25. 25.
    Pimenta, E., Gama, J.: A study on error correcting output codes. In: Portuguese Conference on Artificial Intelligence, pp. 218–223. IEEE (2005)Google Scholar
  26. 26.
    Rifkin, R., Klautau, A.: defense of one-vs-all classification. JMLR 5, 101–141 (2004)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Rodríguez, J.J., García-Osorio, C., Maudes, J.: Forests of nested dichotomies. Pattern Recognit. Lett. 31(2), 125–132 (2010)CrossRefGoogle Scholar
  28. 28.
    Royston, J.: Algorithm AS 177: expected normal order statistics (exact and approximate). J. R. Stat. Soc. Ser. C (Appl. Stat.) 31(2), 161–165 (1982)Google Scholar
  29. 29.
    Wever, M., Mohr, F., Hüllermeier, E.: Ensembles of evolved nested dichotomies for classification. In: GECCO, pp. 561–568. ACM (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Tim Leathart
    • 1
    Email author
  • Eibe Frank
    • 1
  • Bernhard Pfahringer
    • 1
  • Geoffrey Holmes
    • 1
  1. 1.Department of Computer ScienceUniversity of WaikatoHamiltonNew Zealand

Personalised recommendations