Combination of Linear Classifiers Using Score Function – Analysis of Possible Combination Strategies

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 977)


In this work, we addressed the issue of combining linear classifiers using their score functions. The value of the scoring function depends on the distance from the decision boundary. Two score functions have been tested and four different combination strategies were investigated. During the experimental study, the proposed approach was applied to the heterogeneous ensemble and it was compared to two reference methods – majority voting and model averaging respectively. The comparison was made in terms of seven different quality criteria. The result shows that combination strategies based on simple average, and trimmed average are the best combination strategies of the geometrical combination.


Binary classifiers Linear classifiers Geometrical space Potential function 



This work was supported in part by the National Science Centre, Poland under the grant no. 2017/25/B/ST6/01750.


  1. 1.
    Bergmann B, Hommel G (1988) Improvements of general multiple test procedures for redundant systems of hypotheses. In: Multiple hypothesenprüfung/multiple hypotheses testing. Springer, Heidelberg, pp 100–115. Scholar
  2. 2.
    Britto AS, Sabourin R, Oliveira LE (2014) Dynamic selection of classifiers—a comprehensive review. Pattern Recogn 47(11):3665–3680CrossRefGoogle Scholar
  3. 3.
    Burduk R, Walkowiak K (2015) Static classifier selection with interval weights of base classifiers. In: Asian conference on intelligent information and database systems. Springer, pp 494–502Google Scholar
  4. 4.
    Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297. Scholar
  5. 5.
    Cyganek B (2012) One-class support vector ensembles for image segmentation and classification. J Math Imaging Vis 42(2–3):103–117MathSciNetCrossRefGoogle Scholar
  6. 6.
    Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetzbMATHGoogle Scholar
  7. 7.
    Devroye L, Györfi L, Lugosi G (1966) A probabilistic theory of pattern recognition. Springer, New York. Scholar
  8. 8.
    Drucker H, Cortes C, Jackel LD, LeCun Y, Vapnik V (1994) Boosting and other ensemble methods. Neural Comput 6(6):1289–1301CrossRefGoogle Scholar
  9. 9.
    Friedman M (1940) A comparison of alternative tests of significance for the problem of \(m\) rankings. Ann Math Stat 11(1):86–92. Scholar
  10. 10.
    Garcia S, Herrera F (2008) An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. J Mach Learn Res 9:2677–2694zbMATHGoogle Scholar
  11. 11.
    Giacinto G, Roli F (2001) An approach to the automatic design of multiple classifier systems. Pattern Recogn Lett 22:25–33CrossRefGoogle Scholar
  12. 12.
    Gurney K (1997) An introduction to neural networks. Taylor & Francis, London.
  13. 13.
    Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software. SIGKDD Explor Newsl 11(1):10. Scholar
  14. 14.
    Hall MA (1999) Correlation-based feature selection for machine learning. Ph.D. thesis, The University of WaikatoGoogle Scholar
  15. 15.
    Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6(2):65–70. Scholar
  16. 16.
    Hüllermeier E, Fürnkranz J (2010) On predictive accuracy and risk minimization in pairwise label ranking. J Comput Syst Sci 76(1):49–62. Scholar
  17. 17.
    Ko AH, Sabourin R, Britto AS Jr (2008) From dynamic classifier selection to dynamic ensemble selection. Pattern Recogn 41(5):1718–1731CrossRefGoogle Scholar
  18. 18.
    Kuncheva L, Bezdek J (1998) Nearest prototype classification: clustering, genetic algorithms, or random search? IEEE Trans Syst Man Cybern: Part C (Appl Rev) 28(1):160–164. Scholar
  19. 19.
    Kuncheva LI (2004) Combining pattern classifiers: methods and algorithms, 1st edn. Wiley-InterscienceGoogle Scholar
  20. 20.
    Manning CD, Raghavan P, Schutze H (2008) Introduction to information retrieval. Cambridge University Press, New York. Scholar
  21. 21.
    Markiewicz A, Forczmański P (2015) Detection and classification of interesting parts in scanned documents by means of adaboost classification and low-level features verification. In: International conference on computer analysis of images and patterns. Springer, pp 529–540Google Scholar
  22. 22.
    McLachlan GJ (1992) Discriminant analysis and statistical pattern recognition. Wiley series in probability and mathematical statistics: applied probability and statistics. A Wiley-Interscience Publication. Scholar
  23. 23.
    Pearson K (1901) LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philos Mag J Sci 2(11):559–572. Scholar
  24. 24.
    Ponti MP Jr (2011) Combining classifiers: from the creation of ensembles to the decision fusion. In: 2011 24th SIBGRAPI conference on graphics, patterns and images tutorials (SIBGRAPI-T). IEEE, pp 1–10Google Scholar
  25. 25.
    Przybyła-Kasperek M et al (2019) Three conflict methods in multiple classifiers that use dispersed knowledge. Int J Inf Tech Decis Making (IJITDM) 18(02):555–599CrossRefGoogle Scholar
  26. 26.
    Przybyła-Kasperek M, Wakulicz-Deja A (2017) Comparison of fusion methods from the abstract level and the rank level in a dispersed decision-making system. Int J Gener Syst 46(4):386–413MathSciNetCrossRefGoogle Scholar
  27. 27.
    Reif M, Shafait F, Goldstein M, Breuel T, Dengel A (2014) Automatic classifier selection for non-experts. Pattern Anal Appl 17(1):83–96MathSciNetCrossRefGoogle Scholar
  28. 28.
    Rejer I, Burduk R (2017) Classifier selection for motor imagery brain computer interface. In: IFIP international conference on computer information systems and industrial management. Springer, pp 122–130Google Scholar
  29. 29.
    Skurichina M, Duin RP (1998) Bagging for linear classifiers. Pattern Recogn 31(7):909–930. Scholar
  30. 30.
    Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45(4). Scholar
  31. 31.
    Trawiński B, Lasota T, Kempa O, Telec Z, Kutrzyński M (2017) Comparison of ensemble learning models with expert algorithms designed for a property valuation system. In: International conference on computational collective intelligence. Springer, pp 317–327Google Scholar
  32. 32.
    Tulyakov S, Jaeger S, Govindaraju V, Doermann D (2008) Review of classifier combination methods. In: Machine learning in document analysis and recognition. Springer, pp 361–386Google Scholar
  33. 33.
    Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics Bull 1(6):80. Scholar
  34. 34.
    Woźniak M, Graña M, Corchado E (2014) A survey of multiple classifier systems as hybrid systems. Inf Fusion 16:3–17CrossRefGoogle Scholar
  35. 35.
    Xu L, Krzyzak A, Suen CY (1992) Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Trans Syst Man Cybern 22(3):418–435CrossRefGoogle Scholar
  36. 36.
    Yekutieli D, Benjamini Y (2001) The control of the false discovery rate in multiple testing under dependency. Ann Stat 29(4):1165–1188. Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Systems and Computer NetworksWroclaw University of Science and TechnologyWroclawPoland

Personalised recommendations