New Measures of Classifier Competence - Heuristics and Application to the Design of Multiple Classifier Systems

  • Bartlomiej Antosik
  • Marek Kurzynski
Conference paper
Part of the Advances in Intelligent and Soft Computing book series (AINSC, volume 95)


In the paper three new methods based on different heuristics for calculating competence of a classifier are proposed. In the common two-step procedure, first the so-called source competence at validation points are determined and next these competence values are extended to the entire feature space. The first proposition of the source competence reflects both the uncertainty of classifier’s decision and its correctness. In the second method the source competence states the difference of membership degrees to the fuzzy sets of competent and incompetent classifiers. The third method is based on the normalized entropy of supports which classifier gives for particular classes. The dynamic selection (DCS) and dynamic ensemble selection (DES) systems were developed using proposed measures of competence. The performance of multiclassifiers was evaluated using six benchmark databases from the UCI Machine Learning Repository. Classification results obtained for five multiclassifier system with selection and fusion strategy were used for a comparison. The experimental results showed that, regardless of the strategy used by the multiclassifier system, the classification accuracy for homogeneous base classifiers has increased when the measure of competence was employed.


Discriminant Function Membership Degree Validation Point Correct Class Benchmark Database 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alpaydin, E.: Combined 5 x 2 cv F test for comparing supervised classification learning algorithms. Neural Computation 11, 1885–1892 (1999)CrossRefGoogle Scholar
  2. 2.
    Asuncion, A., Newman, D.: UCI Machine Learning Repository. Department of Information and Computer Science. University of California, Irvine (2007), Google Scholar
  3. 3.
    Didaci, L., Giacinto, G., Roli, F., Marcialis, G.: A study on the performances of dynamic classifier selection based on local accuracy estimation. Pattern Recognition 38, 2188–2191 (2005)CrossRefzbMATHGoogle Scholar
  4. 4.
    Duin, R., Juszczak, P., Paclik, P., Pekalska, E., Ridder, D., Tax, D.: PR-Tools 4.1, A Matlab Toolbox for Pattern Recognition, Delft University of Technology (2007),
  5. 5.
    Duda, R., Hart, P., Stork, G.: Pattern Classification. John Wiley and Sons, New York (2000)Google Scholar
  6. 6.
    Eulanda, M., Santos, D., Sabourin, R., Maupin, P.: A dynamic overproduce-and-choose strategy for the selection of classifier ensembles. Pattern Recognition 41, 2993–3009 (2008)CrossRefzbMATHGoogle Scholar
  7. 7.
    Giacinto, G., Roli, F.: Dynamic classifier selection based on multiple classifier behaviour. Pattern Recognition 34, 1879–1881 (2001)CrossRefzbMATHGoogle Scholar
  8. 8.
    Huenupan, F., Yoma, N., et al.: Confidence based multiple classifier fusion in speaker verification. Pattern Recognition Letters 29, 957–966 (2008)CrossRefGoogle Scholar
  9. 9.
    Ko, A., Sabourin, R., Britto, A.: From dynamic classifier selection to dynamic ensemble selection. Pattern Recognition 41, 1718–1733 (2008)CrossRefzbMATHGoogle Scholar
  10. 10.
    Kuncheva, I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley Interscience, Hoboken (2004)CrossRefzbMATHGoogle Scholar
  11. 11.
    Rastrigin, L.A., Erenstein, R.H.: Method of Collective Recognition. Energoizdat, Moscow (1981)zbMATHGoogle Scholar
  12. 12.
    Smits, P.: Multiple classifier systems for supervised remote sensing image classification based on dynamic classifier selection. IEEE Trans. on Geoscience and Remote Sensing 40, 717–725 (2002)CrossRefGoogle Scholar
  13. 13.
    Woloszynski, T., Kurzynski, M.: On a new measure of classifier competence in the feature space. In: Kurzynski, M., Wozniak, M. (eds.) Computer Recognition Systems 3. Advances in Intelligent and Soft Computing, vol. 57, pp. 285–292. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  14. 14.
    Woloszynski, T., Kurzynski, M.: On a new measure of classifier competence applied to the design of multiclassifier systems. In: Foggia, P., Sansone, C., Vento, M. (eds.) ICIAP 2009. LNCS, vol. 5716, pp. 995–1004. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  15. 15.
    Woloszynski, M., Kurzynski, M.: A measure of competence based on randomized reference classifier for dynamic ensemble selection. In: 20th Int. Conf. on Pattern Recognition, pp. 4194–4197. IEEE Computer Press, Istanbul (2010)CrossRefGoogle Scholar
  16. 16.
    Woods, K., Kegelmeyer, W., Bowyer, W.: Combination of multiple classifiers using local accuracy estimates. IEEE Trans. on Pattern Analysis and Machine Learning 19, 405–410 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Bartlomiej Antosik
    • 1
  • Marek Kurzynski
    • 1
  1. 1.Faculty of Electronics, Chair of Systems and Computer NetworksTechnical University of WroclawWroclawPoland

Personalised recommendations