Solve Classification Tasks with Probabilities. Statistically-Modeled Outputs

  • Andrey GritsenkoEmail author
  • Emil Eirola
  • Daniel Schupp
  • Edward Ratner
  • Amaury Lendasse
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10334)


In this paper, an approach for probability-based class prediction is presented. This approach is based on a combination of a newly proposed Histogram Probability (HP) method and any classification algorithm (in this paper results for combination with Extreme Learning Machines (ELM) and Support Vector Machines (SVM) are presented). Extreme Learning Machines is a method of training a single-hidden layer neural network. The paper contains detailed description and analysis of the HP method by the example of the Iris dataset. Eight datasets, four of which represent computer vision classification problem and are derived from Caltech-256 image database, are used to compare HP method with another probability-output classifier [11, 18].


Classification Machine learning Extreme learning machines Gaussian mixture model Multiclass classification Probabilistic classification Histogram distribution Image recognition 


  1. 1.
    Akusok, A., Bjork, K.M., Miche, Y., Lendasse, A.: High performance extreme learning machines: a complete toolbox for big data applications. Access, IEEE (2015)Google Scholar
  2. 2.
    Allen, D.M.: The relationship between variable selection and data agumentation and a method for prediction. Technometrics 16(1), 125–127 (1974)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Bay, H., Tuylelaars, T., Van Gool, L.: Surf speeded up robust features. In: 9th European Conference on Computer Vision, vol. 61, no. 2, pp. 346–359 (2006)Google Scholar
  4. 4.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  5. 5.
    Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)Google Scholar
  6. 6.
    Brier, G.W.: Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 78, 1–3 (1950)CrossRefGoogle Scholar
  7. 7.
    Cambria, E., et al.: Extreme learning machines. IEEE Intell. Syst. 28(6), 30–59 (2013)CrossRefGoogle Scholar
  8. 8.
    Change, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (2011)Google Scholar
  9. 9.
    Dahlquist, G., Björck, Å.: Numerical Methods. Dover Books on Mathematics. Dover Publications, USA (2003)zbMATHGoogle Scholar
  10. 10.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR, pp. 886–893 (2005)Google Scholar
  11. 11.
    Eirola, E., et al.: Extreme learning machines for multiclass classification: refining predictions with gaussian mixture models. In: Rojas, I., Joya, G., Catala, A. (eds.) IWANN 2015. LNCS, vol. 9095, pp. 153–164. Springer, Cham (2015). doi: 10.1007/978-3-319-19222-2_13 CrossRefGoogle Scholar
  12. 12.
    Eirola, E., Lendasse, A., Vandewalle, V., Biernacki, C.: Mixture of gaussians for distance estimation with missing data. Neurocomputing 131, 32–42 (2014)CrossRefGoogle Scholar
  13. 13.
    Fabian, P., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 28252830 (2011)zbMATHMathSciNetGoogle Scholar
  14. 14.
    Fogel, I., Sagi, D.: Gabor filters as texture discriminator. Biol. Cybern. 61(2), 103–113 (1978)Google Scholar
  15. 15.
    Frénay, B., van Heeswijk, M., Miche, Y., Verleysen, M., Lendasse, A.: Feature selection for nonlinear models with extreme learning machines. Neurocomputing 102, 111–124 (2013)CrossRefGoogle Scholar
  16. 16.
    Griffin, G., Holub, A., Perona, P.: Caltech-256 object category dataset. Technical report, CNS-TR-2007-001, California Institute of Technology (2007)Google Scholar
  17. 17.
    Gritsenko, A., Akusok, A., Miche, Y., Björk, K.M., Baek, S., Lendasse, A.: Combined nonlinear visualization and classification: ELMVIS++C. In: The 2016 International Joint Conference on Neural Networks (IJCNN), pp. 2617–2624, IEEE World Congress on Computational Intelligence, July 2016Google Scholar
  18. 18.
    Gritsenko, A., Eirola, E., Schupp, D., Ratner, E., Lendasse, A.: Probabilistic methods for multiclass classification problems. In: Cao, J., Mao, K., Wu, J., Lendasse, A. (eds.) Proceedings of ELM-2015, vol. 7, pp. 375–397. Springer, Berlin (2016)Google Scholar
  19. 19.
    Hand, D.J., Yu, K.: Idiot’s bayes - not so stupid after all? Int. Stat. Rev./Revue Internationale de Statistique 69(3), 385–398 (2001)zbMATHGoogle Scholar
  20. 20.
    Hsu, C.W., Lin, C.J.: A comparison of methods for multiclass support vector machines. IEEE Trans. Neural Netw. 13(2), 415–425 (2002)CrossRefGoogle Scholar
  21. 21.
    Lendasse, A., Akusok, A., Simula, O., Corona, F., Heeswijk, M., Eirola, E., Miche, Y.: Extreme learning machine: a robust modeling technique? Yes! In: Rojas, I., Joya, G., Gabestany, J. (eds.) IWANN 2013. LNCS, vol. 7902, pp. 17–35. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38679-4_2
  22. 22.
    Lichman, M.: UCI ML Repository (2013).
  23. 23.
    Miche, Y., van Heeswijk, M., Bas, P., Simula, O., Lendasse, A.: TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization. Neurocomputing 74(16), 2413–2421 (2011)CrossRefGoogle Scholar
  24. 24.
    Miche, Y., Sorjamaa, A., Bas, P., Simula, O., Jutten, C., Lendasse, A.: OP-ELM: optimally pruned extreme learning machine. IEEE Trans. Neural Netw. 21(1), 158–162 (2010)CrossRefGoogle Scholar
  25. 25.
    Miche, Y., Sorjamaa, A., Lendasse, A.: OP-ELM: theory, experiments and a toolbox. In: Kůrková, V., Neruda, R., Koutník, J. (eds.) ICANN 2008. LNCS, vol. 5163, pp. 145–154. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-87536-9_16 CrossRefGoogle Scholar
  26. 26.
    Nian, R., He, B., Zheng, B., Van Heeswijk, M., Yu, Q., Miche, Y., Lendasse, A.: Extreme learning machine towards dynamic model hypothesis in fish ethology research. Neurocomputing 128, 273–284 (2014)CrossRefGoogle Scholar
  27. 27.
    Patil, A.S., Pawar, B.: Automated classification of web sites using naive bayesian algorithm. In: Proceedings of the International MultiConference of Engineers and Computer Scientists, vol. 1, pp. 519–523 (2012)Google Scholar
  28. 28.
    Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, pp. 61–74. MIT Press (1999)Google Scholar
  29. 29.
    Pouzols, F.M., Lendasse, A.: Evolvin fuzzy optimally pruned extreme learning machine for regression problems. Evolving Syst. 1(1), 43–58 (2010)CrossRefGoogle Scholar
  30. 30.
    Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical Recipes 3rd Edition: The Art of Scientific Computing, 3rd edn. Cambridge University Press, New York (2007)zbMATHGoogle Scholar
  31. 31.
    Robertson, T., Wright, F., Dykstra, R.: Order Restricted Statistical Inference. Probability and Statistics Series. Wiley, Chichester (1988)zbMATHGoogle Scholar
  32. 32.
    Schupp-Omid, D.R., Ratner, E., Gritsenko, A.: Object categorization using statistically-modeled classifier outputs, August 2016., U.S. Patent Application 2017/0046615 A1, Appl. No. 15/237048
  33. 33.
    Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6(2), 461–464 (1978)CrossRefzbMATHMathSciNetGoogle Scholar
  34. 34.
    Westgard, J.O., Carey, R.N., Wold, S.: Criteria for judging precision and accuracy in method development and evaluation. Clin. Chem. 20(7), 825–833 (1974)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Andrey Gritsenko
    • 1
    • 2
    • 3
    Email author
  • Emil Eirola
    • 4
  • Daniel Schupp
    • 3
  • Edward Ratner
    • 3
  • Amaury Lendasse
    • 1
    • 2
    • 4
  1. 1.Department of Industrial EngineeringThe University of IowaIowa CityUSA
  2. 2.Iowa Informatics InitiativeThe University of IowaIowa CityUSA
  3. 3.Lyrical Labs LLCIowa CityUSA
  4. 4.Arcada University of Applied SciencesHelsinkiFinland

Personalised recommendations