Skip to main content
Log in

Entropy-Based Estimation in Classification Problems

  • Intellectual Control Systems, Data Analysis
  • Published:
Automation and Remote Control Aims and scope Submit manuscript

Abstract

The problem of binary classification is considered, an algorithm for its solution is proposed, based on the method of entropy-based estimation of the decision rule parameters. A detailed description of the entropy-based estimation method and the classification algorithm is given, the advantages and disadvantages of this approach are described, the results of numerical experiments and comparisons with the traditional support vector machine for classification accuracy and degree of dependence on the training sample size are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bishop, C., Pattern Recognition and Machine Learning (Information Science and Statistics), New York: Springer, 2006.

    MATH  Google Scholar 

  2. Breiman, L., Friedman, J.H., Olshen, R.A., and Stone, C.J., Classification and Regression Trees, Monterey: Wadsworth & Brooks/Cole Advanced Books & Software, 1984.

    Google Scholar 

  3. Domingos, P. and Pazzani, M., On the Optimality of the Simple Bayesian Classifier under Zero-One Loss, Mach. Learn., 1997, no. 29, pp. 103–130.

    Article  MATH  Google Scholar 

  4. Hosmer, D.W. and Lemeshow, S., Applied Logistic Regression, New York: Chichester, 2002, 2nd ed.

    MATH  Google Scholar 

  5. Dasarathy, B.V., Ed., Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques, Los Alamitos: IEEE Computer Society Press 1991.

    Google Scholar 

  6. Cristianini, N. and Shawe-Taylor, J., An Introduction to Support Vector Machines and Other Kernel- Based Learning Methods, Cambridge: Cambridge Univ. Press, 2000.

    Book  MATH  Google Scholar 

  7. Bühlmann, P. and Hothorn, T., Boosting Algorithms: Regularization, Prediction and Model Fiting, Stat. Sci., 2007, pp. 477–505.

    MATH  Google Scholar 

  8. Cover, T.M. and Thomas, J.A., Elements of Information Theory, New York: Wiley, 1991.

    Book  MATH  Google Scholar 

  9. Abellán, J. and Castellano, J.G., Improving the Naive Bayes Classifier via a Quick Variable Selection Method Using Maximum of Entropy, Entropy, 2017, vol. 19, no. 6, p. 247.

    Article  Google Scholar 

  10. Phillips, S.J., A Brief Tutorial on Maxent. Network of Conservation Educators and Practitioners, Center for Biodiversity and Conservation, American Museum of Natural History, Lessons in Conservation, 2009, vol. 3, pp. 108–135.

    Google Scholar 

  11. Yu, H.-F., Huang, F.-L., and Lin, C.-J., Dual Coordinate Descent Methods for Logistic Regression and Maximum Entropy Models, Mach. Lear., 2011, vol. 85, pp. 41–75.

    Article  MathSciNet  MATH  Google Scholar 

  12. Golan, A., Judge, G.G., and Miller, D., Maximum Entropy Econometrics: Robust Estimation with Limited Data, Chichester: Wiley, 1996.

    MATH  Google Scholar 

  13. Eruygur, H.O., Generalized Maximum Entropy (GME) Estimator: Formulation and a Monte Carlo Study, VII National Sympos. on Econometrics and Statistics, Istanbul, Turkey, 2005, May 26–27.

    Google Scholar 

  14. Popkov, Yu.S., Dubnov, Yu.A., and Popkov, A.Yu., Randomized Machine Learning: Statement, Solution, Applications, Proc. 2016IEEE 8-th Int. Conf. on Intelligent Systems (IS16), September 4–6, 2016, Sofia, Bulgaria, pp. 27–39.

    Google Scholar 

  15. Langford, J., Tutorial on Practical Prediction Theory for Classification, J. Mach. Learn. Research, 2005, vol. 6, pp. 273–306.

    MathSciNet  MATH  Google Scholar 

  16. Popkov, Yu.S., Volkovich, Z., Dubnov, Yu.A., Avros, R., and Ravve, E., Entropy ‘2’-Soft Classification of Objects, Entropy, 2017, vol. 19, no. 4, no. 178.

    Google Scholar 

  17. Fawcett, T., An Introduction to ROC Analysis, Pattern Recogn. Lett., 2006, no. 27, pp. 861–874.

    Article  Google Scholar 

  18. Alcalá-Fdez, J., Fernandez, A., Luengo, J., Derrac, J., Garcia, S., Sánchez, L., and Herrera, F., KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework, J. Multiple-Valued Logic Soft Computing, 2011, vol. 17, no. 2–3, pp. 255–287.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yu. A. Dubnov.

Additional information

Russian Text © Yu.A. Dubnov, 2019, published in Avtomatika i Telemekhanika, 2019, No. 3, pp. 138–151.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dubnov, Y.A. Entropy-Based Estimation in Classification Problems. Autom Remote Control 80, 502–512 (2019). https://doi.org/10.1134/S0005117919030093

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0005117919030093

Keywords

Navigation