Abstract
The problem of binary classification is considered, an algorithm for its solution is proposed, based on the method of entropy-based estimation of the decision rule parameters. A detailed description of the entropy-based estimation method and the classification algorithm is given, the advantages and disadvantages of this approach are described, the results of numerical experiments and comparisons with the traditional support vector machine for classification accuracy and degree of dependence on the training sample size are presented.
Similar content being viewed by others
References
Bishop, C., Pattern Recognition and Machine Learning (Information Science and Statistics), New York: Springer, 2006.
Breiman, L., Friedman, J.H., Olshen, R.A., and Stone, C.J., Classification and Regression Trees, Monterey: Wadsworth & Brooks/Cole Advanced Books & Software, 1984.
Domingos, P. and Pazzani, M., On the Optimality of the Simple Bayesian Classifier under Zero-One Loss, Mach. Learn., 1997, no. 29, pp. 103–130.
Hosmer, D.W. and Lemeshow, S., Applied Logistic Regression, New York: Chichester, 2002, 2nd ed.
Dasarathy, B.V., Ed., Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques, Los Alamitos: IEEE Computer Society Press 1991.
Cristianini, N. and Shawe-Taylor, J., An Introduction to Support Vector Machines and Other Kernel- Based Learning Methods, Cambridge: Cambridge Univ. Press, 2000.
Bühlmann, P. and Hothorn, T., Boosting Algorithms: Regularization, Prediction and Model Fiting, Stat. Sci., 2007, pp. 477–505.
Cover, T.M. and Thomas, J.A., Elements of Information Theory, New York: Wiley, 1991.
Abellán, J. and Castellano, J.G., Improving the Naive Bayes Classifier via a Quick Variable Selection Method Using Maximum of Entropy, Entropy, 2017, vol. 19, no. 6, p. 247.
Phillips, S.J., A Brief Tutorial on Maxent. Network of Conservation Educators and Practitioners, Center for Biodiversity and Conservation, American Museum of Natural History, Lessons in Conservation, 2009, vol. 3, pp. 108–135.
Yu, H.-F., Huang, F.-L., and Lin, C.-J., Dual Coordinate Descent Methods for Logistic Regression and Maximum Entropy Models, Mach. Lear., 2011, vol. 85, pp. 41–75.
Golan, A., Judge, G.G., and Miller, D., Maximum Entropy Econometrics: Robust Estimation with Limited Data, Chichester: Wiley, 1996.
Eruygur, H.O., Generalized Maximum Entropy (GME) Estimator: Formulation and a Monte Carlo Study, VII National Sympos. on Econometrics and Statistics, Istanbul, Turkey, 2005, May 26–27.
Popkov, Yu.S., Dubnov, Yu.A., and Popkov, A.Yu., Randomized Machine Learning: Statement, Solution, Applications, Proc. 2016IEEE 8-th Int. Conf. on Intelligent Systems (IS16), September 4–6, 2016, Sofia, Bulgaria, pp. 27–39.
Langford, J., Tutorial on Practical Prediction Theory for Classification, J. Mach. Learn. Research, 2005, vol. 6, pp. 273–306.
Popkov, Yu.S., Volkovich, Z., Dubnov, Yu.A., Avros, R., and Ravve, E., Entropy ‘2’-Soft Classification of Objects, Entropy, 2017, vol. 19, no. 4, no. 178.
Fawcett, T., An Introduction to ROC Analysis, Pattern Recogn. Lett., 2006, no. 27, pp. 861–874.
Alcalá-Fdez, J., Fernandez, A., Luengo, J., Derrac, J., Garcia, S., Sánchez, L., and Herrera, F., KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework, J. Multiple-Valued Logic Soft Computing, 2011, vol. 17, no. 2–3, pp. 255–287.
Author information
Authors and Affiliations
Corresponding author
Additional information
Russian Text © Yu.A. Dubnov, 2019, published in Avtomatika i Telemekhanika, 2019, No. 3, pp. 138–151.
Rights and permissions
About this article
Cite this article
Dubnov, Y.A. Entropy-Based Estimation in Classification Problems. Autom Remote Control 80, 502–512 (2019). https://doi.org/10.1134/S0005117919030093
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S0005117919030093