Abstract
Along with the increase of data and information, incremental learning ability turns out to be more and more important for machine learning approaches. The online algorithms try not to remember irrelevant information instead of synthesizing all available information (as opposed to classic batch learning algorithms). In this study, we attempted to increase the prediction accuracy of an incremental version of Naive Bayes model by integrating instance based learning. We performed a large-scale comparison of the proposed method with other state-of-the-art algorithms on several datasets and the proposed method produce better accuracy in most cases.
Similar content being viewed by others
References
P. Auer and M. Warmuth, “Tracking the best disjunction,” Machine Learning, vol. 32, no. 2, pp. 127–150, 1998.
F. Chu and C. Zaniolo, “Fast and light boosting for adaptive mining of data streams,” Lecture Notes in Computer Science, vol. 3056, pp. 282–292, 2004.
J. G. Cleary and L. E. Trigg, “K*: an instance-based learner using an entropic distance measure,” Proc. of the 12th International Conference on Machine Learning, pp. 108–114, 1995.
W. Cohen, “fast effective rule induction,” Proc. of Int Conf. of ML-95, pp. 115–123, 1995.
P. Domingos and M. Pazzani, “On the optimality of the simple Bayesian classifier under zero-one loss,” Machine Learning, vol. 29, no. 2–3, pp. 103–130, 1997.
W. Fan, S. Stolfo, and J. Zhang, “The application of AdaBoost for distributed, scalable and online learning,” Proc. of Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, pp. 362–366, 1999.
F. Fdez-Riverola, E. L. Iglesias, F. Díaz, J. R. Méndez, and J. M. Corchado, “Applying lazy learning algorithms to tackle concept drift in spam filtering,” Expert Systems with Applications, vol. 33, no. 1, pp. 36–48, 2007.
A. Fern and R. Givan, “Online ensemble learning: An empirical study,” Proc. of the 17th International Conference on ML, pp. 279–286, 2000.
A. Frank and A. Asuncion, UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, 2010.
Y. Freund and R. Schapire, “Large margin classification using the perceptron algorithm,” Machine Learning, vol. 37, no. 3, pp. 277–296, 1999.
A. Gangardiwala and R. Polikar, “Dynamically weighted majority voting for incremental learning and comparison of three boosting based approaches,” Proc. of Joint Conf. on Neural Networks, pp. 1131–1136, 2005.
M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The WEKA data mining software: an update,” SIGKDD Explorations, vol. 11, no. 1, pp. 10–18, 2009.
L. I. Kuncheva, “Classifier ensembles for changing environments,” Lecture Notes in Computer Science, vol. 3077, pp. 1–15, 2004.
L. Kai and H.-K. Huang, “Incremental learning proximal support vector machine classifiers,” Proc. of International Conference on Machine Learning and Cybernetics, pp. 1635–1637, 2002.
J. Lee, W. Chung, E. Kim, and S. Kim, “A new genetic approach for structure learning of Bayesian networks: matrix genetic algorithm,” International Journal of Control, Automation and Systems, vol. 8, no. 2, pp 398–407, 2010.
Z. Liang and Y. Li, “Incremental support vector machine learning in the primal and applications,” Neurocomputing, vol. 72, no. 10–12, pp. 2249–2258, 2009.
N. Littlestone and M. Warmuth, “The weighted majority algorithm,” Information and Computation, vol. 108, pp. 212–261, 1994.
L. Jing, L. Xue, and Z. Weicai, “Ambiguous decision trees for mining concept-drifting data streams,” Pattern Recognition Letters, vol. 30, no. 15, pp. 1347–1355, 2009.
N. C. Oza and S. Russell, “Online bagging and boosting,” Proc. of Artificial Intelligence and Statistics 2001, pp. 105–112, 2001.
J. R. Quinlan, C4.5: Programs for Machine Learning, Morgan Kaufmann, San Francisco, 1993.
D. Saad, Online Learning in Neural Networks, London, Cambridge University Press, 1998.
M. Sahami, “Learning limited dependence Bayesian classifiers,” Proc. of the 2nd Int. Conf. on Knowledge Discovery in Databases, pp. 335–338, 1996.
R. Sylvain, Nearest Neighbor with Generalization, Christchurch, New Zealand, 2002.
C.-J. Tsai, C.-I. Lee, and W.-P. Yang, “Mining decision rules on data streams in the presence of concept drifts,” Expert Systems with Applications, vol. 36, no. 2, pp. 1164–1178, 2009.
A. Tsymbal, M. Pechenizkiy, P. Cunningham, and S. Puuronen, “Dynamic integration of classifiers for handling concept drift,” Information Fusion, vol. 9, no. 1, pp. 56–68, 2008.
P. Utgoff, N. Berkman, and J. Clouse, “Decision tree induction based on efficient tree restructuring,” Machine Learning, vol. 29, no. 1, pp. 5–44, 1997.
G. I. Webb, J. R. Boughton, and Z. Wang, “Not so naive bayes: aggregating one-dependence estimators,” Machine Learning, vol. 58, pp. 5–24, 2005.
G. Widmer and M. Kubat, “Learning in the presence of concept drift and hidden contexts,” Machine Learning, vol. 23, pp. 69–101, 1996.
I. Witten, E. Frank, and M. Hall, Data Mining: Practical Machine Learning Tools and Techniques, Morgan Kaufmann, 2011.
H.-G. Yeom, S.-M. Park, J. Park, and K.-B. Sim, “Superiority demonstration of variance-considered machines by comparing error rate with support vector machines,” Int. Journal of Control, Automation, and Systems, vol. 9, no. 3, pp. 595–600, 2011.
Author information
Authors and Affiliations
Corresponding author
Additional information
Recommended by Editorial Board member Yuan Fang Zheng under the direction of Editor Myotaeg Lim.
Sotiris Kotsiantis received his bachelor in mathematics in 1999, a Master degree in 2001 and a Ph.D. degree in computer science in 2005 from the University of Patras, Greece. His research interests are mainly in the field of data mining and machine learning. He has a lot of publications to his credit in international journals and conferences.
Rights and permissions
About this article
Cite this article
Kotsiantis, S. Increasing the accuracy of incremental naive bayes classifier using instance based learning. Int. J. Control Autom. Syst. 11, 159–166 (2013). https://doi.org/10.1007/s12555-011-0099-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12555-011-0099-1