Abstract
The naive Bayes classifier continues to be a popular learning algorithm for data mining applications due to its simplicity and linear run-time. Many enhancements to the basic algorithm have been proposed to help mitigate its primary weakness—the assumption that attributes are independent given the class. All of them improve the performance of naive Bayes at the expense (to a greater or lesser degree) of execution time and/or simplicity of the final model. In this paper we present a simple filter method for setting attribute weights for use with naive Bayes. Experimental results show that naive Bayes with attribute weights rarely degrades the quality of the model compared to standard naive Bayes and, in many cases, improves it dramatically. The main advantages of this method compared to other approaches for improving naive Bayes is its run-time complexity and the fact that it maintains the simplicity of the final model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
D. W. Aha. Tolerating noisy, irrelevant, and novel attributes in instance-based learning algorithms. Int. Journal of Man-Machine Studies, 36:267–287, 1992.
C.L. Blake and C.J. Merz. UCI repository of machine learning databases. University of California, Irvine, Dept. of Information and Computer Science, 1998. [www.ics.uci.edu/∼mlearn/MLRepository.html].
C. Cardie. Using decision trees to improve case-based learning. In Proc. of the 10th Int. Conf on Machine Learning, pages 25–32. Morgan Kaufmann, 1993.
C. Cardie and N. Howe. Improving minority class prediction using case-specific feature weights. In Proc. of the 14th Int. Conf. on Machine Learning, pages 57–65. Morgan Kaufmann, 1997.
R. H. Creecy, B. M. Masand, S. J. Smith, and D. L. Waltz. Trading MIPS and memory for knowledge engineering. Communications of the ACM, 35:48–64, 1992.
P. Domingos. Context-sensitive feature selection for lazy learners. Artificial Intelligence Review, 11(227–253), 1997.
P. Domingos and M. J. Pazzani. On the optimality of the simple Bayesian classifier under zero-one loss. Mach. Learning, 29(2–3):103–130, 1997.
U. M. Fayyad and K. B. Irani. Multi-interval discretization of continuous-valued attributes for classification learning. In Proc. of the 13th Int. Joint Conf. on AI, pages 1022–1027. Morgan Kaufmann, 1993.
J. T. A. S. Ferreira, D. G. T Denison, and D. J. Hand. Data mining with products of trees. In Proc. of the 4th Int. Conf on Advances in Intelligent Data Analysis, pages 167–176. Springer, 2001.
M. Hall. Correlation-based feature selection for discrete and numeric class machine learning. In Proc. of the 17th Int. Conf. on Machine Learning, pages 359–366, 2000.
N. Howe and C. Cardie. Examining locally varying weights for nearest neighbor algorithms. In Case-Based Reasoning Research and Development: 2nd Int. Conf on Case-Based Reasoning, pages 455–466. Springer, 1997.
G. John, R. Kohavi, and K. Pfleger. Irrelevant features and the subset selection problem. In Proc. of the 11th Int. Conf. on Machine Learning, pages 121–129. Morgan Kaufmann, 1994.
S. Kim, H. Seo, and H. Rim. Poisson naive Bayes for text classification with feature weighting. In Proc. of the 6th Int. Workshop on Information Retrieval with Asian Languages, pages 33–40, 2003.
K. Kira and L. Rendell. A practical approach to feature selection. In Proc. of the Ninth Int. Conf. on Machine L earning, pages 249–256. Morgan Kaufmann, 1992.
R. Kohavi. Scaling up the accuracy of naive-Bayes classifiers: a decision tree hybrid. In Proc. of the 2nd Int. Conf. on Knowledge Discovery and Data Mining, pages 202–207, 1996.
R. Kohavi, P. Langley, and Y. Yun. The utility of feature weighting in nearest-neighbor algorithms.In M. van Someren and G. Widmer, editors, Poster Papers: Ninth European Conf. on Machine Learning, Prague, Czech Republic, 1997. Unpublished.
M. Kubat, D. Flotzinger, and G. Pfurtscheller. Discovering patterns in EEG signals: Comparative study of a few methods. In Proc. of the 1993 Europ. Conf. on Mach. Learn., pages 367–371. Springer-Verlag, 1993.
P. Langley and S. Sage. Induction of selective Bayesian classifiers. In Proc. of the 10th Conf. on Uncertainly in Artificial Intelligence, pages 399–406. Morgan Kaufmann, 1994.
C. Nadeau and Yoshua Bengio. Inference for the generalization error. In Advances in Neural Information Processing Systems 12, pages 307–313. MIT Press, 1999.
R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.
C. A. Ratanamahatana and D. Gunopulos. Feature selection for the naive Bayesian classifier using decision trees. Applied Artificial Intelligence, 17(5–6):475–487, 2003.
M. Robnik-Sikonja and I. Kononenko. Theoretical and empirical analysis of Relieffand RRelieff. Mach. Learning, 53(1–2):23–69, 2003.
S. L. Salzberg. A nearest hyperrectangle learning method. Machine Learning, 6:251–276, 1991.
C. Stanfill and D. Waltz. Toward memory-based reasoning. Communica tions of the Assoc. for Computing Machinery, 29:1213–1228, 1986.
D. Wettschereck, D. W. Aha, and T. Mohri. A review and empirical comparison of feature weighting methods for a class of lazy learning algorithms. Artificial Intelligence Review, 11:273–314, 1997.
Ian H. Witten and Eibe Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations.Morgan Kaufmann, 2000.
H. Zhang and S. Sheng. Learning weighted naive Bayes with accurate ranking. In Proc. of the 4th IEEE Int. Conf. on Data Mining, pages 567–570, 2004.
Zijian Zheng and Geoffrey I. Webb. Lazy learning of Bayesian rules. Machine Learning, 41(1):53–84, 2000.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2007 Springer-Verlag London Limited
About this paper
Cite this paper
Hall, M. (2007). A Decision Tree-Based Attribute Weighting Filter for Naive Bayes. In: Bramer, M., Coenen, F., Tuson, A. (eds) Research and Development in Intelligent Systems XXIII. SGAI 2006. Springer, London. https://doi.org/10.1007/978-1-84628-663-6_5
Download citation
DOI: https://doi.org/10.1007/978-1-84628-663-6_5
Publisher Name: Springer, London
Print ISBN: 978-1-84628-662-9
Online ISBN: 978-1-84628-663-6
eBook Packages: Computer ScienceComputer Science (R0)