Summary
Although conceptually quite simple, decision trees are still among the most popular classifiers applied to real-world problems. Their popularity is due to a number of factors – core among these is their ease of comprehension, robust performance and fast data processing capabilities. Additionally feature selection is implicit within the decision tree structure.
This chapter introduces the basic ideas behind decision trees, focusing on decision trees which only consider a rule relating to a single feature at a node (therefore making recursive axis-parallel slices in feature space to form their classification boundaries). The use of particle swarm optimization (PSO) to train near optimal decision trees is discussed, and PSO is applied both in a single objective formulation (minimizing misclassification cost), and multi-objective formulation (trading off misclassification rates across classes).
Empirical results are presented on popular classification data sets from the well-known UCI machine learning repository, and PSO is demonstrated as being fully capable of acting as an optimizer for trees on these problems. Results additionally support the argument that multi-objectification of a problem can improve uni-objective search in classification problems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bishop, C.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, Heidelberg (2006)
Duda, R., Hart, P.: Pattern Classification and Scene Analysis, 2nd edn. Wiley, Chichester (2001)
Everson, R., Fieldsend, J.: Multi-objective optimization of safety related systems: An application to short term conflict alert. IEEE Transactions on Evolutionary Computation 10, 187–198 (2006)
Schetinin, V., Fieldsend, J., Partridge, D., Coats, T., Krzanowski, W., Everson, R., Bailey, T., Hernandez, A.: Confident interpretation of bayesian decision tree ensembles for clinical applications. IEEE Transactions on Information Technology in Biomedicine 11, 312–319 (2007)
Asuncion, A., Newman, D.: UCI machine learning repository (2007)
Ripley, B.: Neural networks and related methods for classification (with discussion). Journal of the Royal Statistical Society Series B 56, 409–456 (1994)
Brieman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Chapman & Hall/CRC (1984)
Quinlan, J.: Induction of decision trees. Machine Learning 1, 86–106 (1986)
Quinlan, J.: C4.5 Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993)
Kennedy, J., Eberhart, R.: Particle swarm optimization. In: IEEE International Conference on Neural Networks, Perth, Australia, pp. 1942–1948. IEEE Service Center (1995)
Kennedy, J., Eberhart, R.: A discrete binary version of the particle swarm algorithm. In: Proceedings of the IEEE Conference on Systems, Man and Cybernetics, pp. 4104–4109. IEEE Press, Los Alamitos (1997)
Kim, D.: Minimizing structural risk on decision tree classification. In: Jin, Y. (ed.) Multi-Objective Machine Learning. Studies in Computational Intelligence, vol. 16, pp. 241–260. Springer, Heidelberg (2006)
Denison, D., Holmes, C., Mallick, B., Smith, A.: Bayesian Methods for Nonlinear Classification and Regression. Probability and Statistics. Wiley, Chichester (2002)
Veenhuis, C., Köppen, M., Krüger, J., Nickolay, B.: Tree swarm optimization: An approach to pso-based tree discovery. In: 2005 IEEE Congress on Evolutionary Computation, vol. 2, pp. 1238–1245 (2005)
Everson, R., Fieldsend, J.: Multi-class roc analysis from a multi-objective optimisation perspective. Pattern Recognition Letters 27, 918–927 (2006)
Everson, R., Fieldsend, J.: Multi-objective optimisation for receiver operating characteristic analysis. In: Jin, Y. (ed.) Multi-Objective Machine Learning. Studies in Computational Intelligence, vol. 16, pp. 531–556. Springer, Heidelberg (2006)
Fieldsend, J.: Multi-objective particle swarm optimisation methods. Tech. Rep. 419, Department of Computer Science, University of Exeter (2004)
Coello Coello, C., Pulido, G., Lechuga, M.: Handling multiple objectives with particle swarm optimization. IEEE Transactions on Evolutionary Computation 8, 256–279 (2004)
Alvarez-Benitez, J., Everson, R., Fieldsend, J.: A mopso algorithm based exclusively on pareto dominance concepts. In: The Third International Conference on Evolutionary Mutli-Criterion Optimization, pp. 459–473 (2005)
Fieldsend, J., Everson, R., Singh, S.: Using unconstrained elite archives for multi-objective optimization. IEEE Transactions on Evolutionary Computation 7, 305–323 (2003)
Fieldsend, J., Singh, S.: A multi-objective algorithm based upon particle swarm optimisation, an efficient data structure and turbulence. In: 2002 UK Workshop on Computational Intelligence (UKCI 2002), Birmingham, UK, pp. 37–44 (2002)
Fieldsend, J., Singh, S.: Pareto evolutionary neural networks. IEEE Transactions on Neural Networks 16, 338–354 (2005)
Knowles, J., Watson, R., Corne, D.: Reducing local optima in single-objective problems by multi-objectivization. In: Zitzler, E., Deb, K., Thiele, L., Coello Coello, C.A., Corne, D.W. (eds.) EMO 2001. LNCS, vol. 1993, pp. 269–283. Springer, Heidelberg (2001)
Jensen, M.: Guiding single-objective optimization using multiobjective methods. In: Raidl, G.R., Cagnoni, S., Cardalda, J.J.R., Corne, D.W., Gottlieb, J., Guillot, A., Hart, E., Johnson, C.G., Marchiori, E., Meyer, J.-A., Middendorf, M. (eds.) EvoIASP 2003, EvoWorkshops 2003, EvoSTIM 2003, EvoROB/EvoRobot 2003, EvoCOP 2003, EvoBIO 2003, and EvoMUSART 2003. LNCS, vol. 2611, pp. 268–279. Springer, Heidelberg (2003)
Abbass, H., Deb, K.: Searching under multi-evolutionary pressures. In: Fonseca, C.M., Fleming, P.J., Zitzler, E., Deb, K., Thiele, L. (eds.) EMO 2003. LNCS, vol. 2632, pp. 391–404. Springer, Heidelberg (2003)
Mostaghim, S., Teich, J.: Strategies for finding good local guides in multi-objective particle swarm optimization (mopso). In: IEEE 2003 Swarm Intelligence Symposium (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Fieldsend, J.E. (2009). Optimizing Decision Trees Using Multi-objective Particle Swarm Optimization. In: Coello, C.A.C., Dehuri, S., Ghosh, S. (eds) Swarm Intelligence for Multi-objective Problems in Data Mining. Studies in Computational Intelligence, vol 242. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03625-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-03625-5_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03624-8
Online ISBN: 978-3-642-03625-5
eBook Packages: EngineeringEngineering (R0)