Applying Bagging Techniques to the SA Tabu Miner Rule Induction Algorithm

Conference paper


This paper presents an implementation of bagging techniques over the heuristic algorithm for induction of classification rules called SA Tabu Miner (Simulated Annealing and Tabu Search data miner). The goal was to achieve better predictive accuracy of the derived classification rules. Bagging (Bootstrap aggregating) is an ensemble method that has attracted a lot of attention, both experimentally, since it behaves well on noisy datasets, and theoretically, because of its simplicity. In this paper we present the experimental results of various bagging versions of the SA Tabu Miner algorithm. The SA Tabu Miner algorithm is inspired by both research on heuristic optimization algorithms and rule induction data mining concepts and principles. Several bootstrap methodologies were applied to SA Tabu Miner, including reducing repetition of instances, forcing repetition of instances not to exceed two, using different percentages of the original basic training set. Various experimental approaches and parameters yielded different results on the compared datasets.


Bagging Bootstrap Simulated Annealing SA Tabu Miner Tabu Search Data Mining Rule Induction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Chorbev, I., Mihajlov, D., Jolevski, I.: Web Based Medical Expert System with a Self Training Heuristic Rule Induction Algorithm. In: Proc. of The First International Conference on Advances in Databases, Knowledge, and Data Applications, DBKDA 2009, Cancun, Mexico, March 2009, pp. 143–148 (2009)Google Scholar
  2. 2.
    Dietterich, T.G.: Machine Learning Research: Four Current Directions. AI Magazine 18(4), 97–136 (1997)Google Scholar
  3. 3.
    Dietterich, T.G.: Ensemble Methods in Machine Learning, Oregon State University, Corvallis, Oregon, USA, WWW home page,
  4. 4.
    Gentle, J.E., Härdle, W., Mori, Y.: Handbook of Computational Statistics, ch. 16. Springer, Heidelberg,
  5. 5.
    Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996)zbMATHGoogle Scholar
  6. 6.
    Stefanowski, J.: Bagging and Introduction of Decision Rules. In: Klopotek, M., et al (eds.) Intelligent information systems (2002)Google Scholar
  7. 7.
    Freund, Y., Schapire, R.: Experiments with a new boosting algorithm. In: Proc. of the Thirteenth Int. Conf. on Machine Learning, pp. 148–156 (1996)Google Scholar
  8. 8.
    Pino-Mejías, R., et al.: Bagging Classification Models with Reduced Bootstrap. Structural, Syntactic, and Statistical Pattern Recognition, 966–973 (2004),
  9. 9.
    Grandvalet, Y.: Bagging equalizes influence. Machine Learning 55(3), 251–270 (2004)CrossRefzbMATHGoogle Scholar
  10. 10.
    Friedman, J.: Stochastic gradient boosting. Computational Statistics and Data Analysis 38(4), 367–378 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Parmanto, B., Munro, P., Doyle, H.: Improving Committee Diagnosis with Resampling Techniques. In: Touretzky, D., Mozer, M., Hasselmo, M. (eds.) Advances in Neural Information Processing Systems, vol. 8, pp. 882–888 (1996)Google Scholar
  12. 12.
    Quinlan, J.R.: Bagging, boosting and C4.5. In: Proceedings of the 13th National Conference on Artifitial Intelligence, pp. 725–730 (1996)Google Scholar
  13. 13.
    Bühlmann, P., Yu, B.: Explaining Bagging’. Technical Report 92, Seminar für Statistik, ETH, Zürich (2000)Google Scholar
  14. 14.
    Buja, A., Stuetzle, W.: The Effect of Bagging on Variance, Bias and Mean Squared Error. Technical report, AT&T Labs-Research (2000)Google Scholar
  15. 15.
    Friedman, J.H., Hall, P.: On Bagging and Non-linear Estimation. Technical report, Stanford University, Stanford, CA (2000)Google Scholar
  16. 16.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993)Google Scholar
  17. 17.
  18. 18.
    Buja, A., Stuetzle, W.: Observation of Bagging. Statistica Sinica 16, 323–351 (2006)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Hall, P., Samworth, R.J.: Properties of Bagged Nearest-neighbor Classifiers. J. Roy. Statist. Soc., Ser. B 67, 363–379 (2005)CrossRefzbMATHGoogle Scholar
  20. 20.
  21. 21.
    Weiss, S.M.: Small Sample Error Rate Estimation for k-neares Neighbor Classifiers. IEEE Transaction of pattern analysis and Machine Intelligent 13(3), 285–289 (1991)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Brown, G., Wyatt, J., Harris, R., Yao, X.: Diversity Creation Methods: A Survey and Categorisation. Information Fusion 6(1), 5–20 (2005)CrossRefGoogle Scholar
  23. 23.
    Hansen, L., Salamon, P.: Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 993–1001 (1990)CrossRefGoogle Scholar
  24. 24.
    Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In: Tesauro, G., Touretzky, D., Leen, T. (eds.) Advances in Neural Information Processing Systems, vol. 7, pp. 231–238. MIT Press, Cambridge (1995)Google Scholar
  25. 25.
    Hashem, S.: Optimal linear combinations of neural networks. Neural Networks 10(4), 599–614 (1997)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Opitz, D., Shavlik, J.: Actively searching for an effective neural-network ensemble. Connection Science 8(3/4), 337–353 (1996a)CrossRefGoogle Scholar
  27. 27.
    Opitz, D., Shavlik, J.: Generating accurate and diverse members of a neural-network ensemble. In: Touretsky, D., Mozer, M., Hasselmo, M. (eds.) Advances in Neural Information Processing Systems, vol. 8, pp. 535–541. MIT Press, Cambridge (1996b)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  1. 1.Faculty of Electrical Engineering and Information TechnologiesSkopjeRepublic of Macedonia
  2. 2.Netcetera. ul. Partizanski Odredi 72aSkopjeRepublic of Macedonia

Personalised recommendations