Investigation of Expert Addition Criteria for Dynamically Changing Online Ensemble Classifiers with Multiple Adaptive Mechanisms

  • Rashid Bakirov
  • Bogdan Gabrys
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 412)

Abstract

We consider online classification problem, where concepts may change over time. A prominent model for creation of dynamically changing online ensemble is used in Dynamic Weighted Majority (DWM) method. We analyse this model, and address its high sensitivity to misclassifications resulting in creation of unnecessary large ensembles, particularly while running on noisy data. We propose and evaluate various criteria for adding new experts to an ensemble. We test our algorithms on a comprehensive selection of synthetic data and establish that they lead to the significant reduction in the number of created experts and show slightly better accuracy rates than original models and non-ensemble adaptive models used for benchmarking.

References

  1. 1.
    Bach, S.H., Maloof, M.A.: Paired Learners for Concept Drift. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 23–32 (December 2008)Google Scholar
  2. 2.
    Baena-García, M., del Campo-Ávila, J., Fidalgo, R., Bifet, A., Gavaldà, R., Morales-Bueno, R.: Early drift detection method. In: ECML PKDD 2006 Fourth International Workshop on Knowledge Discovery from Data Streams, Berlin, Germany (2006)Google Scholar
  3. 3.
    Dietterich, T.G.: Ensemble Methods in Machine Learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  4. 4.
    Duin, R.P.W., Juszczak, P., Paclik, P., Pekalska, E., de Ridder, D., Tax, D.M.J., Verzakov, S.: PRTools4.1, A Matlab Toolbox for Pattern Recognition (2007), http://prtools.org
  5. 5.
    Elwell, R., Polikar, R.: Incremental learning of concept drift in nonstationary environments. IEEE Transactions on Neural Networks/A Publication of the IEEE Neural Networks Council 22(10), 1517–1531 (2011)CrossRefGoogle Scholar
  6. 6.
    Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Bazzan, A.L.C., Labidi, S. (eds.) SBIA 2004. LNCS (LNAI), vol. 3171, pp. 286–295. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  7. 7.
    Hazan, E., Seshadhri, C.: Efficient learning algorithms for changing environments. In: ICML 2009 Proceedings of the 26th Annual International Conference on Machine Learning, pp. 393–400 (2009)Google Scholar
  8. 8.
    Jacobs, A., Shalizi, C.R., Clauset, A.: Adapting to Non-stationarity with Growing Expert Ensembles. Tech. rep., Carnegie Mellon University (2010), http://www.santafe.edu/media/cms_page_media/285/AJacobsREUpaper.pdf
  9. 9.
    Kadlec, P., Gabrys, B.: Local learning-based adaptive soft sensor for catalyst activation prediction. AIChE Journal 57(5), 1288–1301 (2011)CrossRefGoogle Scholar
  10. 10.
    Kolter, J.Z., Maloof, M.A.: Dynamic weighted majority: An ensemble method for drifting concepts. The Journal of Machine Learning Research 8, 2755–2790 (2007)MATHGoogle Scholar
  11. 11.
    Kolter, J., Maloof, M.: Using additive expert ensembles to cope with concept drift. In: Proceedings of the 22nd International Conference on Machine Learning, ICML 2005, No. 1990, pp. 449–456 (2005)Google Scholar
  12. 12.
    Littlestone, N., Warmuth, M.: The Weighted Majority Algorithm. Information and Computation 108(2), 212–261 (1994)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Minku, L.L., Yao, X.: DDD: A New Ensemble Approach for Dealing with Concept Drift. IEEE Transactions on Knowledge and Data Engineering 24(4), 619–633 (2012)CrossRefGoogle Scholar
  14. 14.
    Narasimhamurthy, A., Kuncheva, L.: A framework for generating data to simulate changing environments. In: Proceedings of the 25th IASTED International Multi-Conference: Artificial Intelligence and Applications, pp. 384–389. ACTA Press, Anaheim (2007)Google Scholar
  15. 15.
    Ruta, D., Gabrys, B.: Classifier selection for majority voting. Information Fusion 6(1), 63–81 (2005)CrossRefGoogle Scholar
  16. 16.
    Schlimmer, J.C., Granger, R.H.: Incremental Learning from Noisy Data. Machine Learning 1(3), 317–354 (1986)Google Scholar
  17. 17.
    Shapley, L., Grofman, B.: Optimizing group judgmental accuracy in the presence of interdependencies. Public Choice 43(3), 329–343 (1984)CrossRefGoogle Scholar
  18. 18.
    Stanley, K.O.: Learning concept drift with a committee of decision trees. Technical report, UT-AI-TR-03-302, Department of Computer Science, University of Texas in Austin (2003)Google Scholar
  19. 19.
    Vovk, V.G.: Aggregating strategies. In: COLT 1990 Proceedings of the Third Annual Workshop on Computational Learning Theory, pp. 371–386. Morgan Kaufmann Publishers Inc., San Francisco (1990)Google Scholar
  20. 20.
    Žliobaitė, I., Kuncheva, L.I.: Theoretical Window Size for Classification in the Presence of Sudden Concept Drift. Tech. rep., CS-TR-001-2010, Bangor University, UK (2010)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2013

Authors and Affiliations

  • Rashid Bakirov
    • 1
  • Bogdan Gabrys
    • 1
  1. 1.Smart Technology Research CentreBournemouth UniversityUnited Kingdom

Personalised recommendations