Incremental Learning of Variable Rate Concept Drift

  • Ryan Elwell
  • Robi Polikar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5519)


We have recently introduced an incremental learning algorithm, Learn + + .NSE, for Non-Stationary Environments, where the data distribution changes over time due to concept drift. Learn + + .NSE is an ensemble of classifiers approach, training a new classifier on each consecutive batch of data that become available, and combining them through an age-adjusted dynamic error based weighted majority voting. Prior work has shown the algorithm’s ability to track gradually changing environments as well as its ability to retain former knowledge in cases of cyclical or recurring data by retaining and appropriately weighting all classifiers generated thus far. In this contribution, we extend the analysis of the algorithm to more challenging environments experiencing varying drift rates; but more importantly we present preliminary results on the ability of the algorithm to accommodate addition or subtraction of classes over time. Furthermore, we also present comparative results of a variation of the algorithm that employs an error-based pruning in cyclical environments.


nonstationary environment concept drift Learn + + .NSE 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Schlimmer, J.C., Granger, R.H.: Incremental Learning from Noisy Data. Machine Learning 1(3), 317–354 (1986)Google Scholar
  2. 2.
    Widmer, G., Kubat, M.: Learning in the presence of concept drift and hidden contexts. Machine Learning 23(1), 69–101 (1996)Google Scholar
  3. 3.
    Tsymbal, A.: Technical Report: The problem of concept drift: definitions and related work, Trinity College, Dublin, Ireland,TCD-CS-2004-15 (2004)Google Scholar
  4. 4.
    Kuncheva, L.I.: Classifier Ensembles for Changing Environments. In: Roli, F., Kittler, J., Windeatt, T. (eds.) MCS 2004. LNCS, vol. 3077, pp. 1–15. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  5. 5.
    Kuncheva, L.I.: Classifier ensembles for detecting concept change in streaming data: Overview and perspectives. In: European Conference on Artificial Intelligence (ECAI), pp. 5–10 (2008)Google Scholar
  6. 6.
    Rodriguez, J.J., Kuncheva, L.I.: Combining Online Classification Approaches for-Changing Environments. In: International Workshops on Structural and Syntactic Pattern Recognition and Statistical Techniques in Pattern Recognition S+SSPR (2008)Google Scholar
  7. 7.
    Alippi, C., Roveri, M.: Just-in-Time Adaptive Classifiers; Part I: Detecting Nonsta-tionary Changes. IEEE Transactions on Neural Networks 19(7), 1145–1153 (2008)CrossRefGoogle Scholar
  8. 8.
    Da Silva, B.C., Basso, E.W., Bazzan, A.L.C., Engel, P.M.: Dealing with non-stationary environments using context detection. In: 23rd International Conference on Machine Learning - ICML 2006, vol. 2006, pp. 217–224 (2006)Google Scholar
  9. 9.
    Oommen, B.J., Rueda, L.: Stochastic learning-based weak estimation of multinomial random variables and its applications to pattern recognition in non-stationary environments. Pattern Recognition 39(3), 328–341 (2006)CrossRefzbMATHGoogle Scholar
  10. 10.
    Kolter, J.Z., Maloof, M.A.: Dynamic weighted majority: an ensemble method for drifting concepts. Journal of Machine Learning Research 8, 2755–2790 (2007)zbMATHGoogle Scholar
  11. 11.
    Scholz, M., Klinkenberg, R.: Boosting Classifiers for Drifting Concepts. Intelligent Data Analysis, Special Issue on Knowledge Discovery from Data Streams 11(1), 3–28 (2007)Google Scholar
  12. 12.
    Nishida, K., Yamauchi, K.: Adaptive Classifiers-Ensemble System for Tracking Concept Drift. In: 2007 International Conference on Machine Learning and Cybernetics, vol. 6, pp. 3607–3612 (2007)Google Scholar
  13. 13.
    Wang, H., Fan, W., Yu, P., Han, J.: Mining concept-drifting data streams using en-semble classifiers. In: Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 226–235 (2003)Google Scholar
  14. 14.
    Street, W.N., Kim, Y.: A streaming ensemble algorithm (SEA) for large-scale classification. In: Seventh ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2001), pp. 377–382 (2001)Google Scholar
  15. 15.
    Karnick, M., Muhlbaier, M.D., Polikar, R.: Incremental Learning in Non-Stationary Environments with Concept Drift Using a Multiple Classifier Based Approach. In: International Conference on Pattern Recognition (ICPR 2008), pp. 1–4 (2008)Google Scholar
  16. 16.
    Karnick, M., Ahiskali, M., Muhlbaier, M.D., Polikar, R.: Learning concept drift in nonstationary environments using an ensemble of classifiers based approach. In: World Congress on Computational Intelligence, International Joint Conference on Neural Networks, pp. 3455–3462 (2008)Google Scholar
  17. 17.
    Muhlbaier, M., Polikar, R.: An Ensemble Approach for Incremental Learning in Nonstationery Environments. In: Haindl, M., Kittler, J., Roli, F. (eds.) MCS 2007. LNCS, vol. 4472, pp. 490–500. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  18. 18.
    Freund, Y., Schapire, R.E.: Decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55(1), 119–139 (1997)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Ryan Elwell
    • 1
  • Robi Polikar
    • 1
  1. 1.Signal Processing and Pattern Recognition Laboratory Electrical and Computer EngineeringRowan UniversityGlassboroUSA

Personalised recommendations