Advertisement

Lift-Per-Drift: An Evaluation Metric for Classification Frameworks with Concept Drift Detection

  • Robert Anderson
  • Yun Sing Koh
  • Gillian Dobbie
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11320)

Abstract

Data streams with concept drift change over time. Detecting drift allows remedial action, but this can come at a cost e.g. training a new classifier. Prequential accuracy is commonly used to evaluate the impact of drift detection frameworks on data stream classification, but recent work shows frequent periodic drift detection can provide better accuracy than state-of-the-art drift detection techniques. We discuss how sequentiality, the degree of consecutive matching class labels across instances, allows high accuracy without a classifier learning to differentiate classes. We propose a novel metric: lift-per-drift (lpd). This measures drift detection performance through its impact on classification accuracy, penalised by drifts detected in a dataset. This metric solves three problems: lpd cannot be increased by periodic, frequent drifts; lpd clearly shows when using drift detection increases classifier error; and lpd does not require knowledge of where real drifts occurred. We show how lpd can be set to be sensitive to the cost of each drift. Our experiments show lpd is not artificially increased through sequentiality; that lpd highlights when drift detection has caused a loss in accuracy; and that it is sensitive to change in true-positive drift and false-positive drift detection rates.

Keywords

Data streams Concept drift Evaluation Classification 

References

  1. 1.
    Bifet, A., de Francisci Morales, G., Read, J., Holmes, G., Pfahringer, B.: Efficient online evaluation of big data stream classifiers. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 59–68. ACM (2015)Google Scholar
  2. 2.
    Bifet, A.: Classifier concept drift detection and the illusion of progress. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10246, pp. 715–725. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-59060-8_64CrossRefGoogle Scholar
  3. 3.
    Hoens, T.R., Polikar, R., Chawla, N.V.: Learning from streaming data with concept drift and imbalance: an overview. Prog. Artif. Intell. 1(1), 89–101 (2012)CrossRefGoogle Scholar
  4. 4.
    Gama, J., Sebastião, R., Rodrigues, P.P.: Issues in evaluation of stream learning algorithms. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 329–338. ACM (2009)Google Scholar
  5. 5.
    Krempl, G., et al.: Open challenges for data stream mining research. ACM SIGKDD Explor. Newsl. 16(1), 1–10 (2014)CrossRefGoogle Scholar
  6. 6.
    Tsymbal, A.: The problem of concept drift: definitions and related work, vol. 106. Computer Science Department, Trinity College Dublin (2004)Google Scholar
  7. 7.
    Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Bazzan, A.L.C., Labidi, S. (eds.) SBIA 2004. LNCS (LNAI), vol. 3171, pp. 286–295. Springer, Heidelberg (2004).  https://doi.org/10.1007/978-3-540-28645-5_29CrossRefGoogle Scholar
  8. 8.
    Frías-Blanco, I., del Campo-Ávila, J., Ramos-Jiménez, G., Morales-Bueno, R., Ortiz-Díaz, A., Caballero-Mota, Y.: Online and non-parametric drift detection methods based on Hoeffdings bounds. IEEE TKDE 27(3), 810–823 (2015)Google Scholar
  9. 9.
    Žliobaitė, I., Budka, M., Stahl, F.: Towards cost-sensitive adaptation: when is it worth updating your predictive model? Neurocomputing 150, 240–249 (2015)CrossRefGoogle Scholar
  10. 10.
    Bifet, A., Frank, E.: Sentiment knowledge discovery in Twitter streaming data. In: Pfahringer, B., Holmes, G., Hoffmann, A. (eds.) DS 2010. LNCS (LNAI), vol. 6332, pp. 1–15. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-16184-1_1CrossRefGoogle Scholar
  11. 11.
    Žliobaitė, I., Bifet, A., Read, J., Pfahringer, B., Holmes, G.: Evaluation methods and decision theory for classification of streaming data with temporal dependence. Mach. Learn. 98(3), 455–482 (2015)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Robert Anderson
    • 1
  • Yun Sing Koh
    • 1
  • Gillian Dobbie
    • 1
  1. 1.Department of Computer ScienceUniversity of AucklandAucklandNew Zealand

Personalised recommendations