Lift-Per-Drift: An Evaluation Metric for Classification Frameworks with Concept Drift Detection
Data streams with concept drift change over time. Detecting drift allows remedial action, but this can come at a cost e.g. training a new classifier. Prequential accuracy is commonly used to evaluate the impact of drift detection frameworks on data stream classification, but recent work shows frequent periodic drift detection can provide better accuracy than state-of-the-art drift detection techniques. We discuss how sequentiality, the degree of consecutive matching class labels across instances, allows high accuracy without a classifier learning to differentiate classes. We propose a novel metric: lift-per-drift (lpd). This measures drift detection performance through its impact on classification accuracy, penalised by drifts detected in a dataset. This metric solves three problems: lpd cannot be increased by periodic, frequent drifts; lpd clearly shows when using drift detection increases classifier error; and lpd does not require knowledge of where real drifts occurred. We show how lpd can be set to be sensitive to the cost of each drift. Our experiments show lpd is not artificially increased through sequentiality; that lpd highlights when drift detection has caused a loss in accuracy; and that it is sensitive to change in true-positive drift and false-positive drift detection rates.
KeywordsData streams Concept drift Evaluation Classification
- 1.Bifet, A., de Francisci Morales, G., Read, J., Holmes, G., Pfahringer, B.: Efficient online evaluation of big data stream classifiers. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 59–68. ACM (2015)Google Scholar
- 2.Bifet, A.: Classifier concept drift detection and the illusion of progress. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10246, pp. 715–725. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59060-8_64CrossRefGoogle Scholar
- 4.Gama, J., Sebastião, R., Rodrigues, P.P.: Issues in evaluation of stream learning algorithms. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 329–338. ACM (2009)Google Scholar
- 6.Tsymbal, A.: The problem of concept drift: definitions and related work, vol. 106. Computer Science Department, Trinity College Dublin (2004)Google Scholar
- 8.Frías-Blanco, I., del Campo-Ávila, J., Ramos-Jiménez, G., Morales-Bueno, R., Ortiz-Díaz, A., Caballero-Mota, Y.: Online and non-parametric drift detection methods based on Hoeffdings bounds. IEEE TKDE 27(3), 810–823 (2015)Google Scholar