Controlled Permutations for Testing Adaptive Classifiers
We study evaluation of online classifiers that are designed to adapt to changes in data distribution over time (concept drift). A standard procedure to evaluate such classifiers is the test-then-train, which iteratively uses the incoming instances for testing and then for updating a classifier. Comparing classifiers based on such a test risks to give biased results, since a dataset is processed only once in a fixed sequential order. Such a test concludes how well classifiers adapt when changes happen at fixed time points, while the ultimate goal is to assess how well they would adapt when changes of a similar type happen unexpectedly. To reduce the risk of biased evaluation we propose to run multiple tests with permuted data. A random permutation is not suitable, as it makes the data distribution uniform over time and destroys the adaptive learning problem. We develop three permutation techniques with theoretical control mechanisms that ensure that different distributions in data are preserved while perturbing the data order. The idea is to manipulate blocks of data keeping individual instances close together. Our permutations reduce the risk of biased evaluation by making it possible to analyze sensitivity of classifiers to variations in the data order.
Unable to display preview. Download preview PDF.
- 4.Bach, S., Maloof, M.: A bayesian approach to concept drift. In: Advances in Neural Information Processing Systems 23 (NIPS), pp. 127–135 (2010)Google Scholar
- 5.Baena-Garcia, M., del Campo-Avila, J., Fidalgo, R., Bifet, A., Gavalda, R., Morales-Bueno, R.: Early drift detection method. In: Proc. of ECML/PKDD Workshop on Knowledge Discovery from Data Streams (2006)Google Scholar
- 6.Bifet, A., Holmes, G., Kirkby, R., Pfahringer, B.: MOA: Massive Online Analysis. Journal of Machine Learning Research 11, 1601–1604 (2010)Google Scholar
- 8.Bifet, A., Holmes, G., Pfahringer, B., Kirkby, R., Gavalda, R.: New ensemble methods for evolving data streams. In: Proc. of the 15th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD 2009), pp. 139–148 (2009)Google Scholar
- 10.Diaconis, P.: Group representations in probability and statistics. Lecture Notes–Monograph Series, vol. 11. Hayward Inst. of Mathematical Statistics (1988)Google Scholar
- 14.Gama, J., Sebastiao, R., Rodrigues, P.P.: Issues in evaluation of stream learning algorithms. In: Proc. of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2009), pp. 329–338 (2009)Google Scholar
- 15.Harries, M.: Splice-2 comparative evaluation: Electricity pricing. Technical report, The University of South Wales (1999)Google Scholar
- 16.Ikonomovska, E., Gama, J., Dzeroski, S.: Learning model trees from evolving data streams. Data (2010)Google Scholar
- 17.Masud, M., Chen, Q., Khan, L., Aggarwal, C., Gao, J., Han, J., Thuraisingham, B.: Addressing concept-evolution in concept-drifting data streams. In: Proc. of the 10th IEEE Int. Conf. on Data Mining, ICDM 2010 (2010)Google Scholar
- 25.Zliobaite, I.: Controlled permutations for testing adaptive classifiers. Technical report (2011), https://sites.google.com/site/zliobaite/permutations