Date: 24 Oct 2012
On evaluating stream learning algorithms
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Most streaming decision models evolve continuously over time, run in resource-aware environments, and detect and react to changes in the environment generating data. One important issue, not yet convincingly addressed, is the design of experimental work to evaluate and compare decision models that evolve over time. This paper proposes a general framework for assessing predictive stream learning algorithms. We defend the use of prequential error with forgetting mechanisms to provide reliable error estimators. We prove that, in stationary data and for consistent learning algorithms, the holdout estimator, the prequential error and the prequential error estimated over a sliding window or using fading factors, all converge to the Bayes error. The use of prequential error with forgetting mechanisms reveals to be advantageous in assessing performance and in comparing stream learning algorithms. It is also worthwhile to use the proposed methods for hypothesis testing and for change detection. In a set of experiments in drift scenarios, we evaluate the ability of a standard change detection algorithm to detect change using three prequential error estimators. These experiments point out that the use of forgetting mechanisms (sliding windows or fading factors) are required for fast and efficient change detection. In comparison to sliding windows, fading factors are faster and memoryless, both important requirements for streaming applications. Overall, this paper is a contribution to a discussion on best practice for performance assessment when learning is a continuous process, and the decision models are dynamic and evolve over time.
Editor: Bernhard Pfahringer.
Asuncion, A., & Newman, D. (2007). UCI machine learning repository. http://www.ics.uci.edu/~mlearn/MLRepository.html.
Babcock, B., Datar, M., Motwani, R., & O’Callaghan, L. (2003). Maintaining variance and k-medians over data stream windows. In T. Milo (Ed.), Proceedings of the 22nd symposium on principles of database systems, San Diego, USA (pp. 234–243). New York: ACM.
Bach, S. H., & Maloof, M. A. (2008). Paired learners for concept drift. In ICDM (pp. 23–32). Los Alamitos: IEEE Comput. Soc.
Basseville, M., & Nikiforov, I. (1993). Detection of abrupt changes: theory and applications. New York: Prentice Hall
Bifet, A., & Gavaldà, R. (2007). Learning from time-changing data with adaptive windowing. In Proceedings SIAM international conference on data mining, Minneapolis, USA (pp. 443–448). Philadelphia: SIAM.
Bifet, A., Holmes, G., Kirkby, R., & Pfahringer, B. (2010a). MOA: massive online analysis. Journal of Machine Learning Research, 11, 1601–1604.
Bifet, A., Holmes, G., Pfahringer, B., & Frank, E. (2010b). Fast perceptron decision tree learning from evolving data streams. In Advances in knowledge discovery and data mining, 14th Pacific-Asia conference (pp. 299–310). CrossRef
Bishop, C. (1995). Neural networks for pattern recognition. London: Oxford University Press.
Chi, Y., Wang, H., Yu, P. S., & Muntz, R. R. (2006). Catch the moment: maintaining closed frequent itemsets over a data stream sliding window. Knowledge and Information Systems, 10(3), 265–294. CrossRef
Cormode, G., Muthukrishnan, S., & Zhuang, W. (2007). Conquering the divide: continuous clustering of distributed data streams. In ICDE: proceedings of the international conference on data engineering, Istanbul, Turkey (pp. 1036–1045).
Dietterich, T. (1996). Approximate statistical tests for comparing supervised classification learning algorithms. Corvallis, technical report nr. 97.331, Oregon State University.
Domingos, P., & Hulten, G. (2000). Mining high-speed data streams. In I. Parsa, R. Ramakrishnan, & S. Stolfo (Eds.), Proceedings of the ACM sixth international conference on knowledge discovery and data mining, Boston, USA (pp. 71–80). New York: ACM. CrossRef
Duda, R., & Hart, P. (1973). Pattern classification and scene analysis. New York: Willey. MATH
Ferrer-Troyano, F., Aguilar-Ruiz, J. S., & Riquelme, J. C. (2004). Discovering decision rules from numerical data streams. In Proceedings of the ACM symposium on applied computing, Nicosia, Cyprus (pp. 649–653). New York: ACM Press.
Gama, J., & Kosina, P. (2011). Learning decision rules from data streams. In Proceedings of the 22nd international joint conference on artificial intelligence, IJCAI (pp. 1255–1260).
Gama, J., Medas, P., Castillo, G., & Rodrigues, P. (2004). Learning with drift detection. In A. L. C. Bazzan & S. Labidi (Eds.), Lecture notes in computer science: Vol. 3171. Advances in artificial intelligence—SBIA 2004, São Luis, Brasil (pp. 286–295). Berlin: Springer. CrossRef
Gama, J., Rocha, R., & Medas, P. (2003). Accurate decision trees for mining high-speed data streams. In Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, Washington, DC, USA (pp. 523–528). New York: ACM.
Gama, J., Sebastião, R., & Rodrigues, P. P. (2009). Issues in evaluation of stream learning algorithms. In Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, Paris, France (pp. 329–338). New York: ACM. CrossRef
Ghosh, B., & Sen, P. (1991). Handbook of sequential analysis. New York: Dekker. MATH
Giannella, C., Han, J., Pei, J., Yan, X., & Yu, P. (2003). Mining frequent patterns in data streams at multiple time granularities. In H. Kargupta, A. Joshi, K. Sivakumar, & Y. Yesha (Eds.), Next generation data mining. Menlo Park/Cambridge: AAAI Press/MIT Press.
Hartl, C., Baskiotis, N., Gelly, S., & Sebag, M. (2007). Change point detection and meta-bandits for online learning in dynamic environments. In Conférence Francophone sur l’apprentissage automatique, Cepadues (pp. 237–250).
Hulten, G., & Domingos, P. (2001). Catching up with the data: research issues in mining data streams. In Proc. of workshop on research issues in data mining and knowledge discovery, Santa Barbara, USA.
Hulten, G., & Domingos, P. (2003). VFML—a toolkit for mining high-speed time-changing data streams. Technical report, University of Washington. http://www.cs.washington.edu/dm/vfml/
Hulten, G., Spencer, L., & Domingos, P. (2001). Mining time-changing data streams. In Proceedings of the 7th ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, California (pp. 97–106). New York: ACM.
Japkowicz, N. & Shah, M. (Eds.) (2011). Evaluating learning algorithms: a classification perspective. Cambridge: Cambridge University Press. MATH
Katakis, I., Tsoumakas, G., & Vlahavas, I. (2010). Tracking recurring contexts using ensemble classifiers: an application to email filtering. Knowledge and Information Systems, 22, 371–391. CrossRef
Kearns, M., & Vazirani, U. (1994). An introduction to computational learning theory. Cambridge: MIT Press.
Kifer, D., Ben-David, S., & Gehrke, J. (2004). Detecting change in data streams. In Proceedings of the international conference on very large data bases, Toronto, Canada (pp. 180–191). San Mateo: Morgan Kaufmann.
Kirkby, R. (2008). Improving Hoeffding trees. Ph.D. thesis, University of Waikato, New Zealand.
Klinkenberg, R. (2004). Learning drifting concepts: example selection vs. example weighting. Intelligent Data Analysis, 8(3), 281–300.
Kolter, J. Z., & Maloof, M. A. (2007). Dynamic weighted majority: an ensemble method for drifting concepts. Journal of Machine Learning Research, 8, 2755–2790. MATH
Koychev, I. (2000). Gradual forgetting for adaptation to concept drift. In Proceedings of ECAI workshop current issues in spatio-temporal reasoning, Berlin, Germany (pp. 101–106). Leipzig: ECAI Press.
Kuh, A., Petsche, T., & Rivest, R. (1990). Learning time-varying concepts. In Proceedings advances in neural information processing (pp. 183–189). San Mateo: Morgan Kaufmann.
Li, P., Wu, X., & Hu, X. (2010). Mining recurring concept drifts with limited labeled streaming data. Journal of Machine Learning Research—Proceedings Track, 13, 241–252.
Liang, C., Zhang, Y., & Song, Q. (2010). Decision tree for dynamic and uncertain data streams. Journal of Machine Learning Research—Proceedings Track, 13, 209–224.
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006). Yale: rapid prototyping for complex data mining tasks. In ACM SIGKDD int. conf. on knowledge discovery and data mining (pp. 935–940). New York: ACM Press. CrossRef
Mitchell, T. (1997). Machine learning. New York: McGraw-Hill MATH
Mouss, H., Mouss, D., Mouss, N., & Sefouhi, L. (2004). Test of Page-Hinkley, an approach for fault detection in an agro-alimentary production system. In Proceedings of the Asian control conference (Vol. 2, pp. 815–818).
Rodrigues, P. P., Gama, J., & Pedroso, J. P. (2008). Hierarchical clustering of time series data streams. IEEE Transactions on Knowledge and Data Engineering, 20(5), 615–627. CrossRef
Street, W. N., & Kim, Y. (2001). A streaming ensemble algorithm SEA for large-scale classification. In Proceedings 7th ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, California (pp. 377–382). New York: ACM Press.
Widmer, G., & Kubat, M. (1996). Learning in the presence of concept drift and hidden contexts. Machine Learning, 23, 69–101.
- On evaluating stream learning algorithms
Volume 90, Issue 3 , pp 317-346
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- Data streams
- Evaluation design
- Prequential analysis
- Concept drift
- Industry Sectors
- Author Affiliations
- 1. LIAAD – INESC TEC and Faculty of Economics, University of Porto, Rua de Ceuta, 118-6, 4050-190, Porto, Portugal
- 2. LIAAD – INESC TEC and Faculty of Science, University of Porto, Rua de Ceuta, 118-6, 4050-190, Porto, Portugal
- 3. LIAAD – INESC TEC and Faculty of Medicine, University of Porto, Rua de Ceuta, 118-6, 4050-190, Porto, Portugal