Skip to main content
SpringerLink
Log in
Book cover

Learning in Non-Stationary Environments pp 153–184Cite as

Incremental Classifier Fusion and Its Applications in Industrial Monitoring and Diagnostics

  • Chapter
  • First Online:

  • 1040 Accesses

Abstract

Pattern recognition techniques have shown their usefulness for monitoring and diagnosing many industrial applications. The increasing production rates and the growing databases generated by these applications require learning techniques that can adapt their models incrementally, without revisiting previously used data. Ensembles of classifiers have been shown to improve the predictive accuracy as well as the robustness of classification systems. In this work, several well-known classifier fusion methods (Fuzzy Integral, Decision Templates, Dempster–Shafer Combination, and Discounted Dempster–Shafer Combination) are extended to allow incremental adaptation. Additionally, an incremental classifier fusion method using an evolving clustering approach is introduced—named Incremental Direct Cluster-based ensemble. A framework for strict incremental learning is proposed in which the ensemble and its member classifiers are adapted concurrently. The proposed incremental classifier fusion methods are evaluated within this framework for two industrial applications: online visual quality inspection of CD imprints and prediction of maintenance actions for copiers from a large historical database.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Bey-Temsamani, A., Engels, M., Motten, A., Vandenplas, S., Ompusunggu, A.: A practical approach to combine data mining and prognostics for improved predictive maintenance. In: Proceedings of the Data Mining Case Studies Workshop, 15th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 37–44 (2009)

    Google Scholar 

  2. Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996)

    MathSciNet  MATH  Google Scholar 

  3. Breiman, L.: Pasting small votes for classification in large databases and on-line. Mach. Learn. 36(1–2), 85–103 (1999)

    Article  Google Scholar 

  4. Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth International Group, Belmont, CA, USA (1984)

    MATH  Google Scholar 

  5. Cho, S., Kim, J.: Combining multiple neural networks by fuzzy integral for robust classification. IEEE Trans. Syst. Man Cybern. 25(2), 380–384 (1995)

    Article  Google Scholar 

  6. Crawford, S.: Extensions to the CART algorithm. Int. J. Man Mach. Stud. 31(2), 197–217 (1989)

    Article  MathSciNet  Google Scholar 

  7. Dempster, A.: A generalization of Bayesian inference. J. Roy. Stat. Soc. B 30, 205–247 (1968)

    MathSciNet  Google Scholar 

  8. Diehl, C., Cauwenberghs, G.: SVM incremental learning, adaptation and optimization. In: Proceedings of the 2003 International Joint Conference on Neural Networks, pp. 2685–2690 (2003)

    Google Scholar 

  9. Dietterich, T.: Ensemble methods in machine learning. In: J. Kittler, F. Roli (eds.) Multiple Classifier Systems, Lecture Notes in Computer Science, vol. 1857, pp. 1–15. Springer, Berlin/Heidelberg, Germany (2000)

    Google Scholar 

  10. Duda, R., Hart, P.: Pattern Classification and Scene Analysis. John Wiley & Sons, New York (1973)

    MATH  Google Scholar 

  11. Duda, R., Hart, P., Stork, D.: Pattern Classification, 2nd edn. John Wiley & Sons, New York, USA (2000)

    Google Scholar 

  12. Eitzinger, C., Gmainer, M., Heidl, W., Lughofer, E.: Increasing classification robustness with adaptive features. In: A. Gasteratos, M. Vincze, J. Tsotsos (eds.) Computer Vision Systems, Lecture Notes in Computer Science, vol. 5008, pp. 445–453. Springer, Berlin/Heidelberg (2008)

    Chapter  Google Scholar 

  13. Feng, J.: An intelligent decision support system based on machine learning and dynamic track of psychological evaluation criterion. In: J. Kacpzryk (ed.) Intelligent Decision and Policy Making Support Systems, pp. 141–157. Springer, Berlin/Heidelberg (2008)

    Chapter  Google Scholar 

  14. Filev, D., Tseng, F.: Novelty detection based machine health prognostics. In: Proceedings of the 2006 International Symposium on Evolving Fuzzy Systems, pp. 193–199 (2006)

    Google Scholar 

  15. Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. Int. 55(1), 119–139 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  16. Fung, G., Mangasarian, O.: Incremental support vector machine classification. In: Second SIAM International Conference on Data Mining, pp. 247–260 (2002)

    Google Scholar 

  17. Gader, P., Mohamed, M., Keller, J.: Fusion of handwritten word classifiers. Pattern Recognit. Lett. 17(6), 577–584 (1996)

    Article  Google Scholar 

  18. Gama, J.: Knowledge Discovery from Data Streams. Chapman & Hall / CRC, Boca Raton, Florida, USA (2010)

    Book  MATH  Google Scholar 

  19. Gray, R.M.: Vector quantization. IEEE ASSP Mag. 1(2), 4–29 (1984)

    Article  Google Scholar 

  20. Hansen, L., Salamon, P.: Neural network ensembles. IEEE Trans. Pattern Anal. Mach. 12(10), 993–1001 (1990)

    Article  Google Scholar 

  21. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Verlag, New York, USA (2001)

    MATH  Google Scholar 

  22. Huang, Y., Suen, C.: A method of combining multiple experts for the recognition of unconstrained handwritten numerals. IEEE Trans. Pattern Anal. Mach. 17(1), 90–94 (1995)

    Article  Google Scholar 

  23. Jacobs, R., Jordan, M., Nowlan, S., Hinton, G.: Adaptive mixtures of local experts. Neural Comput. 3, 79–87 (1991)

    Article  Google Scholar 

  24. Kerdprasop, N., Kerdprasop, K.: Data partitioning for incremental data mining. In: Proceedings of the First International Forum on Information and Computer Science, pp. 114–118. Hamamatsu, Japan (2003)

    Google Scholar 

  25. Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Trans. Pattern Anal. Mach. 20(3), 226–239 (1998)

    Article  Google Scholar 

  26. Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In: G. Tesauro, D. Touretzky, T. Leen (eds.) Proceedings of the 7th Conference on Advances in Neural Information Processing Systems, vol. 7, pp. 231–238. MIT Press, Cambridge, MA, USA (1995)

    Google Scholar 

  27. Kuncheva, L.: Using measures of similarity and inclusion for multiple classifier fusion by decision templates. Fuzzy Sets Syst. 122(3), 401–407 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  28. Kuncheva, L.: A theoretical study on six classifier fusion strategies. IEEE Trans. Pattern Anal. Mach. 24(2), 281–286 (2002)

    Article  Google Scholar 

  29. Kuncheva, L.: Classifier ensembles for changing environments. In: F. Roli, J. Kittler, T. Windeatt (eds.) Multiple Classifier Systems, Lecture Notes in Computer Science, vol. 3077, pp. 1–15. Springer, Berlin/Heidelberg, Germany (2004)

    Google Scholar 

  30. Kuncheva, L.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, New Jersey (2004)

    Book  MATH  Google Scholar 

  31. Kuncheva, L., Bezdek, J., Duin, R.: Decision templates for multiple classifier fusion: An experimental comparison. Pattern Recognit. 34(2), 299–314 (2001)

    Article  MATH  Google Scholar 

  32. Kuncheva, L., Whitaker, C., Shipp, C., Duin, R.: Limits on the majority vote accuracy in classifier fusion. Pattern Anal. Appl. 6(1), 22–31 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  33. Li, W., Yue, H., Valle-Cervantes, S., Qin, S.: Recursive PCA for adaptive process monitoring. J. Process Control 10(5), 471–486 (2000)

    Article  Google Scholar 

  34. Lim, C., Harrison, R.: Online pattern classification with multiple neural network systems: An experimental study. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 33, 235–247 (2003)

    Google Scholar 

  35. Lughofer, E.: Extensions of vector quantization for incremental clustering. Pattern Recognition, 41(3), 995–1011 (2008)

    Article  MATH  Google Scholar 

  36. Lughofer, E.: Hybrid active learning for reducing the annotation efforts of operators in classification systems. Pattern Recognition, 45(2), 884–896 (2012)

    Article  Google Scholar 

  37. Lughofer, E.: Evolving Fuzzy Systems: Methodologies, Advanced Concepts and Applications. Springer, Berlin/Heidelberg (2011)

    Book  MATH  Google Scholar 

  38. Lughofer, E., Guardiola, C.: On-line fault detection with data-driven evolving fuzzy models. J. Contr. Intell. Syst. 36(4), 307–317 (2008)

    MATH  Google Scholar 

  39. Lughofer, E., Smith, J., Caleb-Solly, P., Tahir, M., Eitzinger, C., Sannen, D., Nuttin, M.: Human-machine interaction issues in quality control based on online image classification. IEEE Trans. Syst. Man Cybern. Part A Syst. Humans 39(5), 960–971 (2009)

    Article  Google Scholar 

  40. Nisbet, R., Elder, J., Miner, G.: Handbook of Statistical Analysis and Data Mining Applications. Academic Press, Burlington, MA, USA (2009)

    MATH  Google Scholar 

  41. Oza, N.: Online ensemble learning. Ph.D. thesis, University of California, USA (2001)

    Google Scholar 

  42. Pang, S., Ozawa, S., Kasabov, N.: Incremental linear discriminant analysis for classification of data streams. IEEE Trans. Syst. Man Cybern. Part B Cybern. 35(5), 905–914 (2005)

    Article  Google Scholar 

  43. Polikar, R., Upda, L., Upda, S., Honavar, V.: Learn++: An incremental learning algorithm for supervised neural networks. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 31(4), 497–508 (2001)

    Article  Google Scholar 

  44. Polikar, R.: Ensemble based systems in decision making. IEEE Circuits Syst. Mag. 6(3), 21–45 (2006)

    Article  Google Scholar 

  45. Quinlan, J.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA, USA (1993)

    Google Scholar 

  46. Rogova, G.: Combining the results of several neural network classifiers. Neural Networks 7(5), 777–781 (1994)

    Article  Google Scholar 

  47. Roubos, J., Setnes, M., Abonyi, J.: Learning fuzzy classification rules from data. Inform. Sciences 150, 77–93 (2003)

    Article  MathSciNet  Google Scholar 

  48. Rüping, S.: Incremental learning with support vector machines. Ph.D. thesis, Universität Dortmund (2002)

    Google Scholar 

  49. Ruta, D., Gabrys, B.: An overview of classifier fusion methods. Comput. and Inform. Syst. J. 7(1), 1–10 (2000)

    Google Scholar 

  50. Sannen, D.: A flexible framework for learning visual quality inspection: Information fusion approaches. Ph.D. thesis, Katholieke Universiteit Leuven, Belgium (2010)

    Google Scholar 

  51. Sannen, D., Nuttin, M., Smith, J., Tahir, M., Caleb-Solly, P., Lughofer, E., Eitzinger, C.: An on-line interactive self-adaptive image classification framework. In: A. Gasteratos, M. Vincze, J. Tsotsos (eds.) Computer Vision Systems, Lecture Notes in Computer Science, vol. 5008, pp. 171–180. Springer, Berlin/Heidelberg, Germany (2008)

    Google Scholar 

  52. Sannen, D., Van Brussel, H., Nuttin, M.: Classifier fusion using discounted Dempster–Shafer combination. In: P. Perner (ed.) Machine Learning and Data Mining in Pattern Recognition, Poster Proceedings, pp. 216–230. IBaI Publishing, Leipzig, Germany (2007)

    Google Scholar 

  53. Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, Princeton, NJ, USA (1976)

    MATH  Google Scholar 

  54. Song, M., Wang, H.: Incremental estimation of gaussian mixture models for online data stream clustering. In: Proceedings of International Conference on Bioinformatics and its Applications (2004)

    Google Scholar 

  55. Stone, M.: Cross-validatory choice and assessment of statistical predictions (with discussion). J. Roy. Stat. Soc. B 36, 111–147 (1974)

    MATH  Google Scholar 

  56. Sutton, R., Whitehead, S.: Online learning with random representations. In: Proceedings of the 10th International Conference on Machine Learning, pp. 314–321. Morgan Kaufmann, San Mateo, CA, USA (1993)

    Google Scholar 

  57. Theodoridis, S., Koutroumbas, K.: Pattern Recognition, 4th edn. Academic Press, Burlington, MA, USA (2009)

    Google Scholar 

  58. Utgoff, P.: Incremental induction of decision trees. Mach. Learn. 4, 161–186 (1989)

    Article  Google Scholar 

  59. Valentini, G., Masulli, F.: Ensembles of learning machines. In: M. Marinaro, R. Tagliaferri (eds.) Neural Nets: 13th Italian Workshop on Neural Nets, Vietri sul Mare, Italy, 2002, Revised Papers, Lecture Notes in Computer Science, vol. 2486, pp. 3–22. Springer, Berlin/Heidelberg, Germany (2002)

    Google Scholar 

  60. Vapnik, V., Golowich, S., Smola, A.: Support vector method for function approximation, regression estimation and signal processing. Adv. Neural Inf. Process. Syst. 9, 281–287 (1997)

    Google Scholar 

  61. Wang, H., Fan, W., Yu, P., Han, J.: Mining concept-drifting data streams using ensemble classifiers. In: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 226–235. ACM, New York, NY, USA (2003)

    Google Scholar 

  62. Wasserman, P.: Advanced Methods in Neural Computing. Van Nostrand Reinhold, New York, USA (1993)

    MATH  Google Scholar 

  63. Widmer, G., Kubat, M.: Learning in the presence of concept drift and hidden contexts. Mach. Learn. 23(1), 69–101 (1996)

    Google Scholar 

  64. Wolpert, D., Macready, W.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)

    Article  Google Scholar 

  65. Woods, K., Kegelmeyer, W., Bowyer, K.: Combination of multiple classifiers using local accuracy estimates. IEEE Trans. Pattern Anal. Mach. 19(4), 405–410 (1997)

    Article  Google Scholar 

  66. Xu, L., Krzyźak, A., Suen, C.: Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Trans. Syst. Man Cybern. 22(3), 418–435 (1992)

    Article  Google Scholar 

Download references

Acknowledgments

This work was partly supported by the European Commission (project Contract No. STRP016429, acronym DynaVis) and partly carried out within the framework of the technological advice project td-diagmon (grant no. 070522), which is financially supported by the Institute for the Promotion of Innovation through Science and Technology in Flanders (IWT-Vlaanderen). This publication reflects only the authors’ views.

Author information

Authors and Affiliations

  1. Flanders Mechatronics Technology Centre, Celestijnenlaan 300B, B-3001, Leuven, Belgium

    Davy Sannen, Jean-Michel Papy & Steve Vandenplas

  2. Department of Knowledge-Based Mathematical Systems, Johannes Kepler University Linz, Altenbergerstrasse 69, A-4040, Linz, Austria

    Edwin Lughofer

  3. Department of Mechanical Engineering, Katholieke Universiteit Leuven, Celestijnenlaan 300B, B-3001, Leuven, Belgium

    Hendrik Van Brussel

Authors
  1. Davy Sannen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jean-Michel Papy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Steve Vandenplas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Edwin Lughofer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hendrik Van Brussel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Davy Sannen .

Editor information

Editors and Affiliations

  1. , Départment Informatique et Automatique, Ecole des Mines de Douai, 941, Rue Charles Bourseul, Douai cedex, 59508, France

    Moamar Sayed-Mouchaweh

  2. University of Linz, Weissdornweg 16, Linz, 4232, Austria

    Edwin Lughofer

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer Science+Business Media New York

About this chapter

Cite this chapter

Sannen, D., Papy, JM., Vandenplas, S., Lughofer, E., Van Brussel, H. (2012). Incremental Classifier Fusion and Its Applications in Industrial Monitoring and Diagnostics. In: Sayed-Mouchaweh, M., Lughofer, E. (eds) Learning in Non-Stationary Environments. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-8020-5_7

Download citation

  • DOI: https://doi.org/10.1007/978-1-4419-8020-5_7

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4419-8019-9

  • Online ISBN: 978-1-4419-8020-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation