Machine Vision and Applications

, Volume 27, Issue 1, pp 103–127 | Cite as

On improving performance of surface inspection systems by online active learning and flexible classifier updates

  • Eva Weigl
  • Wolfgang Heidl
  • Edwin Lughofer
  • Thomas Radauer
  • Christian Eitzinger
Original Paper


Classification of detected events is a central component in state-of-the-art surface inspection systems that still relies on manual parametrization. While machine-learned classifiers promise supreme accuracy, their reliability depends on complete and correct annotation of an extensive training database, leaving the risk of unpredictable behavior in changing production environments. We propose an active learning-based training framework, which selectively presents questionable events for user annotation and is capable of online operation. Evaluation results on two data streams from microfluidic chips and elevator sheaves production show that annotation effort can be reduced by 90 % with negligible loss of accuracy. Simulation runs introducing new event classes show that the online active learning procedure is both efficient in terms of learning speed and robust in maintaining the accuracy levels of existing classes. The results underline the feasibility and potential of our approach that significantly reduces the required effort for inspection system setup and adapts to changes in the production process.


Visual quality inspection Image-based classification of event types Dynamic classifier updates Active learning 



This work was funded via the project ‘Improving the usability of machine learning in industrial inspection systems’ (useML) by the Austrian Research Promotion Agency (FFG) under the scope of the ‘Information and communication technology of the future’ program. This program is promoted by the Austrian Federal Ministry of Transport, Innovation and Technology (BMVIT) and the Federal Ministry of Science, Research and Economy (BMWFW).


  1. 1.
    Angelov, P., Lughofer, E., Zhou, X.: Evolving fuzzy classifiers using different model architectures. Fuzzy Sets Syst. 159(23), 3160–3182 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Belohlavek, R.: Fuzzy Relational Systems. Springer, Heidelberg, Berlin (2002)CrossRefzbMATHGoogle Scholar
  3. 3.
    Bifet, A., Holmes, G., Kirkby, R., Pfahringer, B.: Data stream mining—a practical approach. Technical report, Department of Computer Sciences, University of Waikato, Japan (2011)Google Scholar
  4. 4.
    Bifet, A., Kirkby, R.: Data stream mining—a practical approach. Technical report, Department of Computer Sciences, University of Waikato, Japan (2011)Google Scholar
  5. 5.
    Bordes, A., Ertekin, S., Weston, J., Bottou, L.: Fast kernel classifiers with online and active learning. J. Mach. Learn. Res. 6, 1579–1619 (2005)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Bouchachia, A.: Incremental induction of classification fuzzy rules. In: IEEE Workshop on Evolving and Self-Developing Intelligent Systems (ESDIS) 2009, Nashville, USA, pp. 32–39 (2009)Google Scholar
  7. 7.
    Brazdil, P., Giraud-Carrier, C., Soares, C., Vilalta, R.: Metalearning. Springer, Berlin, Heidelberg (2009)zbMATHGoogle Scholar
  8. 8.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefzbMATHGoogle Scholar
  9. 9.
    Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. Chapman and Hall, Boca Raton (1993)Google Scholar
  10. 10.
    Brinkmann, R.: The Art and Science of Digital Compositing. Morgan Kaufmann, San Francisco (1999)Google Scholar
  11. 11.
    Caleb-Solly, P., Smith, J.E.: Adaptive surface inspection via interactive evolution. Image Vis. Comput. 25(7), 1058–1072 (2007)CrossRefGoogle Scholar
  12. 12.
    Chapelle, O., Schoelkopf, B., Zien, A.: Semi-Supervised Learning. MIT Press, Cambridge (2006)CrossRefGoogle Scholar
  13. 13.
    Cheng, J., Wang, K.: Active learning for image retrieval with co-svm. Pattern Recognit. 40, 330–334 (2006)CrossRefGoogle Scholar
  14. 14.
    Cheng, Y., Chen, Z., Liu, L., Wang, J., Agrawal, A., Choudhary, A.: Feedback-driven multiclass active learning for data streams. In: Proceedings of the 22nd ACM International Conference on Information and Knowledge management, CIKM ’13, pp. 1311–1320. ACM, New York (2013)Google Scholar
  15. 15.
    Chu, W., Zinkevich, M., Li, L., Thomas, A., Zheng, B.: Unbiased online active learning in data streams. In: Proceedings of the KDD 2011, San Diego, California (2011)Google Scholar
  16. 16.
    Dean, J.: Big Data, Data Mining, and Machine Learning: Value Creation for Business Leaders and Practitioners. Wiley, Hoboken (2012)Google Scholar
  17. 17.
    Demant, C., Streicher-Abel, B., Waszkewitz, P.: Industrial Image Processing: Visual Quality Control in Manufacturing. Springer, Berlin, Heidelberg (1999)CrossRefGoogle Scholar
  18. 18.
    Donmez, P., Carbonell, J.G.: From active to proactive learning methods. In: Kornoacki, J., Ras, Z.W., Wierzchon, S.T., Kacprzyk, J. (eds.) Advances in Machine Learning Part I. Studies in Computational Intelligence, vol. 262, pp. 97–120. Springer, Berlin, Heidelberg (2010)Google Scholar
  19. 19.
    Donmez, P., Carbonell, J.G., Bennett, P.N.: Dual strategy active learning. In: Kornoacki, J., Lopez de Mantaras, R., Matwin, S., Mladenic, D., Skowron, A. (eds.) Proceedings of the ECML 2007 Conference. LNCS (LNAI), vol. 4701, pp. 116–127. Springer, Heidelberg (2007)Google Scholar
  20. 20.
    Efron, B., Tibshirani, R.: An Introduction to the Bootstrap. Chapman and Hall/CRC, Boca Raton (1993)CrossRefzbMATHGoogle Scholar
  21. 21.
    Efron, B., Tibshirani, R.: Improvements on cross-validation: the 632 + bootstrap method. J. Am. Stat. Assoc. 92(438), 548–560 (1997)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Eitzinger, C., Heidl, W., Lughofer, E., Raiser, S., Smith, J.E., Tahir, M.A., Sannen, D., Van Brussel, H.: Assessment of the influence of adaptive components in trainable surface inspection systems. Mach. Vis. Appl. 21(5), 613–626 (2009)CrossRefGoogle Scholar
  23. 23.
    Fürnkranz, J.: Round robin classification. J. Mach. Learn. Res. 2, 721–747 (2002)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. J. Mach. Learn. Res. 3, 1157–1182 (2003)zbMATHGoogle Scholar
  25. 25.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference and Prediction, 2nd edn. Springer, New York, Berlin, Heidelberg (2009)CrossRefGoogle Scholar
  26. 26.
    Heidl, W., Thumfart, S., Lughofer, E., Eitzinger, C., Klement, E.-P.: Machine learning based analysis of gender differences in visual inspection decision making. Inf. Sci. 224, 62–76 (2013)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Hu, W., Hu, W., Xi, N., Maybank, S.: Unsupervised active learning based on hierarchical graph-theoretic clustering. IEEE Trans. Syst. Man Cybern. Part B: Cybern. 39(5), 1147–1161 (2009)CrossRefGoogle Scholar
  28. 28.
    Hua, G., Long, C., Yang, M., Gao, Y.: Collaborative active learning of a kernel machine ensemble for recognition. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1209–1216, Dec (2013)Google Scholar
  29. 29.
    Hühn, J., Hüllermeier, E.: FR3: a fuzzy rule learner for inducing reliable classifiers. IEEE Trans. Fuzzy Syst. 17(1), 138–149 (2009)CrossRefGoogle Scholar
  30. 30.
    Hüllermeier, E., Brinker, Klaus: Learning valued preference structures for solving classification problems. Fuzzy Sets Syst. 159(18), 2337–2352 (2008)CrossRefzbMATHGoogle Scholar
  31. 31.
    Joshi, A.J., Porikli, F., Papanikolopoulos, N.: Multi-class active learning for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2372–2379. IEEE Press (2009)Google Scholar
  32. 32.
    Kim, C.W., Koivo, A.J.: Hierarchical classification of surface defects on dusty wood boards. Pattern Recognit. Lett. 15(7), 713–721 (1994)CrossRefGoogle Scholar
  33. 33.
    Krishnamoorthy, K., Mathew, T.: Statistical Tolerance Regions: Theory, Applications, and Computation. Wiley, Hoboken (2009)CrossRefGoogle Scholar
  34. 34.
    Kuncheva, L.: Fuzzy Classifier Design. Physica-Verlag, Heidelberg (2000)CrossRefzbMATHGoogle Scholar
  35. 35.
    Kuncheva, L.: Combining Pattern Classifiers: Methods and Algorithms. Wiley-Interscience (Wiley), Southern Gate, Chichester (2004)Google Scholar
  36. 36.
    Lewis, D.D., Catlett, J.: Heterogeneous uncertainty sampling for supervised learning. In: Proceedings of the 11th International Conference on Machine Learning, New Brunswick, New Jersey, pp. 148–156 (1994)Google Scholar
  37. 37.
    Li, X., Wang, L., Sung, E.: Multilabel SVM active learning for image classification. In: Proceedings of the International Conference on Image Processing (ICIP), Singapore, vol. 4, pp. 2207–2010 (2004)Google Scholar
  38. 38.
    Long, C., Hua, G., Kapoor, A.: Active visual recognition with expertise estimation in crowdsourcing. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 3000–3007, Dec (2013)Google Scholar
  39. 39.
    Loy, C.C., Hospedales,T.M., Xiang, T., Gong, S.: Stream-based joint exploration-exploitation active learning. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1560–1567, June (2012)Google Scholar
  40. 40.
    Lughofer, E.: Extensions of vector quantization for incremental clustering. Pattern Recognit. 41(3), 995–1011 (2008)CrossRefzbMATHGoogle Scholar
  41. 41.
    Lughofer, E.: On-line evolving image classifiers and their application to surface inspection. Image Vis. Comput. 28(7), 1065–1079 (2010)CrossRefGoogle Scholar
  42. 42.
    Lughofer, E.: On-line assurance of interpretability criteria in evolving fuzzy systems—achievements, new concepts and open issues. Inf. Sci. 251, 22–46 (2013)CrossRefGoogle Scholar
  43. 43.
    Lughofer, E., Buchtala, O.: Reliable all-pairs evolving fuzzy classifiers. IEEE Trans. Fuzzy Syst. 21(4), 625–641 (2013)CrossRefGoogle Scholar
  44. 44.
    Lughofer, E., Eitzinger, C., Guardiola, C.: On-line quality control with flexible evolving fuzzy systems. In: Sayed-Mouchaweh, M., Lughofer, E. (eds.) Learning in Non-Stationary Environments: Methods and Applications, pp. 375–406. Springer, New York (2012)CrossRefGoogle Scholar
  45. 45.
    Lughofer, E., Smith, J.E., Tahir, M.A., Caleb-Solly, P., Eitzinger, C., Sannen, D., Nuttin, M.: Human–machine interaction issues in quality control based on on-line image classification. IEEE Trans. Syst. Man Cybern. Part A: Syst. Hum. 39(5), 960–971 (2009)CrossRefGoogle Scholar
  46. 46.
    Nauck, D., Kruse, R.: NEFCLASS-X—a soft computing tool to build readable fuzzy classifiers. BT Technol. J. 16(3), 180–190 (1998)CrossRefGoogle Scholar
  47. 47.
    Pratama, M., Anavatti, S.G., Angelov, P., Lughofer, E.: PANFIS: a novel incremental learning machine. IEEE Trans. Neural Netw. Learn. Syst. 25(1), 55–68 (2014)CrossRefGoogle Scholar
  48. 48.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Francisco (1993)Google Scholar
  49. 49.
    Raghavan, H., Madani, O., Jones, R.: Active learning with feedback on both features and instances. J. Mach. Learn. Res. 7, 1655–1686 (2006)MathSciNetzbMATHGoogle Scholar
  50. 50.
    Raiser, S., Lughofer, E., Eitzinger, C., Smith, J.E.: Impact of object extraction methods on classification performance in surface inspection systems. Mach. Vis. Appl. 21(5), 627–641 (2010)CrossRefGoogle Scholar
  51. 51.
    Ravikumara, S., Ramachandran, K.I., Sugumaran, V.: Machine learning approach for automated visual inspection of machine components. Expert Syst. Appl. 38(4), 3260–3266 (2011)CrossRefGoogle Scholar
  52. 52.
    Rodrigues, F., Pereira, F., Ribeiro, B.: Gaussian process classification and active learning with multiple annotators. In: Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 433–441 (2014)Google Scholar
  53. 53.
    Saffari, A., Leistner, C., Santner, J., Godec, M., Bischof, H.: On-line random forests. In: 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1393–1400 (2009)Google Scholar
  54. 54.
    Sannen, D., van Brussel, H.: A multilevel information fusion approach for visual quality inspection. Inf. Fusion 13(1), 48–59 (2012)CrossRefGoogle Scholar
  55. 55.
    Schoonahd, J.W., Gould, J.D., Miller, L.A.: Studies of visual inspection. Ergonomics 16(4), 365–379 (1973)CrossRefGoogle Scholar
  56. 56.
    Sculley, D.: Online active learning methods for fast label efficient spam filtering. In: Proceedings of the Fourth Conference on Email and AntiSpam, Mountain View, California (2007)Google Scholar
  57. 57.
    Settles, B.: Active Learning. Morgan & Claypool Publishers, San Rafael (2012)zbMATHGoogle Scholar
  58. 58.
    Stone, M.: Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. 36(1), 111–147 (1974)zbMATHGoogle Scholar
  59. 59.
    Tong, S., Koller, D.: Support vector machine active learning with application to text classification. J. Mach. Learn. Res. 2, 45–66 (2001)Google Scholar
  60. 60.
    Wu, X., Kumar, V.: The Top 10 Algorithms in Data Mining. Chapman & Hall, Boca Raton (2009)CrossRefGoogle Scholar
  61. 61.
    Yang, Z., Tang, J., Zhang, Y.: Active learning for streaming networked data. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM ’14, pp. 1129–1138. ACM, New York (2014)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Eva Weigl
    • 1
  • Wolfgang Heidl
    • 1
  • Edwin Lughofer
    • 2
  • Thomas Radauer
    • 3
  • Christian Eitzinger
    • 1
  1. 1.Machine Vision GroupProfactor GmbHSteyr-GleinkAustria
  2. 2.Department of Knowledge-Based Mathematical SystemsJohannes Kepler University LinzLinzAustria
  3. 3.Sony DADC Austria AGAnifAustria

Personalised recommendations