Active Grading Ensembles for Learning Visual Quality Control from Multiple Humans

  • Davy Sannen
  • Hendrik Van Brussel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5519)


When applying Machine Learning technology to real-world applications, such as visual quality inspection, several practical issues need to be taken care of. One problem is posed by the reality that usually there are multiple human operators doing the inspection, who will inevitable contradict each other occasionally. In this paper a framework is proposed which is able to deal with this issue, based on trained ensembles of classifiers. Most ensemble techniques have however difficulties learning in these circumstances. Therefore several novel enhancements to the Grading ensemble technique are proposed within this framework – called Active Grading. The Active Grading algorithm is evaluated on data obtained from a real-world industrial system for visual quality inspection of the printing of labels on CDs, which was labelled independently by four different human operators and their supervisor, and compared to the standard Grading algorithm and a range of other ensemble (classifier fusion) techniques.


Ensemble learning grading classifier fusion visual quality control learning from multiple humans 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Castillo, E., Alvarez, E.: Expert Systems: Uncertainty and Learning. Springer, New York (2007)zbMATHGoogle Scholar
  2. 2.
    Malamas, E., Petrakis, E., Zervakis, M., Petit, L., Legat, J.D.: A survey on industrial vision systems, applications and tools. Image and Vision Computing 21 (2003)Google Scholar
  3. 3.
    Juran, J., Gryna, F.: Juran’s Quality Control Handbook, 4th edn. McGraw-Hill, New York (1988)Google Scholar
  4. 4.
    Seewald, A., Fürnkranz, J.: An evaluation of grading classifiers. In: Hoffmann, F., Adams, N., Fisher, D., Guimarães, G., Hand, D.J. (eds.) IDA 2001. LNCS, vol. 2189, pp. 115–124. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  5. 5.
    Sannen, D., Nuttin, M., Smith, J., Tahir, M.A., Caleb-Solly, P., Lughofer, E., Eitzinger, C.: An on-line interactive self-adaptive image classification framework. In: Gasteratos, A., Vincze, M., Tsotsos, J. (eds.) ICVS 2008. LNCS, vol. 5008, pp. 171–180. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Govindaraju, M., Pennathur, A., Mital, A.: Quality improvement in manufacturing through human performance enhancement. Integrated Manufacturing Systems 12(5) (2001)Google Scholar
  7. 7.
    Duda, R., Hart, P., Stork, D.: Pattern Classification, 2nd edn. John Wiley & Sons, New York (2000)zbMATHGoogle Scholar
  8. 8.
    Kuncheva, L.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, Chichester (2004)CrossRefzbMATHGoogle Scholar
  9. 9.
    Polikar, R.: Ensemble based systems in decision making. IEEE Circuits and Systems Magazine 6(3), 21–45 (2006)CrossRefGoogle Scholar
  10. 10.
    Brown, G., Wyatt, J., Harris, R., Yao, X.: Diversity creation methods: A survey and categorisation. Information Fusion 6(1), 5–20 (2005)CrossRefGoogle Scholar
  11. 11.
    Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth International Group, Belmont (1984)zbMATHGoogle Scholar
  12. 12.
    Kuncheva, L., Whitaker, C., Shipp, C., Duin, R.: Limits on the majority vote accuracy in classifier fusion. Pattern Analysis & Applications 6(1), 22–31 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(3), 226–239 (1998)CrossRefGoogle Scholar
  14. 14.
    Cho, S., Kim, J.: Combining multiple neural networks by fuzzy integral for robust classification. IEEE Transactions on Systems, Man, and Cybernetics 25(2), 380–384 (1995)CrossRefGoogle Scholar
  15. 15.
    Kuncheva, L., Bezdek, J., Duin, R.: Decision templates for multiple classifier fusion: An experimental comparison. Pattern Recognition 34(2), 299–314 (2001)CrossRefzbMATHGoogle Scholar
  16. 16.
    Rogova, G.: Combining the results of several neural network classifiers. Neural Networks 7(5), 777–781 (1994)CrossRefGoogle Scholar
  17. 17.
    Sannen, D., Van Brussel, H., Nuttin, M.: Classifier fusion using Discounted Dempster-Shafer combination. In: Poster Proceedings of the 5th International Conference on Machine Learning and Data Mining, pp. 216–230 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Davy Sannen
    • 1
  • Hendrik Van Brussel
    • 1
  1. 1.Department of Mechanical EngineeringKatholieke Universiteit LeuvenHeverlee (Leuven)Belgium

Personalised recommendations