Neural Computing and Applications

, Volume 21, Issue 4, pp 671–682 | Cite as

Adaptive cascade of boosted ensembles for face detection in concept drift

  • Teo Susnjak
  • Andre L. C. Barczak
  • Ken A. Hawick


We propose an adaptive learning algorithm for cascades of boosted ensembles that is designed to handle the problem of concept drift in nonstationary environments. The goal was to create a real-time adaptive algorithm for dynamic environments that exhibit varying degrees of drift in high-volume streaming data. This we achieved using a hybrid of detect-and-retrain and constant-update approaches. The uniqueness of our method is found in two aspects of our framework. The first is the manner in which individual weak classifiers within each cascade layer of an ensemble are clustered during training and assigned a competence value. Secondly, the idea of learning optimal cascade-layer thresholds during runtime, which enables rapid adaptation to dynamic environments. The proposed adaptive learning method was applied to a binary-class problem with rare-event detection characteristics. For this, we chose the domain of face detection and demonstrate experimentally the ability of our algorithm to achieve an effective trade-off between accuracy and speed of adaptations in dense data streams with unknown rates of change.


Ensemble-based learning Adaptive learning Concept drift Nonstationary environments Boosting Cascades AdaBoost Face detection 


  1. 1.
    Muhlbaier M, Polikar R (2007) Multiple classifiers based incremental learning algorithm for learning in nonstationary environments. In: 2007 international conference on machine learning and cybernetics, vol 6. pp 3618–3623Google Scholar
  2. 2.
    Wang P, Wang H, Wu X, Wang W, Shi B (2007) A low-granularity classifier for data streams with concept drifts and biased class distribution. IEEE Trans Knowl Data Eng 19:1202–1213Google Scholar
  3. 3.
    May M, Berendt B, Cornuejols A, Gama J, Giannotti F, Hotho A, Malerba D, Menasalvas E, Morik K, Pedersen R et al (2008) Research challenges in ubiquitous knowledge discovery. Chapman & Hall/CRC Press, LondonGoogle Scholar
  4. 4.
    May M, Saitta L (2010) Introduction: the challenge of ubiquitous knowledge discovery. In: May M, Saitta L (eds) Ubiquitous knowledge discovery. Volume 6202 of lecture notes in computer science. Springer, Berlin, pp 3–18CrossRefGoogle Scholar
  5. 5.
    Widmer G, Kubat M (1996) Learning in the presence of concept drift and hidden contexts. Mach Learn 23:69–101Google Scholar
  6. 6.
    Schlimmer J, Granger R (1986) Incremental learning from noisy data. Mach Learn 1(3):317–354Google Scholar
  7. 7.
    Kuncheva LI (2004) Classifier ensembles for changing environments. In: Roli F, Kittler J, Windeatt T (eds) Multiple classifier systems. Volume 3077 of lecture notes in computer science. Springer, Berlin, pp 1–15Google Scholar
  8. 8.
    Tsymbal A (2004) The problem of concept drift: definitions and related work. TCD-CS-2004-15, vol 4. Department of Computer Science, Trinity College, DublinGoogle Scholar
  9. 9.
    Nishida K, Yamauchi K, Omori T (2005) Ace: adaptive classifiers-ensemble system for concept-drifting environments. Mult Classif Syst 3541:176–185Google Scholar
  10. 10.
    Japkowicz N (2000) The class imbalance problem: significance and strategies. In: Proceedings of the 2000 international conference on artificial intelligence (ICAI 2000), vol 1. pp 111–117Google Scholar
  11. 11.
    Zhu X, Zhang P, Lin X, Shi Y (2010) Active learning from stream data using optimal weight classifier ensemble. IEEE Trans Syst Man Cybern B Cybern 40:1607–1621Google Scholar
  12. 12.
    Procopio M, Mulligan J, Grudic G (2009) Learning terrain segmentation with classifier ensembles for autonomous robot navigation in unstructured environments. J Field Robot 26(2):145–175CrossRefGoogle Scholar
  13. 13.
    Street W, Kim Y (2001) A streaming ensemble algorithm (sea) for large-scale classification. In: Proceedings of the seventh ACM SIGKDD international conference on knowledge discovery and data mining. KDD ’01. ACM, New York, pp 377–382Google Scholar
  14. 14.
    Wang H, Fan W, Yu P, Han J (2003) Mining concept-drifting data streams using ensemble classifiers. In: Proceedings of the ninth ACM SIGKDD international conference on knowledge discovery and data mining. KDD ’03. ACM, New York, pp 226–235Google Scholar
  15. 15.
    Scholz M, Klinkenberg R (2007) Boosting classifiers for drifting concepts. Intell Data Anal 11(1):3–28Google Scholar
  16. 16.
    Elwell R, Polikar R (2009) Incremental learning of variable rate concept drift. Mult Classif Syst 5519:142–151Google Scholar
  17. 17.
    Rodriguez J, Kuncheva L (2010) Combining online classification approaches for changing environments. Struct Syntactic Stat Pattern Recognit 5342:520–529Google Scholar
  18. 18.
    Karnick M, Muhlbaier M, Polikar R (2008) Incremental learning in non-stationary environments with concept drift using a multiple classifier based approach. In: 19th international conference on pattern recognition, 2008 (ICPR 2008). pp 1–4Google Scholar
  19. 19.
    Kolter J, Maloof M (2003) Dynamic weighted majority: a new ensemble method for tracking concept drift. In: Third IEEE international conference on data mining, 2003 (ICDM 2003). pp 123–130Google Scholar
  20. 20.
    Kolter J, Maloof M (2007) Dynamic weighted majority: an ensemble method for drifting concepts. J Mach Learn Res 8:2755–2790MATHGoogle Scholar
  21. 21.
    Pocock A, Yiapanis P, Singer J, Luján M, Brown G (2010) Online non-stationary boosting. Mult Classif Syst 5997:205–214Google Scholar
  22. 22.
    Oza N (2001) Online ensemble learning. PhD thesis, University of California, BerkeleyGoogle Scholar
  23. 23.
    Huang C, Ai H, Yamashita T, Lao S, Kawade M (2007) Incremental learning of boosted face detector. In: IEEE 11th international conference on computer vision, 2007 (ICCV 2007). pp 1–8Google Scholar
  24. 24.
    Pelossof R, Jones M, Vovsha I, Rudin C (2009) Online coordinate boosting. In: IEEE 12th international conference on computer vision workshops (ICCV workshops), 2009. pp 1354–1361Google Scholar
  25. 25.
    Grabner H, Leistner C, Bischof H (2008) Semi-supervised on-line boosting for robust tracking. In: Forsyth D, Torr P, Zisserman A (eds) Computer vision—ECCV 2008. Volume 5302 of lecture notes in computer science. Springer, Berlin, pp 234–247Google Scholar
  26. 26.
    Bifet A, Holmes G, Pfahringer B, Kirkby R, Gavaldà R (2009) New ensemble methods for evolving data streams. In: Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining. KDD ’09. ACM, New York, pp 139–148Google Scholar
  27. 27.
    Barczak ALC, Johnson MJ, Messom CH (2008) Empirical evaluation of a new structure for adaboost. In: SAC ’08: proceedings of the 2008 ACM symposium on applied computing. ACM, Fortaleza, pp 1764–1765Google Scholar
  28. 28.
    Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: CVPR01, Kauai, HI, IEEE (December), vol I. pp 511–518Google Scholar
  29. 29.
    Susnjak T, Barczak A, Hawick K (2010) A modular approach to training cascades of boosted ensembles. In: Structural, syntactic, and statistical pattern recognition. Volume 6218 of lecture notes in computer science. Springer, Berlin, pp 640–649Google Scholar
  30. 30.
    Freund Y, Schapire RE (1995) A decision-theoretic generalization of on-line learning and an application to boosting. In: EuroCOLT ’95: proceedings of the second European conference on computational learning theory. Springer, London, pp 23–37Google Scholar
  31. 31.
    Georghiades A, Belhumeur P, Kriegman D (2001) From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell 23(6):643–660CrossRefGoogle Scholar
  32. 32.
    Allwein EL, Schapire RE, Singer Y (2001) Reducing multiclass to binary: a unifying approach for margin classifiers. J Mach Learn Res 1:113–141MathSciNetMATHGoogle Scholar
  33. 33.
    Lorena AC, Carvalho AC, Gama JAM (2008) A review on the combination of binary classifiers in multiclass problems. Artif Intell Rev 30:19–37CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Teo Susnjak
    • 1
  • Andre L. C. Barczak
    • 1
  • Ken A. Hawick
    • 1
  1. 1.Massey UniversityAlbanyNew Zealand

Personalised recommendations