Incremental Boolean Combination of Classifiers

  • Wael Khreich
  • Eric Granger
  • Ali Miri
  • Robert Sabourin
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6713)


The incremental Boolean combination (incrBC) technique is a new learn-and-combine approach that is proposed to adapt ensemble-based pattern classification systems over time, in response to new data acquired during operations. When a new block of training data becomes available, this technique generates a diversified pool of base classifiers from the data by varying training hyperparameters and random initializations. The responses of these classifiers are then combined with those of previously-trained classifiers through Boolean combination in the ROC space. Through this process, an ensemble is selected from the pool, where Boolean fusion functions and thresholds are adapted for improved accuracy, while redundant base classifiers are pruned. Results of computer simulations conducted using Hidden Markov Models (HMMs) on synthetic and real-world host-based intrusion detection data indicate that incrBC can sustain a significantly higher level of accuracy than when the parameters of a single best HMM are re-estimated for each new block of data, using reference batch and incremental learning techniques. It also outperforms static fusion techniques such as majority voting for combining the responses of new and previously-generated pools of HMMs. Pruning prevents pool sizes from increasing indefinitely over time, without adversely affecting the overall ensemble performance.


Boolean Function Anomaly Detection Decision Threshold Incremental Learning Boolean Combination 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Connolly, J.F., Granger, E., Sabourin, R.: An adaptive classification system for video-based face recognition. Information Sciences (2010) (in Press)Google Scholar
  2. 2.
    Khreich, W., Granger, E., Miri, A., Sabourin, R.: A comparison of techniques for on-line incremental learning of HMM parameters in anomaly detection. In: Proc. 2nd IEEE Int’l Conf. on Computational Intelligence for Security and Defense Applications, Ottawa, Canada, July 2009, pp. 1–8 (2009)Google Scholar
  3. 3.
    Khreich, W., Granger, E., Miri, A., Sabourin, R.: Iterative Boolean combination of classifiers in the ROC space: An application to anomaly detection with HMMs. Pattern Recognition 43(8), 2732–2752 (2010)CrossRefzbMATHGoogle Scholar
  4. 4.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, Chichester (2004)CrossRefzbMATHGoogle Scholar
  5. 5.
    Mizuno, J., Watanabe, T., Ueki, K., Amano, K., Takimoto, E., Maruoka, A.: On-line estimation of hidden Markov model parameters. In: Proc. 3rd Int’l Conf. on Discovery Science, vol. 1967, pp. 155–169 (2000)Google Scholar
  6. 6.
    Polikar, R., Upda, L., Upda, S., Honavar, V.: Learn++: An incremental learning algorithm for supervised neural networks. IEEE Transactions on Systems, Man and Cybernetics, Part C 31(4), 497–508 (2001)CrossRefGoogle Scholar
  7. 7.
    Tan, K., Maxion, R.: Determining the operational limits of an anomaly-based intrusion detector. IEEE Journal on Selected Areas in Communications 21(1), 96–110 (2003)CrossRefGoogle Scholar
  8. 8.
    Tao, Q., Veldhuis, R.: Threshold-optimized decision-level fusion and its application to biometrics. Pattern Recognition 41(5), 852–867 (2008)Google Scholar
  9. 9.
    Tsoumakas, G., Partalas, I., Vlahavas, I.: An ensemble pruning primer. Applications of Supervised and Unsupervised Ensemble Methods 245, 1–13 (2009)CrossRefGoogle Scholar
  10. 10.
    Tulyakov, S., Jaeger, S., Govindaraju, V., Doermann, D.: Review of classifier combination methods. In: Marinai, S., H.F. (eds.) Studies in Comp. Intelligence: ML in Document Analysis and Recognition, pp. 361–386. Springer, Heidelberg (2008)Google Scholar
  11. 11.
    Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: Alternative data models. In: Proc. IEEE Computer Society Symposium on Research in Security and Privacy, Oakland, CA, pp. 133–145 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Wael Khreich
    • 1
  • Eric Granger
    • 1
  • Ali Miri
    • 2
  • Robert Sabourin
    • 1
  1. 1.Laboratoire d’imagerie, de vision et d’intelligence artificielleÉcole de technologie supérieureMontrealCanada
  2. 2.School of Computer ScienceRyerson UniversityTorontoCanada

Personalised recommendations