Minimal Correlation Classification

  • Noga Levy
  • Lior Wolf
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7577)


When the description of the visual data is rich and consists of many features, a classification based on a single model can often be enhanced using an ensemble of models. We suggest a new ensemble learning method that encourages the base classifiers to learn different aspects of the data. Initially, a binary classification algorithm such as Support Vector Machine is applied and its confidence values on the training set are considered. Following the idea that ensemble methods work best when the classification errors of the base classifiers are not related, we serially learn additional classifiers whose output confidences on the training examples are minimally correlated. Finally, these uncorrelated classifiers are assembled using the GentleBoost algorithm. Presented experiments in various visual recognition domains demonstrate the effectiveness of the method.


Covariance Cyan Vanilla 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. PAMI 28(12), 2037–2041 (2006)CrossRefGoogle Scholar
  3. 3.
    Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996)MathSciNetMATHGoogle Scholar
  4. 4.
    Candes, E., Demanet, L., Donoho, D., Ying, L.: Fast discrete curvelet transforms. In: Multiscale Modeling and Simulation (2006)Google Scholar
  5. 5.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005)Google Scholar
  6. 6.
    Dietterich, T.G.: Ensemble Methods in Machine Learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  7. 7.
    Dzeroski, S., Zenko, B.: Is combining classifiers with stacking better than selecting the best one? Machine Learning 54(3), 255–273 (2004)MATHCrossRefGoogle Scholar
  8. 8.
    Everingham, M., Sivic, J., Zisserman, A.: “Hello! My name is... Buffy” – automatic naming of characters in TV video. In: BMVC (2006)Google Scholar
  9. 9.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive Logistic Regression: a Statistical View of Boosting. The Annals of Statistics 38(2) (2000)Google Scholar
  10. 10.
    Geback, T., Koumoutsakos, P.: Edge detection in microscopy images using curvelets. BMC Bioinformatics 10(75) (2009)Google Scholar
  11. 11.
    Geback, T., Schulz, M., Koumoutsakos, P., Detmar, M.: Tscratch: a novel and simple software tool for automated analysis of monolayer wound healing assays. Biotechniques 46, 265–274 (2009)Google Scholar
  12. 12.
    Hu, G., Mao, Z.: Bagging ensemble of svm based on negative correlation learning. In: IEEE International Conference on ICIS 2009, vol. 1, pp. 279–283 (2009)Google Scholar
  13. 13.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: A database for studying face recognition in unconstrained environments. University of Massachusetts, Amherst, TR 07-49 (October 2007)Google Scholar
  14. 14.
    Kim, H.C., Pang, S., Je, H.M., Kim, D., Bang, S.Y.: Constructing support vector machine ensemble. Pattern Recognition 36(12), 2757–2767 (2003)MATHCrossRefGoogle Scholar
  15. 15.
    Kocsor, A., Kovács, K., Szepesvári, C.: Margin Maximizing Discriminant Analysis. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 227–238. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  16. 16.
    Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. In: ICCV, pp. 365–372 (2009)Google Scholar
  17. 17.
    Lamprecht, M., Sabatini, D., Carpenter, A.: Cellprofiler: free, versatile software for automated biological image analysis. Biotechniques 42, 71–75 (2007)CrossRefGoogle Scholar
  18. 18.
    Liu, Y., Yao, X.: Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 29, 716–725 (1999)CrossRefGoogle Scholar
  19. 19.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  20. 20.
    Murphy, P., Aha, D.: UCI Repository of machine learning databases. Tech. rep., U. California, Dept. of Information and Computer Science, CA, US (1994)Google Scholar
  21. 21.
    Shalev-Shwartz, S., Singer, Y., Srebro, N.: Pegasos: Primal estimated sub-gradient solver for svm. In: ICML (2007)Google Scholar
  22. 22.
    Shivaswamy, P.K., Jebara, T.: Maximum relative margin and data-dependent regularization. Journal of Machine Learning Research (2010)Google Scholar
  23. 23.
    Wolpert, D.H.: Stacked generalization. Neural Networks 5(2), 241–259 (1992)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Zaritsky, A., Natan, S., Horev, J., Hecht, I., Wolf, L., Ben-Jacob, E., Tsarfaty, I.: Cell motility dynamics: A novel segmentation algorithm to quantify multi-cellular bright field microscopy images. PLoS ONE 6 (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Noga Levy
    • 1
  • Lior Wolf
    • 1
  1. 1.The Blavatnik School of Computer ScienceTel Aviv UniversityIsrael

Personalised recommendations