Fast Training of Effective Multi-class Boosting Using Coordinate Descent Optimization

  • Guosheng Lin
  • Chunhua Shen
  • Anton van den Hengel
  • David Suter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7725)


We present a novel column generation based boosting method for multi-class classification. Our multi-class boosting is formulated in a single optimization problem as in [1]. Different from most existing multi-class boosting methods, which use the same set of weak learners for all the classes, we train class specified weak learners (i.e., each class has a different set of weak learners). We show that using separate weak learner sets for each class leads to fast convergence, without introducing additional computational overhead in the training procedure. To further make the training more efficient and scalable, we also propose a fast coordinate descent method for solving the optimization problem at each boosting iteration. The proposed coordinate descent method is conceptually simple and easy to implement in that it is a closed-form solution for each coordinate update. Experimental results on a variety of datasets show that, compared to a range of existing multi-class boosting methods, the proposed method has much faster convergence rate and better generalization performance in most cases. We also empirically show that the proposed fast coordinate descent algorithm needs less training time than the MultiBoost algorithm in [1].


Training Time Column Generation Master Problem Weak Learner Coordinate Descent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Shen, C., Hao, Z.: A direct formulation for totally-corrective multi-class boosting. In: Proc. IEEE Conf. Comp. Vis. Patt. Recogn. (2011)Google Scholar
  2. 2.
    Schapire, R.E., Freund, Y., Bartlett, P., Lee, W.S.: boosting the margin: A new explanation for the effectiveness of voting methods. Annals of Statistics 26, 1651–1686 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vision 57, 137–154 (2004)CrossRefGoogle Scholar
  4. 4.
    Dietterich, T.G., Bakiri, G.: Solving multiclass learning problems via error-correcting output codes. J. Artif. Int. Res. 2, 263–286 (1995)zbMATHGoogle Scholar
  5. 5.
    Guruswami, V., Sahai, A.: Multiclass learning, boosting, and error-correcting codes. In: Proc. Annual Conf. Computational Learning Theory, pp. 145–155. ACM, New York (1999)Google Scholar
  6. 6.
    Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. In: Machine Learn., pp. 80–91 (1999)Google Scholar
  7. 7.
    Demiriz, A., Bennett, K.P., Shawe-Taylor, J.: Linear programming boosting via column generation. Mach. Learn. 46, 225–254 (2002)zbMATHCrossRefGoogle Scholar
  8. 8.
    Shen, C., Li, H.: On the dual formulation of boosting algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 32, 2216–2231 (2010)CrossRefGoogle Scholar
  9. 9.
    Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound constrained optimization. ACM Trans. Math. Software (1994)Google Scholar
  10. 10.
    Yuan, G.X., Chang, K.W., Hsieh, C.J., Lin, C.J.: A comparison of optimization methods and software for large-scale l1-regularized linear classification. J. Mach. Learn. Res., 3183–3234 (2010)Google Scholar
  11. 11.
    Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: A library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)zbMATHGoogle Scholar
  12. 12.
    Guillaumin, M., Verbeek, J., Schmid, C.: Multimodal semi-supervised learning for image classification. In: Proc. IEEE Conf. Comp. Vis. Patt. Recogn. (2010)Google Scholar
  13. 13.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: Proc. IEEE Conf. Comp. Vis. Patt. Recogn., vol. 2, pp. 2169–2178 (2006)Google Scholar
  14. 14.
    Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: SUN database: Large-scale scene recognition from abbey to zoo. In: Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 3485–3492 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Guosheng Lin
    • 1
  • Chunhua Shen
    • 1
  • Anton van den Hengel
    • 1
  • David Suter
    • 1
  1. 1.The Australian Centre for Visual Technologies, School of Computer ScienceThe University of AdelaideAustralia

Personalised recommendations