Machine Learning

, Volume 46, Issue 1, pp 225–254

Linear Programming Boosting via Column Generation

  • Ayhan Demiriz
  • Kristin P. Bennett
  • John Shawe-Taylor

DOI: 10.1023/A:1012470815092

Cite this article as:
Demiriz, A., Bennett, K.P. & Shawe-Taylor, J. Machine Learning (2002) 46: 225. doi:10.1023/A:1012470815092


We examine linear program (LP) approaches to boosting and demonstrate their efficient solution using LPBoost, a column generation based simplex method. We formulate the problem as if all possible weak hypotheses had already been generated. The labels produced by the weak hypotheses become the new feature space of the problem. The boosting task becomes to construct a learning function in the label space that minimizes misclassification error and maximizes the soft margin. We prove that for classification, minimizing the 1-norm soft margin error function directly optimizes a generalization error bound. The equivalent linear program can be efficiently solved using column generation techniques developed for large-scale optimization problems. The resulting LPBoost algorithm can be used to solve any LP boosting formulation by iteratively optimizing the dual misclassification costs in a restricted LP and dynamically generating weak hypotheses to make new LP columns. We provide algorithms for soft margin classification, confidence-rated, and regression boosting problems. Unlike gradient boosting algorithms, which may converge in the limit only, LPBoost converges in a finite number of iterations to a global solution satisfying mathematically well-defined optimality conditions. The optimal solutions of LPBoost are very sparse in contrast with gradient based methods. Computationally, LPBoost is competitive in quality and computational cost to AdaBoost.

ensemble learning boosting linear programming sparseness soft margin 
Download to read the full article text

Copyright information

© Kluwer Academic Publishers 2002

Authors and Affiliations

  • Ayhan Demiriz
    • 1
  • Kristin P. Bennett
    • 2
    • 3
  • John Shawe-Taylor
    • 4
  1. 1.Department of Decision Sciences and Eng. SystemsRensselaer Polytechnic InstituteTroyUSA
  2. 2.Rensselaer Polytechnic InstituteTroyUSA
  3. 3.Microsoft ResearchRedmondUSA
  4. 4.Department of Computer Science, Royal HollowayUniversity of LondonEgham, SurreyUK

Personalised recommendations