Progressive refinement for support vector machines

  • Kiri L. WagstaffEmail author
  • Michael Kocurek
  • Dominic Mazzoni
  • Benyang Tang


Support vector machines (SVMs) have good accuracy and generalization properties, but they tend to be slow to classify new examples. In contrast to previous work that aims to reduce the time required to fully classify all examples, we present a method that provides the best-possible classification given a specific amount of computational time. We construct two SVMs: a “full” SVM that is optimized for high accuracy, and an approximation SVM (via reduced-set or subset methods) that provides extremely fast, but less accurate, classifications. We apply the approximate SVM to the full data set, estimate the posterior probability that each classification is correct, and then use the full SVM to reclassify items in order of their likelihood of misclassification. Our experimental results show that this method rapidly achieves high accuracy, by selectively devoting resources (reclassification) only where needed. It also provides the first such progressive SVM solution that can be applied to multiclass problems.


Support vector machines Efficiency Reclassification 


  1. Burges CJC (1996) Simplified support vector decision rules. In: Proceedings of the thirteenth international conference on machine learning, pp 71–77Google Scholar
  2. Burges CJC (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov 2(2): 121–167CrossRefGoogle Scholar
  3. Castano R, Wagstaff KL, Chien S, Stough TM, Tang B (2007) On-board analysis of uncalibrated data for a spacecraft at Mars. In: Proceedings of the thirteenth international conference on knowledge discovery and data mining, pp 922–930Google Scholar
  4. Chien S, Sherwood R, Tran D, Cichy B, Rabideau G, Castaño R, Davies A, Mandel D, Frye S, Trout B, Shulman S, Boyer D (2005) Using autonomy flight software to improve science return on Earth observing one. J Aerosp Comput Inf Commun 2(4): 196–216CrossRefGoogle Scholar
  5. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20: 273–297zbMATHGoogle Scholar
  6. DeCoste D (2002) Anytime interval-valued outputs for kernel machines: fast support vector machine classification via distance geometry. In: Proceedings of the nineteenth international conference on machine learning, pp 99–106Google Scholar
  7. DeCoste D (2003) Anytime query-tuned kernel machines via Cholesky factorization. In: Proceedings of the SIAM international conference on data mining (SIAMDM-03)Google Scholar
  8. DeCoste D, Mazzoni D (2003) Fast query-optimized kernel machine classification via incremental approximate nearest support vectors. In: Proceedings of the twentieth international conference on machine learning, pp 115–122Google Scholar
  9. DeCoste D, Scholkopf B (2002) Training invariant support vector machines. Mach Learn 26(1–3): 161–190CrossRefGoogle Scholar
  10. Diner DJ, Beckert JC, Reilly TH, Bruegge CJ, Conel JE, Kahn R, Martonchik JV, Ackerman TP, Gordon HR, Muller J-P, Myneni R, Sellers RJ, Pinty B, Verstraete MM (1998) Multiangle imaging spectroradiometer (MISR) instrument description and experiment overview. IEEE Trans Geosci Remote Sens 36: 1072–1087CrossRefGoogle Scholar
  11. Duan K, Keerthi SS (2005) Which is the best multiclass SVM method? An empirical study. In: Proceedings of multiple classifier systems, pp 278–285Google Scholar
  12. Hastie T, Tibshirani R (1998) Classification by pairwise coupling. Ann Appl Stat 26(2): 451–471zbMATHMathSciNetGoogle Scholar
  13. Lin HT, Lin CJ, Weng RC (2003) A note on Platt’s probabilistic outputs for support vector machines. Technical report, National Taiwan UniversityGoogle Scholar
  14. Mazzoni D, Garay MJ, Davies R, Nelson D (2007) An operational MISR pixel classifier using support vector machines. Remote Sens Environ 107(1–2): 149–158CrossRefGoogle Scholar
  15. Newman DJ, Hettich S, Blake CL, Merz CJ (1998) UCI repository of machine learning databases.
  16. Platt JC (1999) Probabilities for SV machines. In: Smola AJ, Bartlett P, Schölkopf B, Schuurmans D (eds) Advances in large margin classifiers. MIT Press, Cambridge, pp 61–74Google Scholar
  17. Ratsch M, Romdhani S, Vetter T (2004) Efficient face detection by a cascaded support vector machine using Haar-like features. In: Proceedings of the German pattern recognition symposium, pp 62–70Google Scholar
  18. Romdhani S, Torr PHS, Schölkopf B, Blake A (2001) Computationally efficient face detection. In: Proceedings of the eighth international conference on computer vision, pp 695–700Google Scholar
  19. Tang B, Mazzoni D (2006) Multiclass reduced-set support vector machines. In: Proceedings of the twenty-third international conference on machine learning, pp 921–928Google Scholar

Copyright information

© The Author(s) 2009

Authors and Affiliations

  • Kiri L. Wagstaff
    • 1
    Email author
  • Michael Kocurek
    • 2
  • Dominic Mazzoni
    • 1
    • 3
  • Benyang Tang
    • 1
  1. 1.Jet Propulsion LaboratoryCalifornia Institute of TechnologyPasadenaUSA
  2. 2.California Institute of TechnologyPasadenaUSA
  3. 3.Google Inc.Santa MonicaUSA

Personalised recommendations