Skip to main content
SpringerLink
Log in
Book cover

Asian Conference on Intelligent Information and Database Systems

ACIIDS 2016: Intelligent Information and Database Systems pp 661–670Cite as

Online DC Optimization for Online Binary Linear Classification

  • Conference paper

  • 1552 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9622))

Abstract

This paper concerns online algorithms for online binary linear classification (OBLC) problems in Machine learning. In a sense of “online” classification, an instance sequence is given step by step and on each round, these problems consist in finding a linear classifier for predicting to which label a new instance belongs. In OBCL, the quality of predictions is assessed by a loss function, specifically 0–1 loss function. In fact, this loss function is nonconvex, nonsmooth and thus, such problems become intractable. In literature, Perceptron is a well-known online classification algorithm, in which one substitutes a surrogate convex loss function for the 0–1 loss function. In this paper, we investigate an efficient DC loss function which is a suitable approximation of the usual 0–1 loss function. Basing on Online DC (Difference of Convex functions) programming and Online DCA (DC Algorithms) [10], we develop an online classification algorithm. Numerical experiments on several test problems show the efficiency of our proposed algorithm with respect to Perceptron.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.ics.uci.edu/~mlearn/MLRepository.html.

  2. 2.

    http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/.

References

  1. Azoury, K., Warmuth, M.: Relative loss bounds for on-line density estimation with the exponential family of distributions. Mach. Learn. 43(3), 211–246 (2001). http://dx.doi.org/10.1023/A3A1010896012157

    Article  MATH  Google Scholar 

  2. Crammer, K., Singer, Y.: Ultraconservative online algorithms for multiclass problems. J. Mach. Learn. Res. 3, 951–991 (2003). http://dx.doi.org/10.1162/jmlr.2003.3.4-5.951

    MathSciNet  MATH  Google Scholar 

  3. Gentile, C.: A new approximate maximal margin classification algorithm. J. Mach. Learn. Res. 2, 213–242 (2002). http://dl.acm.org/citation.cfm?id=944790.944811

    MathSciNet  MATH  Google Scholar 

  4. Gentile, C.: The robustness of the p-norm algorithms. Mach. Learn. 53(3), 265–299 (2003). http://dx.doi.org/10.1023/A:1026319107706

    Article  MATH  Google Scholar 

  5. Hoi, S.C.H., Wang, J., Zhao, P.: Libol: a library for online learning algorithms. J. Mach. Learn. Res. 15(1), 495–499 (2014). http://dl.acm.org/citation.cfm?id=2627435.2627450

    MATH  Google Scholar 

  6. Kalai, A., Vempala, S.: Efficient algorithms for online decision problems. J. Comput. Syst. Sci. 71(3), 291–307 (2005). http://www.sciencedirect.com/science/article/pii/S0022000004001394

    Article  MathSciNet  MATH  Google Scholar 

  7. Kivinen, J., Warmuth, M.: Relative loss bounds for multidimensional regression problems. Mach. Learn. 45(3), 301–329 (2001). http://dx.doi.org/10.1023/A:1017938623079

    Article  MATH  Google Scholar 

  8. Kivinen, J., Warmuth, M.K.: Exponentiated gradient versus gradient descent for linear predictors. Inf. Comput. 132(1), 1–63 (1997). http://www.sciencedirect.com/science/article/pii/S0890540196926127

    Article  MathSciNet  MATH  Google Scholar 

  9. Le Thi, H.A.: DC programming and DCA (2012). http://www.lita.univ-lorraine.fr/~lethi

  10. Le Thi, H.A., Ho, V.T.: Online DC programming and DCA. LITA, Technical report, University of Lorraine, Metz, France (2015)

    Google Scholar 

  11. Le Thi, H.A., Pham Dinh, T.: The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems. Ann. Oper. Res. 133(1–4), 23–46 (2005)

    MathSciNet  MATH  Google Scholar 

  12. Li, Y., Long, P.: The relaxed online maximum margin algorithm. Mach. Learn. 46(1–3), 361–387 (2002). http://dx.doi.org/10.1023/A:1012435301888

    Article  MATH  Google Scholar 

  13. Novikoff, A.B.: On convergence proofs for perceptrons. Proc. Symp. Math. Theor. Automata 12, 615–622 (1963). http://citeseer.comp.nus.edu.sg/context/494822/0

    MathSciNet  MATH  Google Scholar 

  14. Pham Dinh, T., Le Thi, H.A.: Convex analysis approach to d.c. programming: theory, algorithms and applications. Acta Mathematica Vietnamica 22(1), 289–355 (1997)

    MathSciNet  MATH  Google Scholar 

  15. Pham Dinh, T., Le Thi, H.A.: DC optimization algorithms for solving the trust region subproblem. SIAM J. Optim. 8(2), 476–505 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  16. Rosenblatt, F.: The perceptron: a probabilistic model for information storage and organization in the brain. Psychol. Rev. 65(6), 386–408 (1958)

    Article  MathSciNet  Google Scholar 

  17. Shalev-Shwartz, S.: Online learning: theory, algorithms, and applications. Ph.D. thesis, The Hebrew University of Jerusalem (2007)

    Google Scholar 

  18. Shalev-Shwartz, S., Singer, Y.: A primal-dual perspective of online learning algorithms. Mach. Learn. 69(2–3), 115–142 (2007). http://dx.doi.org/10.1007/s10994-007-5014-x

    Article  Google Scholar 

  19. Van Der Malsburg, C.: Frank rosenblatt: principles of neurodynamics: perceptrons and the theory of brain mechanisms. In: Palm, G., Aertsen, A. (eds.) Brain Theory, pp. 245–248. Springer, Heidelberg (1986). http://dx.doi.org/10.1007/978-3-642-70911-1_20

    Chapter  Google Scholar 

  20. Zinkevich, M.: Online convex programming and generalized infinitesimal gradient ascent. In: Fawcett, T., Mishra, N. (eds.) Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 928–936 (2003). http://www.aaai.org/Papers/ICML/2003/ICML03-120.pdf

Download references

Acknowledgements

This research is funded by Foundation for Science and Technology Development of Ton Duc Thang University (FOSTECT), website: http://fostect.tdt.edu.vn, under Grant FOSTECT.2015.BR.15.

Author information

Authors and Affiliations

  1. Laboratory of Theoretical and Applied Computer Science EA 3097, University of Lorraine, Ile du Saulcy, 57045, Metz, France

    Ho Vinh Thanh & Le Thi Hoai An

  2. FPT University, Hoa Lac High Tech Park, Hanoi, Vietnam

    Bui Dinh Chien

Authors
  1. Ho Vinh Thanh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Le Thi Hoai An
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bui Dinh Chien
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ho Vinh Thanh .

Editor information

Editors and Affiliations

  1. Wrocław University of Technology, Wrocław, Poland

    Ngoc Thanh Nguyen

  2. Wrocław University of Technology, Wrocław, Poland

    Bogdan Trawiński

  3. Iwate Prefectural University, Takizawa, Japan

    Hamido Fujita

  4. National University of Kaohsiung, Kaohsiung, Taiwan

    Tzung-Pei Hong

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Thanh, H.V., An, L.T.H., Chien, B.D. (2016). Online DC Optimization for Online Binary Linear Classification. In: Nguyen, N.T., Trawiński, B., Fujita, H., Hong, TP. (eds) Intelligent Information and Database Systems. ACIIDS 2016. Lecture Notes in Computer Science(), vol 9622. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-49390-8_64

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-49390-8_64

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-49389-2

  • Online ISBN: 978-3-662-49390-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation