Can Entropic Regularization Be Replaced by Squared Euclidean Distance Plus Additional Linear Constraints

  • Manfred K. Warmuth
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4005)


There are two main families of on-line algorithms depending on whether a relative entropy or a squared Euclidean distance is used as a regularizer. The difference between the two families can be dramatic. The question is whether one can always achieve comparable performance by replacing the relative entropy regularization by the squared Euclidean distance plus additional linear constraints. We formulate a simple open problem along these lines for the case of learning disjunctions.


Euclidean Distance Weight Vector Relative Entropy Neural Information Processing System Perceptron Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. [KRS01]
    Khardon, R., Roth, D., Servedio, R.: Efficiency versus convergence of Boolean kernels for on-line learning algorithms. In: Advances in Neural Information Processing Systems 14, pp. 423–430. MIT Press, Cambridge (2001)Google Scholar
  2. [KW97]
    Kivinen, J., Warmuth, M.K.: Additive versus exponentiated gradient updates for linear prediction. Information and Computation 132(1), 1–64 (1997)CrossRefMathSciNetMATHGoogle Scholar
  3. [KWA97]
    Kivinen, J., Warmuth, M.K., Auer, P.: The perceptron algorithm vs. winnow: linear vs. logarithmic mistake bounds when few input variables are relevant. Artificial Intelligence 97, 325–343 (1997)CrossRefMathSciNetMATHGoogle Scholar
  4. [Lit88]
    Littlestone, N.: Learning when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning 2, 285–318 (1988)Google Scholar
  5. [LW04]
    Long, P.M., Wu, X.: Mistake bounds for maximum entropy discrimination. In: Advances in Neural Information Processing Systems, vol. 17, MIT Press, Cambridge (December 2004)Google Scholar
  6. [SM05]
    Warmuth, M.K., Vishwanathan, S.V.N.: Leaving the span. In: Auer, P., Meir, R. (eds.) COLT 2005. LNCS, vol. 3559, pp. 366–381. Springer, Heidelberg (2005); A longer journal version is in preperationCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Manfred K. Warmuth
    • 1
  1. 1.Univ. of Calif. at Santa Cruz 

Personalised recommendations