Advertisement

Improved Lower Bounds for Learning Intersections of Halfspaces

  • Adam R. Klivans
  • Alexander A. Sherstov
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4005)

Abstract

We prove new lower bounds for learning intersections of halfspaces, one of the most important concept classes in computational learning theory. Our main result is that any statistical-query algorithm for learning the intersection of \(\sqrt{n}\) halfspaces in n dimensions must make \(2^{\Omega(\sqrt{n})}\) queries. This is the first non-trivial lower bound on the statistical query dimension for this concept class (the previous best lower bound was n \(^{\Omega(log{\it n})}\)). Our lower bound holds even for intersections of low-weight halfspaces. In the latter case, it is nearly tight.

We also show that the intersection of two majorities (low-weight halfspaces) cannot be computed by a polynomial threshold function (PTF) with fewer than n \(^{\Omega((log{\it n})/loglog{\it n})}\) monomials. This is the first super-polynomial lower bound on the PTF length of this concept class, and is nearly optimal. For intersections of k=ω(logn) low-weight halfspaces, we improve our lower bound to \(\min\{2^{\Omega(\sqrt{n})},n^{\Omega(k/\log k)}\},\) which too is nearly optimal. As a consequence, intersections of even two halfspaces are not computable by polynomial-weight PTFs, the most expressive class of functions known to be efficiently learnable via Jackson’s Harmonic Sieve algorithm. Finally, we report our progress on the weak learnability of intersections of halfspaces under the uniform distribution.

Keywords

Boolean Function Unate Function Concept Class Weak Learnability Bend Function 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Alekhnovich, M., Braverman, M., Feldman, V., Klivans, A., Pitassi, T.: Learnability and automatizability. In: Proceedings of the 45th Annual Symposium on Foundations of Computer Science (FOCS) (2004)Google Scholar
  2. 2.
    Aspnes, J., Beigel, R., Furst, M., Rudich, S.: The expressive power of voting polynomials. In: STOC 1991: Proceedings of the twenty-third annual ACM symposium on Theory of computing, pp. 402–409. ACM Press, New York (1991)CrossRefGoogle Scholar
  3. 3.
    Beigel, R., Reingold, N., Spielman, D.: PP is closed under intersection. In: STOC 1991: Proceedings of the twenty-third annual ACM symposium on Theory of computing, pp. 1–9. ACM Press, New York (1991)CrossRefGoogle Scholar
  4. 4.
    Blum, A., Furst, M., Jackson, J., Kearns, M., Mansour, Y., Rudich, S.: Weakly learning DNF and characterizing statistical query learning using Fourier analysis. In: STOC 1994: Proceedings of the twenty-sixth annual ACM symposium on Theory of computing, pp. 253–262. ACM Press, New York (1994)CrossRefGoogle Scholar
  5. 5.
    Blum, A., Kalai, A., Wasserman, H.: Noise-tolerant learning, the parity problem, and the statistical query model. J. ACM 50(4), 506–519 (2003)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Bruck, J.: Harmonic analysis of polynomial threshold functions. SIAM J. Discrete Math. 3(2), 168–177 (1990)CrossRefMathSciNetMATHGoogle Scholar
  7. 7.
    Jackson, J.C.: The harmonic sieve: A novel application of Fourier analysis to machine learning theory and practice. PhD thesis, Carnegie Mellon University (1995)Google Scholar
  8. 8.
    Kearns, M.: Efficient noise-tolerant learning from statistical queries. In: STOC 1993: Proceedings of the twenty-fifth annual ACM symposium on theory of computing, pp. 392–401. ACM Press, New York (1993)CrossRefGoogle Scholar
  9. 9.
    Klivans, A., O’Donnell, R., Servedio, R.: Learning intersections and thresholds of halfspaces. In: Proceedings of the 43rd Annual Symposium on Foundations of Computer Science, pp. 177–186 (2002)Google Scholar
  10. 10.
    Klivans, A.R., Servedio, R.A.: Learning intersections of halfspaces with a margin. In: Shawe-Taylor, J., Singer, Y. (eds.) COLT 2004. LNCS, vol. 3120, pp. 348–362. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  11. 11.
    Krause, M., Pudlák, P.: On the computational power of depth 2 circuits with threshold and modulo gates. In: STOC 1994: Proceedings of the twenty-sixth annual ACM symposium on Theory of computing, pp. 48–57. ACM Press, New York (1994)CrossRefGoogle Scholar
  12. 12.
    Kushilevitz, E., Mansour, Y.: Learning decision trees using the fourier spectrum. In: STOC 1991: Proceedings of the twenty-third annual ACM symposium on Theory of computing, pp. 455–464. ACM Press, New York (1991)CrossRefGoogle Scholar
  13. 13.
    Kwek, S., Pitt, L.: PAC learning intersections of halfspaces with membership queries. Algorithmica 22(1/2), 53–75 (1998)CrossRefMathSciNetMATHGoogle Scholar
  14. 14.
    Minsky, M.L., Papert, S.A.: Perceptrons: expanded edition. MIT Press, Cambridge (1988)MATHGoogle Scholar
  15. 15.
    O’Donnell, R., Servedio, R.A.: New degree bounds for polynomial threshold functions. In: STOC 2003: Proceedings of the thirty-fifth annual ACM symposium on Theory of computing, pp. 325–334. ACM Press, New York (2003)CrossRefGoogle Scholar
  16. 16.
    Saks, M.E.: Slicing the hypercube. Surveys in combinatorics, 211–255 (1993)Google Scholar
  17. 17.
    Valiant, L.G.: A theory of the learnable. Commun. ACM 27(11), 1134–1142 (1984)CrossRefMATHGoogle Scholar
  18. 18.
    Vempala, S.: A random sampling based algorithm for learning the intersection of halfspaces. In: Proceedings of the 38th Annual Symposium on Foundations of Computer Science, pp. 508–513 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Adam R. Klivans
    • 1
  • Alexander A. Sherstov
    • 1
  1. 1.Department of Computer SciencesThe University of Texas at AustinAustinUSA

Personalised recommendations