On Noise-Tolerant Learning of Sparse Parities and Related Problems

  • Elena Grigorescu
  • Lev Reyzin
  • Santosh Vempala
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6925)


We consider the problem of learning sparse parities in the presence of noise. For learning parities on r out of n variables, we give an algorithm that runs in time \(\mathrm{poly}\left(\log \frac{1}{\delta}, \frac{1}{1-2\eta}\right) n^{ \left(1+(2\eta)^2+ o(1)\right)r/2}\) and uses only \(\frac{r \log(n/\delta) \omega(1)}{(1 - 2\eta)^2}\) samples in the random noise setting under the uniform distribution, where η is the noise rate and δ is the confidence parameter. From previously known results this algorithm also works for adversarial noise and generalizes to arbitrary distributions. Even though efficient algorithms for learning sparse parities in the presence of noise would have major implications to learning other hypothesis classes, our work is the first to give a bound better than the brute-force O(n r ). As a consequence, we obtain the first nontrivial bound for learning r-juntas in the presence of noise, and also a small improvement in the complexity of learning DNF, under the uniform distribution.


Arbitrary Distribution Statistical Query Noise Rate Noiseless Case Time Poly 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alekhnovich, M., Braverman, M., Feldman, V., Klivans, A.R., Pitassi, T.: The complexity of properly learning simple concept classes. J. Comput. Syst. Sci. 74(1), 16–34 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Andoni, A., and Indyk, P. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In: FOCS, pp. 459–468 (2006)Google Scholar
  3. 3.
    Angluin, D., Laird, P.D.: Learning from noisy examples. Machine Learning 2(4), 343–370 (1987)Google Scholar
  4. 4.
    Blum, A., Frieze, A.M., Kannan, R., Vempala, S.: A polynomial-time algorithm for learning noisy linear threshold functions. Algorithmica 22(1/2), 35–52 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Blum, A., Furst, M.L., Jackson, J.C., Kearns, M.J., Mansour, Y., Rudich, S.: Weakly learning dnf and characterizing statistical query learning using fourier analysis. In: STOC, pp. 253–262 (1994)Google Scholar
  6. 6.
    Blum, A., Kalai, A., Wasserman, H.: Noise-tolerant learning, the parity problem, and the statistical query model. J. ACM 50(4), 506–519 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Buhrman, H., García-Soriano, D., Matsliah, A.: Learning parities in the mistake-bound model. Inf. Process. Lett. 111(1), 16–21 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Erickson, J.: Lower bounds for linear satisfiability problems. In: SODA, Philadelphia, PA, USA, pp. 388–395 (1995)Google Scholar
  9. 9.
    Feldman, V., Gopalan, P., Khot, S., Ponnuswami, A.K.: On agnostic learning of parities, monomials, and halfspaces. SIAM J. Comput. 39(2), 606–645 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Goldreich, O., Levin, L.A.: A hard-core predicate for all one-way functions. In: STOC, pp. 25–32 (1989)Google Scholar
  11. 11.
    Hopper, N.J., Blum, M.: Secure human identification protocols. In: Boyd, C. (ed.) ASIACRYPT 2001. LNCS, vol. 2248, pp. 52–66. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  12. 12.
    Jackson, J.C., Lee, H.K., Servedio, R.A., Wan, A.: Learning random monotone DNF. In: Goel, A., Jansen, K., Rolim, J.D.P., Rubinfeld, R. (eds.) APPROX and RANDOM 2008. LNCS, vol. 5171, pp. 483–497. Springer, Heidelberg (2008)Google Scholar
  13. 13.
    Kalai, A.T., Servedio, R.A.: Boosting in the presence of noise. J. Comput. Syst. Sci. 71(3), 266–290 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Katz, J.: Efficient cryptographic protocols based on the hardness of learning parity with noise. In: IMA Int. Conf., pp. 1–15 (2007)Google Scholar
  15. 15.
    Kearns, M.J.: Efficient noise-tolerant learning from statistical queries. In: STOC, pp. 392–401 (1993)Google Scholar
  16. 16.
    Klivans, A.R., Servedio, R.A.: Toward attribute efficient learning of decision lists and parities. In: Shawe-Taylor, J., Singer, Y. (eds.) COLT 2004. LNCS (LNAI), vol. 3120, pp. 224–238. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  17. 17.
    Kushilevitz, E., Mansour, Y.: Learning decision trees using the fourier spectrum. SIAM J. Comput. 22(6), 1331–1348 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Lyubashevsky, V.: The parity problem in the presence of noise, decoding random linear codes, and the subset sum problem. In: Chekuri, C., Jansen, K., Rolim, J.D.P., Trevisan, L. (eds.) APPROX 2005 and RANDOM 2005. LNCS, vol. 3624, pp. 378–389. Springer, Heidelberg (2005)Google Scholar
  19. 19.
    Mossel, E., O’Donnell, R., Servedio, R.A.: Learning functions of k relevant variables. J. Comput. Syst. Sci. 69(3), 421–434 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Panigrahy, R., Talwar, K., Wieder, U.: A geometric approach to lower bounds for approximate near-neighbor search and partial match. In: FOCS, pp. 414–423 (2008)Google Scholar
  21. 21.
    Panigrahy, R., Talwar, K., Wieder, U.: Lower bounds on near neighbor search via metric expansion. In: FOCS, pp. 805–814 (2010)Google Scholar
  22. 22.
    Peikert, C.: Public-key cryptosystems from the worst-case shortest vector problem: extended abstract. In: STOC, pp. 333–342 (2009)Google Scholar
  23. 23.
    Regev, O.: On lattices, learning with errors, random linear codes, and cryptography. J. ACM 56(6) (2009)Google Scholar
  24. 24.
    Sellie, L.: Learning random monotone dnf under the uniform distribution. In: COLT, pp. 181–192 (2008)Google Scholar
  25. 25.
    Sellie, L.: Exact learning of random dnf over the uniform distribution. In: STOC, pp. 45–54 (2009)Google Scholar
  26. 26.
    Valiant, L.G.: A theory of the learnable. Commun. ACM 27(11), 1134–1142 (1984)CrossRefzbMATHGoogle Scholar
  27. 27.
    Verbeurgt, K.A.: Learning dnf under the uniform distribution in quasi-polynomial time. In: COLT, pp. 314–326 (1990)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Elena Grigorescu
    • 1
  • Lev Reyzin
    • 1
  • Santosh Vempala
    • 1
  1. 1.School of Computer ScienceGeorgia Institute of TechnologyAtlanta

Personalised recommendations