Skip to main content
Book cover

Learning Theory and Kernel Machines pp 463–476Cite as

Learning Arithmetic Circuits via Partial Derivatives

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNAI,volume 2777)

Abstract

We present a polynomial time algorithm for learning several models of algebraic computation. We show that any arithmetic circuit whose partial derivatives induce a “low”-dimensional vector space is exactly learnable from membership and equivalence queries. As a consequence we obtain the first polynomial time algorithm for learning depth three multilinear arithmetic circuits. In addition, we give the first polynomial time algorithms for learning restricted algebraic branching programs and noncommutative arithmetic formulae. Previously only restricted versions of depth-2 arithmetic circuits were known to be learnable in polynomial time. Our learning algorithms can be viewed as solving a generalization of the well known polynomial interpolation problem where the unknown polynomial has a succinct representation. We can learn representations of polynomials encoding exponentially many monomials. Our techniques combine a careful algebraic analysis of arithmetic circuits’ partial derivatives with the “multiplicity automata” techniques due to Beimel et al [BBB+00].

Keywords

  • PAC learning
  • learning with queries

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-540-45167-9_34
  • Chapter length: 14 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-540-45167-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aspnes, J., Beigel, R., Furst, M., Rudich, S.: The expressive power of voting polynomials. In: Awerbuch, B. (ed.) Proceedings of the 23rd Annual ACM Symposium on the Theory of Computing, pp. 402–409. ACM Press, New York (1991)

    Google Scholar 

  2. Arora, S., Lund, C., Motwani, R., Sudan, M., Szegedy, M.: Proof verification and the hardness of approximation problems. Journal of the ACM 45(3), 501–555 (1998)

    Google Scholar 

  3. Angluin, D.: Queries and concept learning. Machine Learning 2, 319 (1987)

    Google Scholar 

  4. Bshouty, D., Bshouty, N.H.: On interpolating arithmetic readonce formulas with exponentiation. Journal of Computer and System Sciences 56(1), 112–124 (1998)

    CrossRef  MATH  MathSciNet  Google Scholar 

  5. Beimel, A., Bergadano, F., Bshouty, N.H., Kushilevitz, E., Varricchio, S.: Learning functions represented as multiplicity automata. J. ACM 47(3), 506–530 (2000)

    Google Scholar 

  6. Bergadano, F., Bshouty, N.H., Tamon, C., Varricchio, S.: On learning branching programs and small depth circuits. In: Ben-David, S. (ed.) EuroCOLT 1997. LNCS (LNAI), vol. 1208, pp. 150–161. Springer, Heidelberg (1997)

    Google Scholar 

  7. Bshouty, N., Mansour, Y.: Simple learning algorithms for decision trees and multivariate polynomials. SICOMP: SIAM Journal on Computing 31 (2002)

    Google Scholar 

  8. Bshouty, N.H., Tamon, C., Wilson, D.K.: On learning width two branching programs. In: Proc. 9th Annu. Conf. on Comput. Learning Theory, pp. 224–227. ACM Press, New York (1996)

    CrossRef  Google Scholar 

  9. Carlyle, J.W., Paz, A.: Realizations by stochastic finite automata. Journal of Computer and System Sciences 5(1), 26–40 (1971)

    CrossRef  MATH  MathSciNet  Google Scholar 

  10. Fliess, M.: Matrices de Hankel. J. Math. Pures et Appl. 53, 197–224 (1974)

    MATH  MathSciNet  Google Scholar 

  11. Grigoriev, D., Karpinski, M., Singer, M.F.: Computational complexity of sparse rational interpolation. SIAM Journal on Computing 23(1), 1–11 (1994)

    CrossRef  MATH  MathSciNet  Google Scholar 

  12. Grigoriev, D., Razborov, A.A.: Exponential complexity lower bounds for depth 3 arithmetic circuits in algebras of functions over finite fields. In: IEEE (ed.) 39th Annual Symposium on Foundations of Computer Science: proceedings, Palo Alto, California, 1109 Spring Street, Suite 300, Silver Spring, MD 20910, USA, November 8–11, pp. 269–278. IEEE Computer Society Press, Los Alamitos (1998)

    Google Scholar 

  13. Hastad, J.: Almost optimal lower bounds for small depth circuits. In: Proceedings of the 18th Annual ACM Symposium on Theory of Computing, Berkeley, California, May 28-30, pp. 6–20 (1986)

    Google Scholar 

  14. Huang, M., Rao, T.: Interpolation of sparse multivariate polynomials over large finite fields with applications. ALGORITHMS: Journal of Algorithms 33 (1999)

    Google Scholar 

  15. Jackson, J.C., Klivans, A.R., Servedio, R.A.: Learnability beyond ACÔ. In: Proceedings of the 34th Annual ACM Symposium on Theory of Computing (STOC 2002), New York, May 19–21, pp. 776–784. ACM Press, New York (2002)

    CrossRef  Google Scholar 

  16. Klivans, A.R., Spielman, D.: Randomness efficient identity testing of multivariate polynomials. In: ACM (ed.) Proceedings of the 33rd Annual ACM Symposium on Theory of Computing: Hersonissos, Crete, Greece, New York, NY, USA, July 6–8, pp. 216–223. ACM Press, New York (2001)

    Google Scholar 

  17. Kaltofen, E., Trager, B.M.: Computing with polynomials given by black boxes for their evaluations: Greatest common divisors, factorization, separation of numerators and denominators. Journal of Symbolic Computation 9(3), 300–320 (301–320) (1990)

    Google Scholar 

  18. Linial, N., Mansour, Y., Nisan, N.: Constant depth circuits, Fourier transform and learnability. Journal of the ACM 40(3), 607–620 (1993)

    CrossRef  MATH  MathSciNet  Google Scholar 

  19. Nisan, N.: Lower bounds for non-commutative computation (extended abstract). In: Proceedings of the 23rd Annual ACM Symposium on Theory of Computing, New Orleans, Louisiana, May 6–8, pp. 410–418 (1991)

    Google Scholar 

  20. Nisan, N., Wigderson, A.: Lower bounds on arithmetic circuits via partial derivatives. CMPCMPL: Computational Complexity 6 (1997)

    Google Scholar 

  21. Ohnishi, H., Seki, H., Kasami, T.: A polynomial time learning algorithm for recognizable series. TIEICE: IEICE Transactions on Communications/ Electronics/Information and Systems (1994)

    Google Scholar 

  22. Schapire, R.E., Sellie, L.M.: Learning sparse multivariate polynomials over a field with queries and counterexamples. Journal of Computer and System Sciences 52(2), 201–213 (1996)

    CrossRef  MATH  MathSciNet  Google Scholar 

  23. Schwartz, J.T.: Fast probabilistic algorithms for verification of polynomial identities. Journal of the ACM 27(4), 701–717 (1980)

    CrossRef  MATH  Google Scholar 

  24. Sudan, M., Trevisan, L., Vadhan, S.: Pseudorandom generators without the XOR lemma. JCSS: Journal of Computer and System Sciences 62 (2001)

    Google Scholar 

  25. Shpilka, A., Wigderson, A.: Depth-3 arithmetic circuits over fields of characteristic zero. CMPCMPL: Computational Complexity 10 (2001)

    Google Scholar 

  26. Zippel, R.: Probabilistic algorithms for sparse polynomials. In: Ng, K.W. (ed.) EUROSAM 1979 and ISSAC 1979. LNCS, vol. 72. Springer, Heidelberg (1979)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Division of Engineering and Applied Sciences, Harvard, Cambridge, MA, 02139, USA

    Adam R. Klivans

  2. LCS,MIT, DEAS, Harvard, Cambridge, MA, USA

    Amir Shpilka

Authors
  1. Adam R. Klivans
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amir Shpilka
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. MPI for Biological Cybernetics, Spemannstr. 38, 72076, Tübingen, Germany

    Bernhard Schölkopf

  2. University of California, Santa Cruz

    Manfred K. Warmuth

Rights and permissions

Reprints and Permissions

Copyright information

© 2003 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Klivans, A.R., Shpilka, A. (2003). Learning Arithmetic Circuits via Partial Derivatives. In: Schölkopf, B., Warmuth, M.K. (eds) Learning Theory and Kernel Machines. Lecture Notes in Computer Science(), vol 2777. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-45167-9_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-45167-9_34

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-40720-1

  • Online ISBN: 978-3-540-45167-9

  • eBook Packages: Springer Book Archive