COLT 2002: Computational Learning Theory pp 287-302 | Cite as

On the Proper Learning of Axis Parallel Concepts

  • Nader H. Bshouty
  • Lynn Burroughs
Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2375)

Abstract

We study the proper learnability of axis parallel concept classes in the PAC learning model and in the exact learning model with membership and equivalence queries. These classes include union of boxes, DNF, decision trees and multivariate polynomials. For the constant dimensional axis parallel concepts C we show that the following problems have the same time complexity
  1. 1.

    C is α-properly exactly learnable (with hypotheses of size at most α times the target size) from membership and equivalence queries.

     
  2. 2.

    C is α-properly PAC learnable (without membership queries) under any product distribution.

     
  3. 3.

    There is an α-approximation algorithm for the MinEqui C problem. (given a g ∈ C find a minimal size f ∈ C that is equivalent to g).

     

In particular, C is α-properly learnable in poly time from membership and equivalence queries if and only if C is α-properly PAC learnable in poly time under the product distribution if and only if MinEqui C has a poly time α-approximation algorithm. Using this result we give the first proper learning algorithm of decision trees over the constant dimensional domain and the first negative results in proper learning from membership and equivalence queries for many classes.

For the non-constant dimensional axis parallel concepts we show that with the equivalence oracle (1) ⇒ (3). We use this to show that (binary) decision trees are not properly learnable in polynomial time (assuming P≠NP) and DNF is not sε-properly learnable (∈ < 1) in polynomial time even with an NP-oracle (assuming Σ 2 p ≠ P NP ).

Keywords

Polynomial Time Boolean Function Multivariate Polynomial Membership Query Equivalence Query 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [A88]
    D. Angluin. Queries and concept learning. Machine Learning, 319–342, 1988.Google Scholar
  2. [ABSS97]
    S. Arora, L. Babai, J. Stern, Z. Sweedyk. The hardness of approximate optima in lattices, codes, and systems of linear equations. JCSS, 43, 317–331, 1997.MathSciNetGoogle Scholar
  3. [BBBKV00]
    A. Beimel, F. Bergadano, N. H. Bshouty, E. Kushilevitz, S. Varricchio. Learning functions represented as multiplicity automata. JACM 47(3): 506–530, 2000.MATHCrossRefMathSciNetGoogle Scholar
  4. [BK98]
    A. Beimel and E. Kushilevitz. Learning boxes in high dimension. Algorithmica, 22(1/2):76–90, 1998.MATHCrossRefMathSciNetGoogle Scholar
  5. [BBK97]
    S. Ben-David, N. H. Bshouty, E. Kushilevitz. A composition theorem for learning algorithms with applications to geometric concept classes. 29th STOC, 1997.Google Scholar
  6. [BCV96]
    F. Bergadano, D. Catalano, S. Varricchio. Learning sat-k-DNF formulas from membership queries. In 28th STOC, pp 126–130, 1996.Google Scholar
  7. [BD92]
    P. Berman, B. DasGupta. Approximating the rectilinear polygon cover problems. In Proceedings of the 4th Canadian Conference on Computational Geometry, pages 229–235, 1992.Google Scholar
  8. [Bs95]
    N. H. Bshouty, Exact learning of boolean functions via the monotone theory. Information and Computation, 123, pp. 146–153, 1995.MATHCrossRefMathSciNetGoogle Scholar
  9. [B95a]
    N. H. Bshouty. Simple learning algorithms using divide and conquer. In Proceedings of the Annual ACM Workshop on Computational Learning Theory, 1995.Google Scholar
  10. [B98]
    N. H. Bshouty. A new composition theorem for learning algorithms. Proceedings of the 30th annual ACM Symposium on Theory of Computing (STOC), May 1998.Google Scholar
  11. [BCG96]
    N. H. Bshouty, R. Cleve, R. Gavalda, S. Kannan, C. Tamon. Oracles and queries that are sufficient for exact learning. JCSS 52(3): pp. 421–433, 1996.MATHMathSciNetGoogle Scholar
  12. [BGGM99]
    N. H. Bshouty, P. W. Goldberg, S. A. Goldman, D. H. Mathias. Exact learning of discretized geometric concepts. SIAM J. of Comput. 28(2), 678–699, 1999.MathSciNetGoogle Scholar
  13. [CH96]
    Z. Chen and W. Homer. The bounded injury priority method and the learnability of unions of rectangles. Annals of Pure and Applied Logic 77(2):143–168, 1996.MATHCrossRefMathSciNetGoogle Scholar
  14. [CM94]
    Z. Chen, W. Maass. On-line learning of rectangles and unions of rectangles, Machine Learning, 17(1/2), pages 201–223, 1994.MATHGoogle Scholar
  15. [DK91]
    V. J. Dielissen, A. Kaldewaij. Rectangular partition is polynomial in two dimensions but NP-complete in three. In IPL, pages 1–6, 1991.Google Scholar
  16. [Fr89]
    D. Franzblau. Performance guarantees on a sweep-line heuristic for covering rectilinear polygons with rectangles. SIAM J. on Discrete Math 2, 307–321, 1989.MATHCrossRefMathSciNetGoogle Scholar
  17. [HPRW96]
    L. Hellerstein, K. Pillaipakkamnatt, V. Raghavan, D. Wilkins. How many queries are needed to learn? JACM, 43(5), pages 840–862, 1996.MATHCrossRefMathSciNetGoogle Scholar
  18. [HR02]
    L. Hellerstein and V. Raghavan. Exact learning of DNF formulas using DNF hypotheses. In 34rd Annual Symposium on Theory of Computing (STOC), 2002.Google Scholar
  19. [KS01]
    A. Klivans and R. Servedio. Learning DNF in Time 2O(n1/3). In 33rd Annual Symposium on Theory of Computing (STOC), 2001, pp. 258–265.Google Scholar
  20. [LLLMP79]
    W. Lipski, Jr., E. Lodi, F. Luccio, C. Mugnai and L. Pagli. On two-dimensional data organization II. In Fund. Inform. 2, pages 245–260, 1979.MathSciNetMATHGoogle Scholar
  21. [MW98]
    W. Maass and M. K. Warmuth Efficient learning with virtual threshold gates Information and Computation, 141(1): 66–83, 1998.MATHCrossRefMathSciNetGoogle Scholar
  22. [PR96]
    K. Pillaipakkamntt, V. Raghavan, On the limits of proper learnability of subclasses of DNF formulas, Machine Learning, 25(2/3), pages 237–263, 1996.CrossRefGoogle Scholar
  23. [PV88]
    L. Pitt and L. G. Valiant. Computational Limitations on Learning from Examples. Journal of the ACM, 35(4):965–984, October 1988.Google Scholar
  24. [SS93]
    R. E. Schapire, L. M. Sellie. Learning sparse multivariate polynomial over a field with queries and counterexamples. In Proceedings of the Sixth Annual ACM Workshop on Computational Learning Theory. July, 1993.Google Scholar
  25. [U99]
    C. Umans. Hardness of approximating Σ2 p minimization problems. In Proceedings of the 40th Symposium on Foundations of Computer Science, 1999.Google Scholar
  26. [Val84]
    L. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, November 1984.Google Scholar
  27. [ZB00]
    H. Zantema, H. Bodlaender. Finding small equivalent decision trees is hard. In International Journal of Foundations of Computer Science, 11(2): 343–354, 2000.CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Nader H. Bshouty
    • 1
  • Lynn Burroughs
    • 2
  1. 1.Department of Computer ScienceTechnionHaifaIsrael
  2. 2.Department of Computer ScienceUniversity of CalgaryCalgaryCanada

Personalised recommendations