Abstract
We consider the problem of learning DNF formulae in the mistake-bound and the PAC models. We develop a new approach, which is called polynomial explainability, that is shown to be useful for learning some new subclasses of DNF (and CNF) formulae that were not known to be learnable before. Unlike previous learnability results for DNF (and CNF) formulae, these subclasses are not limited in the number of terms or in the number of variables per term; yet, they contain the subclasses of κ-DNF and κ-term-DNF (and the corresponding classes of CNF) as special cases. We apply our DNF results to the problem of learning visual concepts and obtain learning algorithms for several natural subclasses of visual concepts that appear to have no natural boolean counterpart. On the other hand, we show that learning some other natural subclasses of visual concepts is as hard as learning the class of all DNF formulae. We also consider the robustness of these results under various types of noise.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Aizenstein, H. & L. Pitt. (1991). Exact learning of read-twice DNF formulas. In Proceedings of the IEEE Symp. on Foundation of Computer Science, number 32, pages 170–179, San Juan.
Aizenstein, H. & L. Pitt. (1992). Exact learning of read-k disjoint DNF and not-so-disjoint DNF. In Proceedings of COLT '92, pages 71–76.
Angluin, D. (1980). Finding patterns common to a set of strings. Journal of Computer and System Sciences, 21(1):46–62.
Angluin, D., M. Frazier, & L. Pitt. (1992). Learning conjunctions of Horn clauses. Machine Learning, 9:147–164.
Angluin, D. & P. Laird. (1988). Learning from noisy examples. Machine Learning, 2(4):343–370.
Aslam, J. A. & S. E. Decatur. (1993). General bounds on statistical query learning and PAC learning with noise via hypothesis boosting. In Proceedings of the 34th Annual Symposium on Foundations of Computer Science, pages 282–291.
Basri, R. (1994). Private Communication.
Bender, M. & D. Roth. (1994). Learning human motion as DNF formulae. (Unpublished).
Blum, A. (1992). Learning boolean functions in an infinite attribute space. Machine Learning, 9(4):373–386.
Blum, A., R. Khardon, A. Kushilevitz, L. Pitt, & D. Roth. (1994). On learning read-k satisfy-j DNF. In Proceedings of the Annual ACM Workshop on Computational Learning Theory, pages 110–117. (Submitted for publication)
Blum, A. & S. Rudich. (1992). Fast learning of k-term DNF formulas with queries. In Proceedings of Twenty-Fourth ACM Symposium on Theory of Computing, pages 382–389.
Blumer, A., A. Ehrenfeucht, D. Haussler, & M. K. Warmuth. (1987). Occam's razor. Information Processing Letters, 24:377–380.
Bshouty, N. H. (1993). Exact learning via the monotone theory. In Proceedings of the IEEE Symp. on Foundation of Computer Science, pages 302–311, Palo Alto, CA.
Decatur, S. E. (1993). Statistical queries and faulty PAC oracles. In Proceedings of the Sixth Annual ACM Workshop on Computational Learning Theory, pages 262–268.
Hancock, T. (1991). Learning 21 DNF formulas and k1 decision trees. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, pages 199–209.
Haussler, D., M. Kearns, N. Littlestone, & M. K. Warmuth. (1991). Equivalence of models for polynomial learnability. Information and Computation, 95(2):129–161.
Jackson, J. (1994). An efficient membership-query algorithm for learning DNF with respect to the uniform distribution. In Proceedings of the IEEE Symp. on Foundation of Computer Science. To Appear.
Jerrum, M. (1991). Simple translation-invariant concepts are hard to learn. Technical Report CSR–12–91, University of Edinburgh, Department of Computer Science.
Kearns, M. (1993). Efficient noise-tolerant learning from statistical queries. In Proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computing, pages 392–401.
Kearns, M. & M. Li. (1993). Learning in the precence of malicious error. Siam Journal of Computing, 22(4).
Kearns, M., M. Li, L. Pitt, & L. G. Valiant. (1987). On the learnability of boolean formulae. In Proceedings of the Nineteenth Annual ACM Symposium on Theory of Computing, pages 285–295.
Kearns, M. & L. Pitt. (1989). A polynomial-time algorithm for learning k-variable pattern languages from examples. In Proceedings of the Second Annual Workshop on Computational Learning Theory, pages 57–71.
Kushilevitz, E. & Y. Mansour. (1993). Learning decision trees using the fourier spectrum. Siam Journal of Computing, 22(6):1331–1348. Earlier version appeared in Proc. 23rd Ann. IEEE Symp. on Foundations of Computer Science, 1991.
Li, M. & P. M. B. Vitanyi. (1989). A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proceedings of the Thirtieth Annual Symposium on Foundations of Computer Science, pages 34–39.
Littlestone, N. (1988). Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2:285–318.
Littlestone, N. (1989). Mistake bounds and logarithmic linear-threshold learning algorithms. Ph.D. thesis, U. C. Santa Cruz.
Mitchell, T.M., R.M. Keller, & S.T. Kedar-Cabelli. (1986). Explanation Based Learning. Machine Learning, 1(1):47–80.
Pillapakkamnatt, K. & V. Raghavan. (1993). Read twice DNF formulas are properly learnable. Technical Report TR-CS–93–59, Vanderbilt University, Computer Science Department. To appear, Proceedings of the 1st European Conference on Computational Learning Theory (EuroColt 93).
Rivest, R. L. (1987). Learning decision lists. Machine Learning, 2(3):229–246.
Schapire, R. E. (1990). Pattern languages are not learnable. In Proceedings of COLT '90, pages 122–129.
Shackelford, G. & D. Volper. (1988). Learning k-DNF with noise in the attributes. In First Workshop on Computatinal Learning Theory, pages 97–103.
Shvaytser, H. (1990). Learnable and nonlearnable visual concepts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(5):459–466.
Valiant, L. G. (1984). A theory of the learnable. Communications of the ACM, 27(11):1134–1142.
Valiant, L. G. (1985). Learning disjunctions of conjunctions. In Proceedings of the International Joint Conference
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Kushilevitz, E., Roth, D. On Learning Visual Concepts and DNF Formulae. Machine Learning 24, 65–85 (1996). https://doi.org/10.1023/A:1018098129371
Issue Date:
DOI: https://doi.org/10.1023/A:1018098129371