Abstract
We study heuristic learnability of classes of Boolean formulas, a model proposed by Pitt and Valiant. In this type of example-based learning of a concept class C by a hypothesis class H, the learner seeks a hypothesis h∈ H that agrees with all of the negative (resp. positive) examples, and a maximum number of positive (resp. negative) examples. This learning is equivalent to the problem of maximizing agreement with a training sample, with the constraint that the misclassifications be limited to examples with positive (resp. negative) labels. Several recent papers have studied the more general problem of maximizing agreements without this one-sided error constraint. We show that for many classes (though not all), the maximum agreement problem with one-sided error is more difficult than the general maximum agreement problem. We then provide lower bounds on the approximability of these one-sided error problems, for many concept classes, including Halfspaces, Decision Lists, XOR, k-term DNF, and neural nets.
Article PDF
Similar content being viewed by others
References
Amaldi, E., & Kann, V. (1995). The complexity and approximability of finding maximum feasible subsystems of linear relations. Theoretical Computer Science, 147:1/2, 181–210.
Angluin, D., & Laird, P. D. (1987). Learning from noisy examples. Machine Learning, 2:4, 343–370.
Bartlett, P. L., & Ben-David, S. (1999). Hardness results for neural network approximation problems. In Proceedings of the 4th European Conference on Computational Learning Theory (pp. 50–62).
Ben-David, S., Eiron, N., & Long, P. M. (2003). On the difficulty of approximately maximizing agreements. Journal of Computer and System Sciences, 66:3, 496–514.
Blum, A., Furst, M., Jackson, J., Kearns, M., Mansour, Y., & Rudich, S. (1994). Weakly learning DNF and characterizing statistical query learning using Fourier analysis. In Proceedings of the 26th Annual ACM Symposium on Theory of Computing (pp. 253–262).
Blum, A. L., & Rivest, R. L. (1988). Training a 3-node neural network is NP-complete. In Proceedings of the 1988 Workshop on Computational Learning Theory (pp. 9–18).
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K. (1987). Occam’s razor. Information Processing Letters, 24:6, 377–380.
Bshouty, N. H., & Burroughs, L. (2002a). Bounds for the minimum disagreement problem with applications to learning theory. In Proceedings of the 15th Annual Conference on Computational Learning Theory (pp. 271–286).
Bshouty, N. H., & Burroughs, L. (2002b). Maximizing agreements and coagnostic learning. In Proceedings of the 13th International Conference on Algorithmic Learning Theory.
Håstad, J. (1996). Clique is hard to approximate within n1−ε. In Proceedings of the 37th Annual IEEE Symposium on Foundations of Computer Science (pp. 627–636).
Håstad, J. (1997). Some optimal inapproximability results. In Proceedings of the 29th Annual ACM Symposium on Theory of Computing (pp. 1–10).
Höffgen, K.-U., Simon, H.-U., & Van Horn, K. S. (1995). Robust trainability of single neurons, JCSS, 50:1, 114–125.
Itoh, T. (2000). Approximating the maximum weight of linear codes is APX-complete. On Fundamentals of Electronics, Communications and Computer Sciences, E83-A:4, 606–613.
Kann, V., Khanna, S., Lagergren, J., & Panconesi, A. (1996). On the hardness of approximating max-k-cut and its dual. In Proceedings of the Fourth Israeli Symposium on Theory of Computing and Systems (pp. 61–67).
Kearns, M., & Li, M. (1993). Learning in the presence of malicious errors. SIAM Journal on Computing, 22:4, 807–837.
Kuhlmann, C. (2000). Hardness results for general two-layer neural networks. In Proceedings of the 13th Annual Conference on Computational Learning Theory (pp. 275–285).
Panconesi, A., & Ranjan, D. (1993). Quantifiers and approximation. Theoretical Computer Science, 107:1, 145–163.
Papadimitriou, C., & Yannakakis, M. (1991). Optimization, approximation and complexity classes. Journal of Computer and System Sciences, 43, 425–440.
Pitt, L., & Valiant, L. G. (1988). Computational limitations on learning from examples. JACM, 35:4, 965–984.
Valiant, L. G. (1984). A theory of the learnable. Communications of the ACM, 27:11, 1134–1142.
Author information
Authors and Affiliations
Corresponding author
Additional information
Editor
Philip M. Long
This research was supported by the fund for promotion of research at the Technion. Research no. 120-025.
Rights and permissions
About this article
Cite this article
Bshouty, N.H., Burroughs, L. Maximizing Agreements with One-Sided Error with Applications to Heuristic Learning. Mach Learn 59, 99–123 (2005). https://doi.org/10.1007/s10994-005-0464-5
Received:
Revised:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s10994-005-0464-5