Generalization Error Bounds Using Unlabeled Data
We present two new methods for obtaining generalization error bounds in a semi-supervised setting. Both methods are based on approximating the disagreement probability of pairs of classifiers using unlabeled data. The first method works in the realizable case. It suggests how the ERM principle can be refined using unlabeled data and has provable optimality guarantees when the number of unlabeled examples is large. Furthermore, the technique extends easily to cover active learning. A downside is that the method is of little use in practice due to its limitation to the realizable case.
The idea in our second method is to use unlabeled data to transform bounds for randomized classifiers into bounds for simpler deterministic classifiers. As a concrete example of how the general method works in practice, we apply it to a bound based on cross-validation. The result is a semi-supervised bound for classifiers learned based on all the labeled data. The bound is easy to implement and apply and should be tight whenever cross-validation makes sense. Applying the bound to SVMs on the MNIST benchmark data set gives results that suggest that the bound may be tight enough to be useful in practice.
Unable to display preview. Download preview PDF.
- 1.Madani, O., Pennock, D.M., Flake, G.W.: Co-validation: Using model disagreement to validate classification algorithms. In: NIPS 2004 Preproceedings (2004)Google Scholar
- 2.Balcan, M.F., Blum, A.: A PAC-style model for learning from labeled and unlabeled data, Draft (2004)Google Scholar
- 10.Kääriäinen, M.: Relating the Rademacher and VC bounds. Technical Report Report C-2004-57, Department of Computer Science, Series of Publications C (2004)Google Scholar
- 13.Cesa-Bianchi, N., Gentile, C.: Improved risk tail bounds for on-line algorithms A presentation in the (Ab)use of Bounds workshop (2004)Google Scholar
- 14.Blum, A., Kalai, A., Langford, J.: Beating the hold-out: bounds for k-fold and progressive cross-validation. In: Proceedings of the 12th Annual Conference on Computational Learning Theory, New York, NY, pp. 203–208. ACM Press, New York (1999)Google Scholar
- 16.Langford, J.: Practical prediction theory for classification (2003), A tutorial presented at ICML 2003., Available at http://hunch.net/jl/projects/prediction_bounds/tutorial/tutorial.pdf
- 17.Kutin, S., Niyogi, P.: Almost-everywhere algorithmic stability and generalization error. In: Proceedings of Uncertainty in AI, pp. 275–282 (2002)Google Scholar
- 18.Joachims, T.: Making large-scale SVM learning practical. In: Schölkopf, B., Burges, C., Smola, A. (eds.) Advances in Kernel Methods – Support Vector Learning, MIT Press, Cambridge (1999)Google Scholar
- 19.Seeger, M.: Bayesian Gaussian Process Models: PAC-Bayesian Generalisation Error Bounds and Sparse Approximations. PhD thesis, University of Edinburgh (2003)Google Scholar