Abstract
We introduce a novel semi-supervised version of the least squares classifier. This implicitly constrained least squares (ICLS) classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-supervised methods, our approach does not introduce explicit additional assumptions into the objective function, but leverages implicit assumptions already present in the choice of the supervised least squares classifier. We show this approach can be formulated as a quadratic programming problem and its solution can be found using a simple gradient descent procedure. We prove that, in a certain way, our method never leads to performance worse than the supervised classifier. Experimental results corroborate this theoretical result in the multidimensional case on benchmark datasets, also in terms of the errorĀ rate.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bache, K., Lichman, M.: UCI Machine Learning Repository (2013). http://archive.ics.uci.edu/ml
Bennett, K.P., Demiriz, A.: Semi-supervised support vector machines. Adv. Neural Inf. Process. Syst. 11, 368ā374 (1998)
Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Lechevallier, Y., Saporta, G. (eds.) COMPSTAT 2010, pp. 177ā186. Springer, Heidelberg (2010)
Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. 16(5), 1190ā1208 (1995)
Chapelle, O., Schƶlkopf, B., Zien, A.: Semi-Supervised Learning. MIT press, Cambridge (2006)
Cozman, F., Cohen, I.: Risks of semi-supervised learning. In: Chapelle, O., Schƶlkopf, B., Zien, A. (eds.) Semi-Supervised Learning, Chap. 4, pp. 56ā72. MIT press (2006)
Cozman, F.G., Cohen, I., Cirelo, M.C.: Semi-supervised learning of mixture models. In: Proceedings of the Twentieth International Conference on Machine Learning (2003)
Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical Learning. Spinger, New York (2001)
Krijthe, J.H., Loog, M.: Implicitly constrained semi-supervised linear discriminant analysis. In: International Conference on Pattern Recognition, pp. 3762ā3767, Stockholm (2014)
Li, Y.F., Zhou, Z.H.: Towards making unlabeled data never hurt. IEEE Trans. Pattern Anal. Mach. Intell. 37(1), 175ā188 (2015)
Loog, M., Jensen, A.: Semi-supervised nearest mean classification through a constrained log-likelihood. IEEE Trans. Neural Networks Learn. Syst. 26(5), 995ā1006 (2015)
Loog, M.: Semi-supervised linear discriminant analysis through moment-constraint parameter estimation. Pattern Recognit. Lett. 37, 24ā31 (2014)
McLachlan, G.J.: Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. J. Am. Stat. Assoc. 70(350), 365ā369 (1975)
Nigam, K., McCallum, A.K., Thrun, S., Mitchell, T.: Text classification from labeled and unlabeled documents using EM. Mach. Learn. 34, 1ā34 (2000)
Opper, M., Kinzel, W.: Statistical mechanics of generalization. In: Domany, E., Hemmen, J.L., Schulten, K. (eds.) Models of Neural Networks III, pp. 151ā209. Springer, New York (1996)
Poggio, T., Smale, S.: The mathematics of learning: dealing with data. Not. AMS 50, 537ā544 (2003)
Raudys, S., Duin, R.P.: Expected classification error of the fisher linear classifier with pseudo-inverse covariance matrix. Pattern Recogn. Lett. 19(5ā6), 385ā392 (1998)
Rifkin, R., Yeo, G., Poggio, T.: Regularized least-squares classification. Nato Sci. Ser. Sub Ser. III Comput. Syst. Sci. 190, 131ā154 (2003)
Seeger, M.: Learning with labeled and unlabeled data. Technical report (2001)
Singh, A., Nowak, R.D., Zhu, X.: Unlabeled data: now it helps, now it doesnt. In: Advances in Neural Information Processing Systems, pp. 1513ā1520 (2008)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soci. Ser. B 58(1), 267ā288 (1996)
Widrow, B., Hoff, M.E.: Adaptive switching circuits. IRE WESCON Convention Rec. 4, 96ā104 (1960)
Zhu, X., Goldberg, A.B.: Introduction to Semi-Supervised Learning, vol. 3. Morgan & Claypool, San Rafael (2009)
Acknowledgments
Part of this work was funded by project P23 of the Dutch public-private research community COMMIT.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
Ā© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Krijthe, J.H., Loog, M. (2015). Implicitly Constrained Semi-supervised Least Squares Classification. In: Fromont, E., De Bie, T., van Leeuwen, M. (eds) Advances in Intelligent Data Analysis XIV. IDA 2015. Lecture Notes in Computer Science(), vol 9385. Springer, Cham. https://doi.org/10.1007/978-3-319-24465-5_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-24465-5_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-24464-8
Online ISBN: 978-3-319-24465-5
eBook Packages: Computer ScienceComputer Science (R0)