Abstract
Semi-supervised learning, a system dedicated to making networks less dependent on labeled data, has become a popular paradigm due to its strong performance. A common approach is to use pseudo-labels with unlabeled data for training, however, pseudo-labels cannot correct their own errors. In this paper, we propose a semi-supervised method that uses nearest neighbor samples to obtain pseudo-labels and combines consistency regularization for image classification. Our method obtains pseudo-labels by computing the similarity of the data distribution between the weakly-augmented version of the unlabeled data and the labeled data stored in the support set and combines the consistency of the strongly-augmented version and the weakly-augmented version of the unlabeled data. We compared with several standard semi-supervised learning benchmarks and achieved a competitive performance. For example, we achieved an accuracy of \(94.02\%\) on CIFAR-10 with 250 labels and \(97.50\%\) on SVNH with 250 labels. It even achieved \(91.59\%\) accuracy with only 40 labels data in the CIFAR-10.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arazo, E., Ortego, D., Albert, P., O’Connor, N.E., McGuinness, K.: Pseudo-labeling and confirmation bias in deep semi-supervised learning. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2020)
Berthelot, D., et al.: Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 (2019)
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.A.: Mixmatch: A holistic approach to semi-supervised learning. In: Advances in Neural Information Processing Systems 32 (2019)
Chapelle, O., Scholkopf, B., Zien, A.: Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks 20(3), 542–542 (2009)
Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 (2018)
Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702–703 (2020)
Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.: What makes paris look like paris? ACM Trans. Graph. 31(4) (2012)
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., Zisserman, A.: With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9588–9597 (2021)
Fan, H., Zhang, F., Gao, Y.: Self-supervised time series representation learning by inter-intra relational reasoning. arXiv preprint arXiv:2011.13548 (2020)
Fan, H., Zhang, F., Wang, R., Huang, X., Li, Z.: Semi-supervised time series classification by temporal relation prediction. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3545–3549. IEEE (2021)
Han, T., Xie, W., Zisserman, A.: Self-supervised co-training for video representation learning (2020)
Hays, J., Efros, A.A.: Scene completion using millions of photographs. ACM Trans. Graph. (ToG) 26(3), 4-es (2007)
Hays, J., Efros, A.A.: Im2gps: estimating geographic information from a single image. In: 2008 IEEE Conference On Computer Vision And Pattern Recognition, pp. 1–8. IEEE (2008)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016)
Lee, D.H., et al.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges In Representation Learning, ICML, vol. 3, p. 896 (2013)
Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
McClosky, D., Charniak, E., Johnson, M.: Effective self-training for parsing. In: Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics, pp. 152–159. Citeseer (2006)
Miyato, T., Maeda, S.I., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 41(8), 1979–1993 (2018)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning (2011)
Oliver, A., Odena, A., Raffel, C.A., Cubuk, E.D., Goodfellow, I.: Realistic evaluation of deep semi-supervised learning algorithms. In: Advances in Neural Information Processing Systems 31 (2018)
Pham, H., Le, Q.V.: Semi-supervised learning by coaching (2020). https://openreview.net/forum?id=rJe04p4YDB
Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964)
Rizve, M.N., Duarte, K., Rawat, Y.S., Shah, M.: In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329 (2021)
Rosenberg, C., Hebert, M., Schneiderman, H.: Semi-supervised self-training of object detection models (2005)
Sajjadi, M., Javanmardi, M., Tasdizen, T.: Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In: Advances in Neural Information Processing Systems 29 (2016)
Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning, pp. 1139–1147. PMLR (2013)
Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: Advances in Neural Information Processing Systems 30 (2017)
Thewlis, J., Albanie, S., Bilen, H., Vedaldi, A.: Unsupervised learning of landmarks by descriptor vector exchange. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6361–6371 (2019)
Wang, R., Wu, Y., Chen, H., Wang, L., Meng, D.: Neighbor matching for semi-supervised learning. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 439–449. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_41
Wu, J., Fan, H., Zhang, X., Lin, S., Li, Z.: Semi-supervised semantic segmentation via entropy minimization. In: 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2021)
Xie, Q., Dai, Z., Hovy, E., Luong, T., Le, Q.: Unsupervised data augmentation for consistency training. Adv. Neural. Inf. Process. Syst. 33, 6256–6268 (2020)
Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with noisy student improves imagenet classification. In: Proceedings of the IEEE/CVF Conference On Computer Vision And Pattern Recognition, pp. 10687–10698 (2020)
Yarowsky, D.: Unsupervised word sense disambiguation rivaling supervised methods. In: 33rd Annual Meeting of the Association For Computational Linguistics, pp. 189–196 (1995)
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
Zhang, B., et al.: Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In: Advances in Neural Information Processing Systems 34 (2021)
Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
Zou, Y., Yu, Z., Vijaya Kumar, B.V.K., Wang, J.: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 297–313. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_18
Acknowledgements
This work is partially supported by National Natural Science Foundation of China (61972187), Natural Science Foundation of Fujian Province (2020J02024).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, G. et al. (2023). Semi-supervised Learning with Nearest-Neighbor Label and Consistency Regularization. In: Xu, Y., Yan, H., Teng, H., Cai, J., Li, J. (eds) Machine Learning for Cyber Security. ML4CS 2022. Lecture Notes in Computer Science, vol 13656. Springer, Cham. https://doi.org/10.1007/978-3-031-20099-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-20099-1_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20098-4
Online ISBN: 978-3-031-20099-1
eBook Packages: Computer ScienceComputer Science (R0)