Abstract
Semi-supervised learning (SSL) has achieved new progress recently with the emerging framework of self-training deep networks, where the criteria for selection of unlabeled samples with pseudo labels play a key role in the empirical success. In this work, we propose such a new criterion based on consistency among multiple, stochastic classifiers, termed Stochastic Consensus (STOCO). Specifically, we model parameters of the classifiers as a Gaussian distribution whose mean and standard deviation are jointly optimized during training. Due to the scarcity of labels in SSL, modeling classifiers as a distribution itself provides additional regularization that mitigates overfitting to the labeled samples. We technically generate pseudo labels using a simple but flexible framework of deep discriminative clustering, which benefits from the overall structure of data distribution. We also provide theoretical analysis of our criterion by connecting with the theory of learning from noisy data. Our proposed criterion can be readily applied to self-training based SSL frameworks. By choosing the representative FixMatch as the baseline, our method with multiple stochastic classifiers achieves the state of the art on popular SSL benchmarks, especially in label-scarce cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Angluin, D., Laird, P.: Learning from noisy examples. Mach. Learn. 2, 343–370 (1988)
Assran, M., et al.: Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In: ICCV, pp. 8443–8452 (2021)
Balcan, M.F., Blum, A., Yang, K.: Co-training and expansion: towards bridging theory and practice. In: NeurIPS, pp. 89–96 (2004)
Berthelot, D., et al.: Remixmatch: semi-supervised learning with distribution matching and augmentation anchoring. In: ICLR (2020)
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.A.: Mixmatch: a holistic approach to semi-supervised learning. In: NeurIPS, vol. 32 (2019)
Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: COLT, pp. 92–100 (1998)
d’Alché Buc, F., Grandvalet, Y., Ambroise, C.: Semi-supervised marginboost. In: NeurIPS, pp. 553–560 (2001)
Chen, M., Weinberger, K.Q., Blitzer, J.: Co-training for domain adaptation. In: NeurIPS, vol. 24 (2011)
Chen, Z., Zhuang, J., Liang, X., Lin, L.: Blending-target domain adaptation by adversarial meta-adaptation networks. In: CVPR, pp. 2243–2252 (2019)
Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 215–223 (2011)
Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.: Randaugment: practical automated data augmentation with a reduced search space. In: NeurIPS, vol. 33, pp. 18613–18624 (2020)
Dietterich, T.G.: Ensemble methods in machine learning. In: MCS, pp. 1–15 (2000)
Dizaji, K.G., Herandi, A., Deng, C., Cai, W., Huang, H.: Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In: ICCV, pp. 5747–5756 (2017)
van Engelen, J.E., Hoos, H.H.: A survey on semi-supervised learning. Mach. Learn. 109(2), 373–440 (2019). https://doi.org/10.1007/s10994-019-05855-6
Fei-Fei, L., Fergus, R., Perona, P.: One-shot learning of object categories. IEEE TPAMI 28, 594–611 (2006)
French, G., Mackiewicz, M., Fisher, M.: Self-ensembling for visual domain adaptation. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=rkpoTaxA-
Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: Balcan, M.F., Weinberger, K.Q. (eds.) Proceedings of International Conference Machine Learning. Proceedings of Machine Learning Research, 20–22 June 2016, vol. 48, pp. 1050–1059. PMLR, New York (2016). https://proceedings.mlr.press/v48/gal16.html
Grandvalet, Y., Bengio, Y.: Semi-supervised learning by entropy minimization. In: NeurIPS, pp. 529–536 (2004)
Guo, L.Z., Zhang, Z.Y., Jiang, Y., Li, Y.F., Zhou, Z.H.: Safe deep semi-supervised learning for unseen-class unlabeled data. In: III, H.D., Singh, A. (eds.) Proceedings of International Conference on Machine Learning. Proceedings of Machine Learning Research, 13–18 July 2020, vol. 119, pp. 3897–3906. PMLR (2020)
Jabi, M., Pedersoli, M., Mitiche, A., Ayed, I.B.: Deep clustering: on the link between discriminative models and k-means. IEEE TPAMI 43, 1887–1896 (2021)
Karim, M.R., et al.: Deep learning-based clustering approaches for bioinformatics. Brief. Bioinf. 22, 393–415 (2020)
Kingma, D., Welling, M.: Auto-encoding variational bayes. In: ICLR (2014)
Krizhevsky, A.: Learning multiple layers of features from tiny images. In: Technical report (2009)
Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. In: ICLR (2016)
Lee, D.H.: Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. In: Proceedings of International Conference on Machine Learning Workshop (2013)
Li, J., Xiong, C., Hoi, S.C.: Comatch: semi-supervised learning with contrastive graph regularization. In: ICCV, pp. 9475–9484 (2021)
Li, W., Foo, C., Bilen, H.: Learning to impute: a general framework for semi-supervised learning. CoRR abs/1912.10364 (2019). http://arxiv.org/abs/1912.10364
Liang, J., Yang, J., Lee, H.-Y., Wang, K., Yang, M.-H.: Sub-GAN: an unsupervised generative model via subspaces. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 726–743. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_43
Lu, Z., Yang, Y., Zhu, X., Liu, C., Song, Y.Z., Xiang, T.: Stochastic classifiers for unsupervised domain adaptation. In: CVPR, pp. 9108–9117 (2020)
Luo, Y., Zhu, J., Li, M., Ren, Y., Zhang, B.: Smooth neighbors on teacher graphs for semi-supervised learning. In: CVPR, pp. 8896–8905 (2018)
van der Maaten, L., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9, 2579–2605 (2008)
McLachlan, G.J.: Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. J. Am. Stat. Assoc. 70, 365–369 (1975)
Miyato, T., Maeda, S.I., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE TPAMI 41, 1979–1993 (2019)
Mousavi, S.M., Zhu, W., Ellsworth, W., Beroza, G.: Unsupervised clustering of seismic signals using deep convolutional autoencoders. IEEE Geosci. Remote Sens. Lett. 16, 1693–1697 (2019)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: Workshop of Proceedings of Neural Information Processing System (2011)
Pham, H., Dai, Z., Xie, Q., Le, Q.V.: Meta pseudo labels. In: CVPR, pp. 11557–11568 (2021)
Quinlan, J.R.: Miniboosting decision trees (1999)
Rasmus, A., Valpola, H., Honkala, M., Berglund, M., Raiko, T.: Semi-supervised learning with ladder networks. In: NeurIPS, pp. 3546–3554 (2015)
Rizve, M.N., Duarte, K., Rawat, Y.S., Shah, M.: In defense of pseudo-labeling: an uncertainty-aware pseudo-label selection framework for semi-supervised learning. In: ICLR (2021). https://openreview.net/forum?id=-ODN6SbiUU
Rosenberg, C., Hebert, M., Schneiderman, H.: Semi-supervised self-training of object detection models. In: Seventh IEEE Workshops on Applications of Computer Vision, vol. 1, pp. 29–36 (2005)
Saito, K., Ushiku, Y., Harada, T.: Asymmetric tri-training for unsupervised domain adaptation. In: Proceedings of International Conference Machine Learning, pp. 2988–2997 (2017)
Sajjadi, M., Javanmardi, M., Tasdizen, T.: Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In: NeurIPS, vol. 29 (2016)
Scudder, H.: Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Inf. Theory 11, 363–371 (1965)
Sohn, K., et al.: Fixmatch: simplifying semi-supervised learning with consistency and confidence. In: NeurIPS, vol. 33, pp. 596–608 (2020)
Tang, H., Chen, K., Jia, K.: Unsupervised domain adaptation via structurally regularized deep clustering. In: CVPR, pp. 8725–8735 (2020)
Tarvainen, A., Valpola, H.: Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: NeurIPS, vol. 30 (2017)
Wang, Y., Guo, J., Song, S., Huang, G.: Meta-semi: a meta-learning approach for semi-supervised learning. CoRR abs/2007.02394 (2020). https://arxiv.org/abs/2007.02394
Wei, C., Shen, K., Chen, Y., Ma, T.: Theoretical analysis of self-training with deep networks on unlabeled data. In: ICLR (2021)
Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: Proceedings of International Conference on Machine Learning, pp. 478–487 (2016)
Xie, Q., Dai, Z., Hovy, E., Luong, T., Le, Q.: Unsupervised data augmentation for consistency training. In: NeurIPS, vol. 33, pp. 6256–6268 (2020)
Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with noisy student improves imagenet classification. In: CVPR, pp. 10687–10698 (2020)
Zagoruyko, S., Komodakis, N.: Wide residual networks. In: BMVC (2016)
Zhang, B., et al.: Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) NeurIPS (2021). https://openreview.net/forum?id=3qMwV98zLIk
Zhang, L., Qi, G.J.: Wcp: worst-case perturbations for semi-supervised deep learning. In: CVPR, pp. 3911–3920 (2020)
Zhou, Z.H., Li, M.: Tri-training: exploiting unlabeled data using three classifiers. IEEE Trans. Knowl. Data Eng. 17, 1529–1541 (2005)
Zou, Y., Yu, Z., Liu, X., Kumar, B.V.K.V., Wang, J.: Confidence regularized self-training. In: ICCV, pp. 5981–5990 (2019)
Zou, Y., Yu, Z., Vijaya Kumar, B.V.K., Wang, J.: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 297–313. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_18
Acknowledgments
This work is supported in part by Program for Guangdong Introducing Innovative and Enterpreneurial Teams (No.: 2017ZT07X183), National Natural Science Foundation of China (No.: 61771201), and Guangdong R &D key project of China (No.: 2019B010155001). Correspondence to Kui Jia (email: kuijia@scut.edu.cn).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tang, H., Sun, L., Jia, K. (2022). Stochastic Consensus: Enhancing Semi-Supervised Learning with Consistency of Stochastic Classifiers. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13691. Springer, Cham. https://doi.org/10.1007/978-3-031-19821-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-19821-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19820-5
Online ISBN: 978-3-031-19821-2
eBook Packages: Computer ScienceComputer Science (R0)