Abstract
Neural architecture search (NAS) techniques can discover outstanding network architecture while saving tremendous labor from human experts. Recent advancements further reduce the computational overhead to an affordable level. However, it is still cumbersome to deploy NAS in real-world applications due to the fussy procedures and the supervised learning paradigm. In this work, we propose the self-supervised and weight-preserving neural architecture search (SSWP-NAS) as an extension of the current NAS framework to allow the self-supervision and retain the concomitant weights discovered during the search stage. As such, we merge the process of architecture search and weight pre-training, and simplify the workflow of NAS to a one-stage and proxy-free procedure. The searched architectures can achieve state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets without using manual labels. Moreover, experiments demonstrate that using the concomitant weights as initialization consistently outperforms the random initialization and a separate weight pre-training process by a clear margin under semi-supervised learning scenarios. Codes are available at https://github.com/LzVv123456/SSWP-NAS.
Z. Li and Y. Gao—Equal contributions.
Y. Gao and Z. Zha—This work was done during the internship at SenseTime.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anandalingam, G., Friesz, T.L.: Hierarchical optimization: an introduction. Ann. Oper. Res. 34(1), 1–11 (1992)
Cai, H., Zhu, L., Han, S.: ProxylessNAS: direct neural architecture search on target task and hardware. In: International Conference on Learning Representations (2019)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. (2020) arXiv preprint arXiv:2002.05709
Chen, X., Xie, L., Wu, J., Tian, Q.: Progressive differentiable architecture search: bridging the depth gap between search and evaluation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1294–1303 (2019)
Chen, X., He, K.: Exploring simple siamese representation learning. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745–15753 (2021)
Colson, B., Marcotte, P., Savard, G.: An overview of bilevel optimization (2007).https://doi.org/10.1007/s10479-007-0176-2
Doersch, C., Gupta, A.K., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1422–1430 (2015)
Ghiasi, G., Lin, T.Y., Pang, R., Le, Q.V.: NAS-FPN: Learning scalable feature pyramid architecture for object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7029–7038 (2019)
Grill, J.B., et al.: Bootstrap your own latent - a new approach to self-supervised learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. Curran Associates, Inc. 33, pp. 21271–21284 (2020)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.B.: Momentum contrast for unsupervised visual representation learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726–9735 (2020)
He, K., Girshick, R.B., Dollár, P.: Rethinking imagenet pre-training. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4917–4926 (2019)
Howard, A.G., et al.: Searching for mobilenetv3. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1314–1324 (2019)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708 (2017)
Kaplan, S., Giryes, R.: Self-Supervised neural architecture search. CoRR abs/2007.01500 (2020)
Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep. (2009)
Li, J., Zhou, P., Xiong, C., Hoi, S.: Prototypical contrastive learning of unsupervised representations. In: International Conference on Learning Representations (2021)
Liang, H., et al.: DARTS+: improved differentiable architecture search with early stopping. CoRR abs/1909.06035 (2019)
Liu, C., et al.: Auto-DeepLab: Hierarchical neural architecture search for semantic image segmentation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 82–92 (2019)
Liu, C., Doll’ar, P., He, K., Girshick, R.B., Yuille, A.L., Xie, S.: Are labels necessary for neural architecture search? In: ECCV (2020)
Liu, C., et al.: Progressive neural architecture search. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 19–34 (2018)
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. In: International Conference on Learning Representations (2018)
Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. In: International Conference on Learning Representations (2019)
Luo, R., Tian, F., Qin, T., Chen, E., Liu, T.Y.: Neural architecture optimization. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7827–7838 (2018)
Ma, N., Zhang, X., Zheng, H.T., Sun, J.: ShuffleNet v2: practical guidelines for efficient CNN architecture design. In: Proceedings of the European Conference on Computer Vision (ECCV) September 2018
Nguyen, N., Chang, J.M.: Contrastive self-supervised neural architecture search. CoRR abs/2102.10557 (2021)
Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV (2016)
van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. ArXiv abs/1807.03748 (2018)
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 4095–4104. PMLR 10–15 Jul 2018
Raghu, M., Zhang, C., Kleinberg, J., Bengio, S.: Transfusion: understanding transfer learning for medical imaging. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 4780–4789 (2019)
Ruder, S.: An overview of gradient descent optimization algorithms. CoRR abs/1609.04747 (2016)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Shu, Y., Wang, W., Cai, S.: Understanding architectures learnt by cell-based neural architecture search. In: International Conference on Learning Representations (2020)
Sohn, K.: Improved deep metric learning with multi-class n-pair loss objective. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 29. Curran Associates, Inc. (2016)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) June 2016
Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding (2020)
Wang, N., et al.: NAS-FCOS: Fast neural architecture search for object detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) June 2020
Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733–3742 (2018)
Xie, S., Zheng, H., Liu, C., Lin, L.: SNAS: stochastic neural architecture search. In: International Conference on Learning Representations (2019)
Xu, Y., et al.: Pc-darts: Partial channel connections for memory-efficient architecture search. In: International Conference on Learning Representations (2020)
Ying, C., Klein, A., Christiansen, E., Real, E., 0002, K.M., Hutter, F.: NAS-Bench-101: Towards Reproducible Neural Architecture Search. In: Proceedings of the 36th International Conference on Machine Learning, pp. 7105–7114. PMLR (2019)
Zhang, R., Isola, P., Efros, A.A.: Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 645–654 (2017)
Zhang, X., Hou, P., Zhang, X., Sun, J.: Neural architecture search with random labels. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10902–10911 (2021)
Zhang, Y., Qiu, Z., Liu, J., Yao, T., Liu, D., Mei, T.: Customizable architecture search for semantic segmentation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11633–11642 (2019)
Zhou, H., Yang, M., Wang, J., Pan, W.: BayesNAS: A Bayesian Approach for Neural Architecture Search. In: Proceedings of the 36th International Conference on Machine Learning, pp. 7603–7613. PMLR (2019)
Zhuang, C., Zhai, A., Yamins, D.: Local aggregation for unsupervised learning of visual embeddings. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6001–6011 (2019)
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. (2017) ArXiv abs/1611.01578
Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697–8710 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Z. et al. (2023). Towards Self-Supervised and Weight-preserving Neural Architecture Search. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13804. Springer, Cham. https://doi.org/10.1007/978-3-031-25069-9_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-25069-9_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25068-2
Online ISBN: 978-3-031-25069-9
eBook Packages: Computer ScienceComputer Science (R0)