Abstract
In neural architecture search, differentiable architecture search algorithm has become one of the mainstream methods. However, no matter in the search or evaluation stage, the architecture is repeatedly stacked by two kinds of Cells, namely Normal Cell and Reduction Cell, respectively. This undoubtedly limits the performance of the evaluation architecture to a large extent due to the architecture restriction, resulting in sub-optimal performance. In order to alleviate the impact of architecture restriction on network performance, this paper proposes to post-optimize the architecture searched by differentiable architecture search algorithms by freezing the architecture parameters of partial Cells and further searching other Cells to bring more diversity into the stacked Cells. The proposed post-optimizing methods consist of the global post-optimizing search method and the local post-optimizing search method, respectively. The performance of the evaluation architecture can benefit from the diverse stacked Cells with less architecture restriction. In the experiments, the proposed post-optimizing method is applied to the mainstream differentiable architecture search algorithms such as DARTS and P-DARTS, and superior results have been achieved on CIFAR-10 and CIFAR-100 datasets. Moreover, the proposed method can obtain the post-optimized architecture with limited computing resources.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: Scaled-yolov4: scaling cross stage partial network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13029–13038 (2021)
Dong, L., Xu, B.: Cif: continuous integrate-and-fire for end-to-end speech recognition. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6079–6083, IEEE (2020)
Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. In: 7th International Conference on Learning Representations. (2019)
Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: The Thirty-Third AAAI Conference on Artificial Intelligence, pp. 4780–4789 (2019)
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. In: 5th International Conference on Learning Representations (2017)
Chen, X., Xie, L., Wu, J., Tian, Q.: Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In: 2019 IEEE/CVF International Conference on Computer Vision, pp. 1294–1303 (2019)
Xu, Y., et al.: PC-DARTS: partial channel connections for memory efficient architecture search. In: 8th International Conference on Learning Representations (2020)
Bi, K., Xie, L., Chen, X., Wei, L., Tian, Q., “Gold-nas: gradual, one-level, differentiable”. arXiv preprint arXiv:2007.03331 (2020)
Chu, X., Zhou, T., Zhang, B., Li, J.: Fair DARTS: eliminating unfair advantages in differentiable architecture search. Comput. Vis. ECCV 12360, 465–480 (2020)
Chu, X., Zhang, B.: Noisy differentiable architecture search. In: 32nd British Machine Vision Conference, p. 217 (2021)
Liang, H., et al.: Darts+: improved differentiable architecture search with early stopping. arXiv preprint arXiv:1909.06035 (2019)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2261–2269 (2017)
Liu, C., Zoph, B., Neumann, M., Shlens, J.: Progressive neural architecture search. Comput. Vis. ECCV 11205, 19–35 (2018)
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameter sharing. In: Proceedings of the 35th International Conference on Machine Learning, vol. 80, pp. 4092–4101 (2018)
Zela, A., Elsken, T., Saikia, T., Marrakchi, Y., Brox, T., Hutter, F.: Understanding and robustifying differentiable architecture search. In: 8th International Conference on Learning Representations (2020)
Chen, X., Hsieh, C.J.: Stabilizing differentiable architecture search via perturbation-based regularization. In: Proceedings of the 37th International Conference on Machine Learning, vol. 119, pp. 1554–1565 (2020)
Chu, X., Wang, X., Zhang, B., Lu, S., Wei, X., Yan, J.: DARTS-: robustly stepping out of performance collapse without indicators. In: 9th International Conference on Learning Representations (2021)
Ye, P., Li, B., Li, Y., Chen, T., Fan, J., Ouyang, W.: β-darts: Beta-decay regularization for differentiable architecture search. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10864–10873 (2022)
Acknowledgement
This work was supported in part by National Natural Science Foundation of China (NSFC) under Grant No.61772061.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hao, D., Pei, S. (2023). PO-DARTS: Post-optimizing the Architectures Searched by Differentiable Architecture Search Algorithms. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds) Artificial Neural Networks and Machine Learning – ICANN 2023. ICANN 2023. Lecture Notes in Computer Science, vol 14262. Springer, Cham. https://doi.org/10.1007/978-3-031-44201-8_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-44201-8_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44200-1
Online ISBN: 978-3-031-44201-8
eBook Packages: Computer ScienceComputer Science (R0)