Abstract
Although Differentiable Architecture Search (DARTS) has achieved promising performance in many machine learning tasks, it still suffers from a problem during searching: due to those different operations in candidate set may need different levels of optimization, directly handling them with the same training scheme will make DARTS in favor of networks with fast convergence, resulting in a performance drop correspondingly. This problem will become more serious at the later searching stages. In this paper, we propose an adaptive dropout method for DARTS (AD-DARTS), which zeros the output of each operation with a probability according to structure parameters which can be considered as the variable representing the difficulty-level training such a candidate operation, thus serving to balance the training procedures for different operations. The operations with more parameters can be trained more adequately to strengthen the characterization ability of the network. Our analysis further shows that the proposed AD-DARTS are also with high search stability. The proposed method effectively solves the aforementioned problem and can achieve better performance compared with other baselines based on DRATS on CIFAR-10, CIFAR-100, and ImageNet.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cai, H., Zhu, L., Han, S.: Proxylessnas: direct neural architecture search on target task and hardware. In: International Conference on Learning Representations (ICLR) (2019)
Chen, X., Xie, L., Wu, J., Tian, Q.: Progressive differentiable architecture search: bridging the depth gap between search and evaluation. In: International Conference on Computer Vision (ICCV), pp. 1294–1303 (2019)
Chu, X., Wang, X., Zhang, B., Lu, S., Wei, X., Yan, J.: Darts-: robustly stepping out of performance collapse without indicators. arXiv preprint arXiv:2009.01027 (2020)
Chu, X., Zhang, B., Li, X.: Noisy differentiable architecture search. arXiv preprint arXiv:2005.03566 (2020)
Chu, X., Zhou, T., Zhang, B., Li, J.: Fair DARTS: eliminating unfair advantages in differentiable architecture search. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12360, pp. 465–480. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58555-6_28
Dong, X., Yang, Y.: Searching for a robust neural architecture in four GPU hours. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1761–1770 (2019)
Franceschi, L., Frasconi, P., Salzo, S., Grazzi, R., Pontil, M.: Bilevel programming for hyperparameter optimization and meta-learning. In: International Conference on Machine Learning (ICML), pp. 1568–1577. PMLR (2018)
Gong, X., Chang, S., Jiang, Y., Wang, Z.: Autogan: neural architecture search for generative adversarial networks. In: International Conference on Computer Vision (ICCV), pp. 3224–3234 (2019)
Hong, W., Li, G., Zhang, W., Tang, R., Wang, Y., Li, Z., Yu, Y.: Dropnas: grouped operation dropout for differentiable architecture search. In: International Joint Conference on Artificial Intelligence (IJCAI) (2020)
Liang, H., et al.: Darts+: Improved differentiable architecture search with early stopping. arXiv preprint arXiv:1909.06035 (2019)
Liu, C., et al.: Progressive neural architecture search. In: European Conference on Computer Vision (ECCV), pp. 19–34 (2018)
Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. In: International Conference on Learning Representations (ICLR) (2018)
Liu, H., Simonyan, K., Yang, Y.: Darts: differentiable architecture search. In: International Conference on Learning Representations (ICLR) (2019)
Maclaurin, D., Duvenaud, D., Adams, R.: Gradient-based hyperparameter optimization through reversible learning. In: International Conference on Machine Learning (ICML), pp. 2113–2122. PMLR (2015)
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: International Conference on Machine Learning (ICML), pp. 4095–4104. PMLR (2018)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: AAAI conference on Artificial Intelligence (AAAI), pp. 4780–4789 (2019)
Stamoulis, D., Ding, R., Wang, D., Lymberopoulos, D., Priyantha, B., Liu, J., Marculescu, D.: Single-Path NAS: designing hardware-efficient ConvNets in less than 4 hours. In: Brefeld, U., Fromont, E., Hotho, A., Knobbe, A., Maathuis, M., Robardet, C. (eds.) ECML PKDD 2019. LNCS (LNAI), vol. 11907, pp. 481–497. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46147-8_29
Veniat, T., Denoyer, L.: Learning time/memory-efficient deep architectures with budgeted super networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3492–3500 (2018)
Xie, S., Zheng, H., Liu, C., Lin, L.: Snas: stochastic neural architecture search. In: International Conference on Learning Representations (ICLR) (2019)
Xu, Y., et al.: Pc-darts: partial channel connections for memory-efficient architecture search. In: International Conference on Learning Representations (ICLR) (2020)
Zhong, Z., Yan, J., Wu, W., Shao, J., Liu, C.L.: Practical block-wise neural network architecture generation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2423–2432 (2018)
Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8697–8710 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, Z., Yang, L., Wang, L., Li, F. (2021). AD-DARTS: Adaptive Dropout for Differentiable Architecture Search. In: Fang, L., Chen, Y., Zhai, G., Wang, J., Wang, R., Dong, W. (eds) Artificial Intelligence. CICAI 2021. Lecture Notes in Computer Science(), vol 13070. Springer, Cham. https://doi.org/10.1007/978-3-030-93049-3_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-93049-3_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93048-6
Online ISBN: 978-3-030-93049-3
eBook Packages: Computer ScienceComputer Science (R0)