Abstract
Predictor-based Neural Architecture Search (NAS) offers a promising solution for enhancing the efficiency of traditional NAS methods. However, it is non-trivial to train the predictor with limited architecture evaluations for efficient NAS. While current approaches typically focus on better utilizing the labeled architectures, the valuable knowledge contained in unlabeled data remains unexplored. In this paper, we propose a self-supervised transformer-based model that effectively leverages unlabeled data to learn meaningful representations of neural architectures, reducing the reliance on labeled data to train a high-performance predictor. Specifically, the predictor is pre-trained with a masking strategy to reconstruct input features in both latent and raw data spaces. To further enhance its representative capability, we introduce a multi-head attention-masking mechanism that guides the model to attend to different representation subspaces from both explicit and implicit perspectives. Extensive experimental results on NAS-Bench-101, NAS-Bench-201 and NAS-Bench-301 demonstrate that our predictor requires less labeled data and achieves superior performance compared to existing predictors. Furthermore, when combined with search strategies, our predictor exhibits promising capability in discovering high-quality architectures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Dong, X., Yang, Y.: Nas-bench-201: extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 (2020)
Dudziak, L., Chau, T., Abdelfattah, M., Lee, R., Kim, H., Lane, N.: Brp-nas: prediction-based nas using gcns. Adv. Neural. Inf. Process. Syst. 33, 10480–10490 (2020)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16000–16009 (2022)
Hou, Z., et al.: Graphmae2: a decoding-enhanced masked self-supervised graph learner. In: Proceedings of the ACM Web Conference 2023, pp. 737–746 (2023)
Howard, A., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)
Jing, K., Xu, J., Li, P.: Graph masked autoencoder enhanced predictor for neural architecture search. In: Thirty-First International Joint Conference on Artificial Intelligence, vol. 4, pp. 3114–3120 (2022)
Li, C., et al.: Hw-nas-bench: Hardware-aware neural architecture search benchmark. arXiv preprint arXiv:2103.10584 (2021)
Liu, C., et al.: Progressive neural architecture search. In: Proceedings of the European conference on computer vision (ECCV), pp. 19–34 (2018)
Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)
Lu, S., Li, J., Tan, J., Yang, S., Liu, J.: Tnasp: a transformer-based nas predictor with a self-evolution framework. Adv. Neural. Inf. Process. Syst. 34, 15125–15137 (2021)
Luo, R., Tian, F., Qin, T., Chen, E., Liu, T.Y.: Neural architecture optimization. In: Advances in Neural Information Processing Systems 31 (2018)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4780–4789 (2019)
Siems, J., Zimmer, L., Zela, A., Lukasik, J., Keuper, M., Hutter, F.: Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09777 (2020)
Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
Tan, Q., et al.: S2gae: self-supervised graph autoencoders are generalizable learners with graph masking. In: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 787–795 (2023)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Wang, C., Gong, L., Li, X., Zhou, X.: A ubiquitous machine learning accelerator with automatic parallelization on fpga. IEEE Trans. Parallel Distrib. Syst. 31(10), 2346–2359 (2020)
Wen, W., Liu, H., Chen, Y., Li, H., Bender, G., Kindermans, P.J.: Neural predictor for neural architecture search. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX, pp. 660–676. Springer (2020)
White, C., Neiswanger, W., Savani, Y.: Bananas: Bayesian optimization with neural architectures for neural architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 10293–10301 (2021)
Xu, Y., et al.: Renas: relativistic evaluation of neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4411–4420 (2021)
Yan, S., Song, K., Liu, F., Zhang, M.: Cate: computation-aware neural architecture encoding with transformers. In: International Conference on Machine Learning, pp. 11670–11681. PMLR (2021)
Ying, C., et al.: Do transformers really perform badly for graph representation? Adv. Neural. Inf. Process. Syst. 34, 28877–28888 (2021)
Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: Nas-bench-101: towards reproducible neural architecture search. In: International Conference on Machine Learning, pp. 7105–7114. PMLR (2019)
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)
Acknowledgments
This work was supported in part by the National Key R &D Program of China under Grants 2022YFB4501603, in part by the National Natural Science Foundation of China under Grants 62102383, 61976200, and 62172380, in part by Jiangsu Provincial Natural Science Foundation under Grant BK20210123, in part by Youth Innovation Promotion Association CAS under Grant Y2021121, and in part by the USTC Research Funds of the Double First-Class Initiative under Grant YD2150002005.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liang, Q., Gong, L., Wang, C., Zhou, X., Li, X. (2024). NeuralMAE: Data-Efficient Neural Architecture Predictor with Masked Autoencoder. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14432. Springer, Singapore. https://doi.org/10.1007/978-981-99-8543-2_12
Download citation
DOI: https://doi.org/10.1007/978-981-99-8543-2_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8542-5
Online ISBN: 978-981-99-8543-2
eBook Packages: Computer ScienceComputer Science (R0)