True Rank Guided Efficient Neural Architecture Search for End to End Low-Complexity Network Discovery

Abstract

Neural architecture search (NAS) aims to automate neural network design process and has shown promising results for image classification tasks. Owing to combinatorially huge neural network design spaces coupled with training cost of candidates, NAS is computationally demanding. Hence, many NAS works focus on efficiency by constraining the search to only network building blocks (modular search) instead of searching for the entire architectures (global search), and by approximating candidates’ performance instead of expensive training. Modular search, however, offers only partial network discovery and final architecture configuration such as network’s depth or width requires manual trial and error. Further, approximating candidates’ performance incur misleading search directions due to their inaccurate relative rankings. In this work, we revisit NAS for end to end network discovery and guide the search using true rankings of candidates by training each from scratch. However, it is computationally infeasible for existing search strategies to navigate huge search spaces and determine accurate rankings at the same time. Therefore, we propose a novel search space and an efficient search algorithm that offers high accuracy low complexity network discovery with competitive search cost. Our proposed approach is evaluated on the CIFAR-10, yielding an error rate of 4% while the search cost is just 4.5 GPU days. Moreover, our model is 7.3$\times$, 3.7$\times$ and 5.5$\times$ smaller than the smallest models discovered by RL, evolutionary and gradient-based NAS methods respectively (Code and results available at: https://github.com/siddikui/TRG-NAS).

Christos Kyrkou would like to acknowledge the support of NVIDIA with the donation of GPU platform.