Advertisement

Using Neural Architecture Search to Optimize Neural Networks for Embedded Devices

  • Thomas CassimonEmail author
  • Simon Vanneste
  • Stig Bosmans
  • Siegfried Mercelis
  • Peter Hellinckx
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 96)

Abstract

Recent advances in the field of Neural Architecture Search (NAS) have made it possible to develop state-of-the-art deep learning systems without requiring extensive human expertise and hyperparameter tuning. In most previous research, little concern was given to the resources required to run the generated systems. In this paper, we present an improvement on a recent NAS method, Efficient Neural Architecture Search (ENAS). We adapt ENAS to not only take into account the network’s performance, but also various constraints that would allow these networks to be ported to embedded devices. Our results show ENAS’ ability to comply with these added constraints. In order to show the efficacy of our system, we demonstrate it by designing a Recurrent Neural Network (RNN) that predicts words as they are spoken, and meets the constraints set out for operation on an embedded device.

References

  1. 1.
    Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning (2017). https://arxiv.org/abs/1611.01578
  2. 2.
    Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992)zbMATHGoogle Scholar
  3. 3.
    Pham, H., Guan, M., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameters sharing (2018). http://proceedings.mlr.press/v80/pham18a/pham18a.pdf
  4. 4.
    Elsken, T., Metzen, J.H., Hutter, F.: Efficient multi-objective neural architecture search via lamarckian evolution. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=ByME42AqK7
  5. 5.
    Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. CoRR, vol. abs/1707.07012 (2017). http://arxiv.org/abs/1707.07012
  6. 6.
    Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. CoRR, vol. abs/1802.01548 (2018). http://arxiv.org/abs/1802.01548
  7. 7.
    Kandasamy, K., Schneider, J., Pczos, B., Xing, E.P.: Neural architecture search with Bayesian optimisation and optimal transport. In: Conference in Neural Information Processing Systems (2018). https://papers.nips.cc/paper/7472-neural-architecture-search-with-bayesian-optimisation-and-optimal-transport.pdf
  8. 8.
    Horowitz, M.: 1.1 computing’s energy problem (and what we can do about it). In: 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10–14, February 2014Google Scholar
  9. 9.
    Wong, B.L.: Essential Study Skills, 8th edn. Cengage, Boston (2015). ISBN 9781285430096Google Scholar
  10. 10.
    Inan, H., Khosravi, K., Socher, R.: Tying word vectors and word classifiers: a loss framework for language modeling. CoRR, vol. abs/1611.01462 (2016). http://arxiv.org/abs/1611.01462
  11. 11.
    Zilly, J.G., Srivastava, R.K., Koutník, J., Schmidhuber, J.: Recurrent highway networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, PMLR, 06–11 Augus 2017, vol. 70, pp. 4189–4198. International Convention Centre, Sydney (2017). http://proceedings.mlr.press/v70/zilly17a.html
  12. 12.
    Marcus, M., Kim, G., Marcinkiewicz, M.A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., Schasberger, B.: The penn treebank: annotating predicate argument structure. In: Proceedings of the Workshop on Human Language Technology, HLT 1994, pp. 114–119. Association for Computational Linguistics, Stroudsburg (1994).  https://doi.org/10.3115/1075812.1075835
  13. 13.
    Li, L., Talwalkar, A.: Random search and reproducibility for neural architecture search. CoRR, vol. abs/1902.07638 (2019). http://arxiv.org/abs/1902.07638
  14. 14.
    Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)
  15. 15.
    Van Moffaert, K., Drugan, M.M., Nowé, A.: Hypervolume-based multi-objective reinforcement learning. In: Purshouse, R.C., Fleming, P.J., Fonseca, C.M., Greco, S., Shaw, J. (eds.) Evolutionary Multi-Criterion Optimization, pp. 352–366. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  16. 16.
    Van Moffaert, K., Drugan, M.M., Now, A.: Scalarized multi-objective reinforcement learning: novel design techniques. In: 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 191–199, April 2013Google Scholar
  17. 17.
    Cai, H., Zheng, V.W., Chang, K.C.-C.: A comprehensive survey of graph embedding: problems, techniques and applications. IEEE Trans. Knowl. Data Eng. 30, 1616–1637 (2017)CrossRefGoogle Scholar
  18. 18.
    Perozzi, B., Al-Rfou, R., Skiena, S.: DeepWalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2014, pp. 701–710. ACM, New York (2014).  https://doi.org/10.1145/2623330.2623732

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Thomas Cassimon
    • 1
    Email author
  • Simon Vanneste
    • 1
  • Stig Bosmans
    • 1
  • Siegfried Mercelis
    • 1
  • Peter Hellinckx
    • 1
  1. 1.IDLab - Faculty of Applied EngineeringUniversity of Antwerp - imecAntwerpBelgium

Personalised recommendations