Skip to main content

Advertisement

SpringerLink
Log in

Hyperparameter optimization of neural networks based on Q-learning

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Machine learning algorithms are sensitive to hyperparameters, and hyperparameter optimization techniques are often computationally expensive, especially for complex deep neural networks. In this paper, we use Q-learning algorithm to search for good hyperparameter configurations for neural networks, where the learning agent searches for the optimal hyperparameter configuration by continuously updating the Q-table to optimize hyperparameter tuning strategy. We modify the initial states and termination conditions of Q-learning to improve search efficiency. The experimental results on hyperparameter optimization of a convolutional neural network and a bidirectional long short-term memory network show that our method has higher search efficiency compared with tree of Parzen estimators, random search and genetic algorithm and can find out the optimal or near-optimal hyperparameter configuration of neural network models with minimum number of trials.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

All accompanying data are provided in the manuscript.

References

  1. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)

  2. Reddy, A.H., Kolli, K., Kiran, Y.L.: Deep cross feature adaptive network for facial emotion classification. SIViP 16(2), 369–376 (2022)

    Article  Google Scholar 

  3. Redmon, J., Divvala, S., Girshick, R., et al.: You only look once: Unified, real-time object detection. In: CVPR (2016)

  4. Hutter, F., Kotthoff, L., Vanschoren, J.: Automated Machine Learning: Methods, Systems, Challenges. Springer Nature, Berlin (2019)

    Book  Google Scholar 

  5. Abreu, S.: Automated architecture design for deep neural networks. arXiv preprint arXiv:1908.10714. (2019)

  6. Luo, G.: A review of automatic selection methods for machine learning algorithms and hyper-parameter values. Netw. Model. Anal. Health Inform. Bioinform. 5(1), 1–16 (2016)

    Article  Google Scholar 

  7. Kohavi, R., John, G.H.: Automatic parameter selection by minimizing estimated error. In: Kohavi, R., John, G.H.: Machine Learning Proceedings 1995, pp. 304–312. Elsevier (1995)

  8. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(2) (2012)

  9. Hinz, T., Navarro-Guerrero, N., Magg, S., et al.: Speeding up the hyperparameter optimization of deep convolutional neural networks. Int. J. Comput. Intell. Appl. 17(02), 1850008 (2018)

    Article  Google Scholar 

  10. Klein, A., Falkner, S., Bartels, S., et al.: Fast Bayesian optimization of machine learning hyperparameters on large datasets. In: AISTATS (2017)

  11. Swersky, K., Snoek, J., Adams, R.P.: Multi-task bayesian optimization. In: NIPS (2013)

  12. Bergstra, J., Bardenet, R., Bengio, Y., et al.: Algorithms for hyper-parameter optimization. In: NIPS (2011)

  13. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: LION (2011)

  14. Xie, L., Yuille, A.: Genetic cnn. In: ICCV (2017)

  15. Miikkulainen, R., Liang, J., Meyerson, E., et al.: Evolving deep neural networks. In: Miikkulainen, R., Liang, J., Meyerson, E., et al. (eds) Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293–312. Elsevier (2019)

  16. Xiao, X., Yan, M., Basodi, S., et al.: Efficient hyperparameter optimization in deep learning using a variable length genetic algorithm. arXiv preprint arXiv:2006.12703 (2020)

  17. Lorenzo, P.R., Nalepa, J., Kawulok, M., et al.: Particle swarm optimization for hyper-parameter selection in deep neural networks. In: GECCO (2017)

  18. Guo, Y., Li, J.-Y., Zhan, Z.-H.: Efficient hyperparameter optimization for convolution neural networks in deep learning: A distributed particle swarm optimization approach. Cybern. Syst. 52(1), 36–57 (2020)

    Article  Google Scholar 

  19. Li, L., Jamieson, K., DeSalvo, G., et al.: Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18(1), 6765–6816 (2017)

    MATH  MathSciNet  Google Scholar 

  20. Falkner, S., Klein, A., Hutter, F.: BOHB: Robust and efficient hyperparameter optimization at scale. In: ICML (2018)

  21. Awad, N., Mallik, N., Hutter, F.: DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization. arXiv preprint arXiv:2105.09821 (2021)

  22. Misra, U., Liaw, R., Dunlap, L., et al. RubberBand: cloud-based hyperparameter tuning. In: Proceedings of the Sixteenth European Conference on Computer Systems (2021)

  23. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578. (2016)

  24. Baker, B., Gupta, O., Naik, N., et al.: Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167. (2016)

  25. Zhong, Z., Yan, J., Wu, W., et al.: Practical block-wise neural network architecture generation. In: CVPR (2018)

  26. Dong, X., Shen, J., Wang, W., et al.: Dynamical hyperparameter optimization via deep reinforcement learning in tracking. IEEE Trans. Pattern Anal. Mach. Intell. 43(5), 1515–1529 (2019)

  27. Wu, J., Chen, S., Liu, X.: Efficient hyperparameter optimization through model-based reinforcement learning. Neurocomputing 409, 381–393 (2020)

    Article  Google Scholar 

  28. Jomaa, H.S., Grabocka, J., Schmidt-Thieme, L.: Hyp-rl: Hyperparameter optimization by reinforcement learning. arXiv preprint arXiv:1906.11527 (2019)

  29. Chen, S., Wu, J., Liu, X.: EMORL: Effective multi-objective reinforcement learning method for hyperparameter optimization. Eng. Appl. Artif. Intell. 104, 104315 (2021)

    Article  Google Scholar 

  30. Liu, X., Wu, J., Chen, S.: A context-based meta-reinforcement learning approach to efficient hyperparameter optimization. Neurocomputing 478, 89–103 (2022)

    Article  Google Scholar 

  31. Howard, R.A.: Dynamic programming and markov processes (1960)

  32. LeCun, Y., Boser, B., Denker, J.S., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)

    Article  Google Scholar 

  33. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009)

  34. Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18(5–6), 602–610 (2005)

    Article  Google Scholar 

  35. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  36. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)

    MATH  Google Scholar 

Download references

Funding

Received no external funding.

Author information

Authors and Affiliations

  1. Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, Hung Hom, Hong Kong SAR, China

    Xin Qi & Bing Xu

Authors
  1. Xin Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bing Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization was contributed by X.Q. and B.X.; Methodology and experiments were contributed by X.Q.; Formal analysis were contributed by X.Q. and B.X.; Writing of the original draft was contributed by X.Q.; Review and editing were contributed by B.X.; This work was completed under the supervision of B.X.

Corresponding author

Correspondence to Bing Xu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qi, X., Xu, B. Hyperparameter optimization of neural networks based on Q-learning. SIViP 17, 1669–1676 (2023). https://doi.org/10.1007/s11760-022-02377-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02377-y

Keywords

Search

Navigation