Skip to main content
Book cover

International Conference on Intelligent Data Engineering and Automated Learning

IDEAL 2020: Intelligent Data Engineering and Automated Learning – IDEAL 2020 pp 335–347Cite as

Meta-hyperband: Hyperparameter Optimization with Meta-learning and Coarse-to-Fine

  • 991 Accesses

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12490)

Abstract

Hyperparameter optimization is one of the main pillars of machine learning algorithms. In this paper, we introduce Meta-Hyperband: a Hyperband based algorithm that improves the hyperparameter optimization by adding levels of exploitation. Unlike Hyperband method, which is a pure exploration bandit-based approach for hyperparameter optimization, our meta approach generates a trade-off between exploration and exploitation by combining the Hyperband method with meta-learning and Coarse-to-Fine modules. We analyze the performance of Meta-Hyperband on various datasets to tune the hyperparameters of CNN and SVM. The experiments indicate that in many cases Meta-Hyperband can discover hyperparameter configurations with higher quality than Hyperband, using similar amounts of resources. In particular, we discovered a CNN configuration for classifying CIFAR10 dataset which has a 3% higher performance than the configuration founded by Hyperband, and is also 0.3% more accurate than the best-reported configuration of the Bayesian optimization approach. Additionally, we release a publicly available pool of historically well-performed configurations on several datasets for CNN and SVM to ease the adoption of Meta-Hyperband.

Keywords

  • Hyperparameter optim.
  • Meta-learning
  • Coarse-to-Fine

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-62365-4_32
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-62365-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   139.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.

Notes

  1. 1.

    Our sources \(\rightarrow \) https://github.com/saminpayro/Meta_Hyperband_implementation.

  2. 2.

    See “example layers” directory in http://code.google.com/p/cuda-convnet/.

  3. 3.

    https://code.google.com/archive/p/cuda-convnet2/.

References

  1. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)

    MathSciNet  MATH  Google Scholar 

  2. Bergstra, J., Rémi B., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Neural Information Processing Systems, pp. 2546–2554 (2011)

    Google Scholar 

  3. Charniak, E., Johnson, M.: Coarse-to-fine n-best parsing and maxent discriminative reranking. In: Annual Meeting on Association for Computational Linguistics, pp. 173–180. ACL 2005 (2005)

    Google Scholar 

  4. Deng, L., Li, X.: Machine learning paradigms for speech recognition: an overview. Trans. Audio Speech Lang. Process. 21(5), 1060–1089 (2013)

    CrossRef  Google Scholar 

  5. Feurer, M., Springenberg, J.T., Hutter, F.: Initializing bayesian hyperparameter optimization via meta-learning. In: AAAI Conference on Artificial Intelligence (2015)

    Google Scholar 

  6. Guo, G., Li, S.Z., Chan, K.: Face recognition by support vector machines. In: International Conference on Automatic Face and Gesture Recognition (2000)

    Google Scholar 

  7. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40

    CrossRef  Google Scholar 

  8. Jamieson, P., Talwalkar, A.: Non-stochastic best arm identification and hyperparameter optimization. AISTATS (2015)

    Google Scholar 

  9. Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto (2009)

    Google Scholar 

  10. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    CrossRef  Google Scholar 

  11. Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: a novel bandit-based approach to hyperparameter optimization. In: ICLR (2017)

    Google Scholar 

  12. Moshkelgosha, V., Behzadi-Khormouji, H., Yazdian-Dehkordi, M.: Coarse-to-fine parameter tuning for content-based object categorization. In: International Conference on Pattern Recognition and Image Analysis (IPRIA), pp. 160–165. IEEE (2017)

    Google Scholar 

  13. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, YA.: Reading digits in natural images with unsupervised feature learning. In: NIPS (2011)

    Google Scholar 

  14. Rejani, Y.I.A., Selvi, S.T.: Early detection of breast cancer using SVM classifier technique. CoRR abs/0912.2314 (2009)

    Google Scholar 

  15. Rolland, P., Scarlett, J., Bogunovic, I., Cevher, V.: High-dimensional bayesian optimization via additive models with overlapping groups. AISTATS (2018)

    Google Scholar 

  16. Sadeghi, A., Graux, D., Yazdi, H.S., Lehmann, J.: MDE: multi distance embeddings for link prediction in knowledge graphs. In: 24th European Conference on Artificial Intelligence (ECAI) (2020)

    Google Scholar 

  17. Snoek, J., Larochelle, H., Adams, R.: Practical bayesian optimization of machine learning algorithms. In: Neural Information Processing Systems (NIPS) (2012)

    Google Scholar 

  18. Vilalta, R., Drissi, Y.: A perspective view and survey of meta-learning. Artif. Intell. Rev. 18(2), 77–95 (2002)

    CrossRef  Google Scholar 

  19. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

Download references

Acknowledgments

This study is supported by MLwin (Maschinelles Lernen mit Wissensgraphen, GA 01IS18050F of the German Federal Ministry of Education and Research), by the EU project Cleopatra (GA 812997) and by the Marie Skłodowska-Curie GA 801522 at the ADAPT SFI Research Centre (grant 13/RC/2106).

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Bonn, Bonn, Germany

    Samin Payrosangari, Afshin Sadeghi & Jens Lehmann

  2. Fraunhofer IAIS, Sankt Augustin, Germany

    Afshin Sadeghi & Jens Lehmann

  3. ADAPT SFI Research Centre, Trinity College Dublin, Dublin, Ireland

    Damien Graux

Authors
  1. Samin Payrosangari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Afshin Sadeghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Damien Graux
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jens Lehmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Afshin Sadeghi .

Editor information

Editors and Affiliations

  1. University of Minho, Braga, Portugal

    Cesar Analide

  2. University of Minho, Braga, Portugal

    Dr. Paulo Novais

  3. Technical University of Madrid, Madrid, Spain

    David Camacho

  4. University of Manchester, Manchester, UK

    Dr. Hujun Yin

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Payrosangari, S., Sadeghi, A., Graux, D., Lehmann, J. (2020). Meta-hyperband: Hyperparameter Optimization with Meta-learning and Coarse-to-Fine. In: Analide, C., Novais, P., Camacho, D., Yin, H. (eds) Intelligent Data Engineering and Automated Learning – IDEAL 2020. IDEAL 2020. Lecture Notes in Computer Science(), vol 12490. Springer, Cham. https://doi.org/10.1007/978-3-030-62365-4_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62365-4_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62364-7

  • Online ISBN: 978-3-030-62365-4

  • eBook Packages: Computer ScienceComputer Science (R0)