Skip to main content
SpringerLink
Log in

Automatic Optimization-Based Methods in Machine Learning: A Systematic Review

  • Conference paper
  • First Online:
Intelligent Systems and Applications (IntelliSys 2023)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 823))

Included in the following conference series:

  • 16 Accesses

Abstract

Machine Learning (ML) is a subfield of Artificial Intelligence (AI) that has been applied to various fields ranging from industrial to medical sectors, to perform miscellaneous Computer Vision tasks such as image classification, image segmentation, object detection, and language modeling. Notwithstanding, having a suitable model with practical applicability requires performing appropriate structural operations upon datasets, building adequate CNN architectures from the scratch or resorting to the ones available in the state-of-the-art, and, either way, parameterizing them to improve machine learning skills, usually, in a trial-and-error fashion. Aligned with this context, despite the several semi-/fully automatic approaches that can be found in the literature, (e.g., grid search for hyperparameter fine-tuning, auto-Machine Learning for self-configurable model development, and automatic methods for data arrangement and augmentation), which are often integrated with combination to establish automatic pipelines for the effective implementation of solutions powered by AI, surveys documenting such topic seem to be scarce. Therefore, the main goal of this work is to present an updated yet extensive literature review focusing on this class of approaches, considering the importance of their role in the perspective of ML Optimization.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Tuggener, L., et al.: Automated machine learning in practice: state of the art and recent results. In: 2019 6th Swiss Conference on Data Science (SDS), Bern, Switzerland, pp. 31–36. https://doi.org/10.1109/SDS.2019.00-11

  2. Hutter, F., Kotthoff, L., Vanschoren, J. (eds.): Automated Machine Learning: Methods, Systems, Challenges. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5

  3. Vaccaro, L., Sansonetti, G., and Micarelli, A.: An empirical review of automated machine learning. In: Computers, vol. 10, no. 1, Art. no. 1, Jan. 2021. https://doi.org/10.3390/computers10010011

  4. Bergstra, J., Bardenet, R., Bengio, Y., an d Kegl, B.: Algorithms for hyper-parameter optimization. In: 25th Annual Conference on Neural Information Processing Systems (NIPS 2011) (2011)

    Google Scholar 

  5. Baymurzina, D., Golikov, E., Burtsev, M.: A review of neural architecture search. Neurocomputing 474, 82–93 (2022). https://doi.org/10.1016/j.neucom.2021.12.014

    Article  Google Scholar 

  6. Jaafra, Y., Luc Laurent, J., Deruyver, A., Naceur, M.S.: Reinforcement learning for neural architecture search: a review. Image Vis. Comput. 89, 57–66 (2019). https://doi.org/10.1016/j.imavis.2019.06.005

  7. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 1–21 (2019)

    Google Scholar 

  8. Karl, F., et al.: Multi-objective hyperparameter optimization—an overview. arXiv:2206.07438. Accessed 13 Jan 2023

  9. Khalid, R., Javaid, N.: A survey on hyperparameters optimization algorithms of forecasting models in smart grid. ScienceDirect 61, 102275 (2020). https://doi.org/10.1016/j.scs.2020.102275

    Article  Google Scholar 

  10. Yu, T., Zhu, H.: Hyper-parameter optimization: a review of algorithms and applications. arXiv:2003.05689. Accessed 13 Jan 2023

  11. Yang, L., Shami, A.: On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing 415, 295–316 (2020). https://doi.org/10.1016/j.neucom.2020.07.061

    Article  Google Scholar 

  12. Bischl, B., et al.: Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges. arXiv:2107.05847. Accessed 13 Jan 2023

  13. Hudalizaman, Ardiyanto, I., Wibirama, S.: Network architecture search method on hyperparameter optimization of convolutional neural network: review. In: 2020 6th International Conference on Science and Technology (ICST), Yogyakarta, Indonesia, pp. 1–6 (2020). https://doi.org/10.1109/ICST50505.2020.9732800

  14. Dong, X., Kedziora, D.J., Musial, K., Gabrys, B.: Automated Deep Learning: Neural Architecture Search Is Not the End. arXiv:2112.09245. Accessed 13 Jan 2023

  15. Yang, S., Xiao, W., Zhang, M., Guo, S., Zhao, J., Shen, F.: Image Data Augmentation for Deep Learning: A Survey. arXiv:2204.08610. Accessed 13 Jan 2023

  16. Raileanu, R., Goldstein, M., Yarats, D., Kostrikov, I., Fergus, R.: Automatic Data Augmentation for Generalization in Reinforcement Learning

    Google Scholar 

  17. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G.: PRISMA group, ‘preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. (2009)

    Google Scholar 

  18. Pedregosa, F. et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 2825–2830

    Google Scholar 

  19. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 281–305 (2012)

    Google Scholar 

  20. Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., de Freitas, N.: Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104(1), 148–175 (2016). https://doi.org/10.1109/JPROC.2015.2494218

    Article  Google Scholar 

  21. Mitchell, M.: Genetic algorithms: an overview. Complexity 1(1), 31–39 (1995). https://doi.org/10.1002/cplx.6130010108

    Article  Google Scholar 

  22. Blume, S., Benedens, T., Schramm, D.: Hyperparameter optimization techniques for designing software sensors based on artificial neural networks. Sensors 21(24), 8435 (2021). https://doi.org/10.3390/s21248435

    Article  Google Scholar 

  23. Alibrahim, H., Ludwig, S.A.: Hyperparameter optimization: comparing genetic algorithm against grid search and Bayesian optimization. In: 2021 IEEE Congress on Evolutionary Computation (CEC), Kraków, Poland, pp. 1551–1559 (2021). https://doi.org/10.1109/CEC45853.2021.9504761

  24. Di Francescomarino, C., et al.: Genetic algorithms for hyperparameter optimization in predictive business process monitoring. Inf. Syst. 74, 67–83 (2018). https://doi.org/10.1016/j.is.2018.01.003

    Article  Google Scholar 

  25. Belete, D.M., Huchaiah, M.D.: Grid search in hyperparameter optimization of machine learning models for prediction of HIV/AIDS test results. Int. J. Comput. Appl. 44(9), 875–886 (2022). https://doi.org/10.1080/1206212X.2021.1974663

    Article  Google Scholar 

  26. Shekar, B.H., Dagnew, G.: Grid search-based hyperparameter tuning and classification of microarray cancer data. In: 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP), Gangtok, India, pp. 1–8 (2019). https://doi.org/10.1109/ICACCP.2019.8882943

  27. El-Hasnony, I.M., Elzeki, O.M., Alshehri, A., Salem, H.: Multi-label active learning-based machine learning model for heart disease prediction. Sensors 22(3), 1184 (2022). https://doi.org/10.3390/s22031184

    Article  Google Scholar 

  28. Qu, Z., Xu, J., Wang, Z., Chi, R., Liu, H.: Prediction of electricity generation from a combined cycle power plant based on a stacking ensemble and its hyperparameter optimization with a grid-search method. ScienceDirect 227, 120309 (2021). https://doi.org/10.1016/j.energy.2021.120309

    Article  Google Scholar 

  29. Sanchez, O.R., Repetto, M., Carrega, A., Bolla, R.: Evaluating ML-based DDoS detection with grid search hyperparameter optimization. In: 2021 IEEE 7th International Conference on Network Softwarization (NetSoft), Tokyo, Japan, pp. 402–408 (2021). https://doi.org/10.1109/NetSoft51509.2021.9492633

  30. Kim, C., Park, T.: Predicting determinants of lifelong learning intention using gradient boosting machine (GBM) with grid search. MDPI 14(9), 5256 (2022). https://doi.org/10.3390/su14095256

    Article  Google Scholar 

  31. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. Accessed 13 Jan 2023. http://arxiv.org/abs/1611.01578

  32. Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, pp. 8697–8710 (2018). https://doi.org/10.1109/CVPR.2018.00907

  33. Liu, C., et al.: Progressive neural architecture search. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018, vol. 11205, pp. 19–35. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_2

  34. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) Learning and Intelligent Optimization, vol. 6683, pp. 507–523. Springer, Berlin (2011). https://doi.org/10.1007/978-3-642-25566-3_40

  35. Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. Proc. AAAI Conf. Artif. Intell. 33(01), 4780–4789 (2019). https://doi.org/10.1609/aaai.v33i01.33014780

    Article  Google Scholar 

  36. Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. http://arxiv.org/abs/1802.03268. Accessed 13 Jan 2023

  37. Cai, H., Chen, T., Zhang, W., Yu, Y., Wang, J.: Efficient architecture search by network transformation. Proc. AAAI Conf. Artif. Intell. 32(1) (2018). https://doi.org/10.1609/aaai.v32i1.11709

  38. Wei, T., Wang, C., Rui, Y., Chen, C.W.: ‘Network Morphism’. http://arxiv.org/abs/1603.01670. Accessed 13 Jan 2023

  39. Elsken, T., Metzen, J.H., Hutter, F.: Efficient multi-objective neural architecture search via lamarckian evolution. http://arxiv.org/abs/1804.09081. Accessed 13 Jan 2023

  40. Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. http://arxiv.org/abs/1806.09055. Accessed 13 Jan 2023

  41. Jin, H., Song, Q., Hu, X.: Auto-keras: an efficient neural architecture search system. http://arxiv.org/abs/1806.10282. Accessed 13 Jan 2023

  42. Saxena, S., Verbeek, J.: Convolutional neural fabrics. http://arxiv.org/abs/1606.02492. Accessed 13 Jan 2023

  43. Luo, R., Tian, F., Qin, T., Chen, E., Liu, T.-Y.: Neural architecture optimization. http://arxiv.org/abs/1808.07233. Accessed 13 Jan 2023

  44. Brock, A., Lim, T., Ritchie, J.M., Weston, N.: SMASH: one-shot model architecture search through hypernetworks. http://arxiv.org/abs/1708.05344. Accessed 13 Jan 2023

  45. Ha, D., Dai, A., Le, Q.V.: ‘HyperNetworks’. http://arxiv.org/abs/1609.09106. Accessed 13 Jan 2023 2023

  46. Baker, B., Gupta, O., Naik, N., Raskar, R.: Designing neural network architectures using reinforcement learning. http://arxiv.org/abs/1611.02167. Accessed 13 Jan 2023

  47. Schweitzer, P.J., Gavish, B.: An optimality principle for Markovian decision processes. J. Math. Anal. Appl. 54(1), 173–184 (1976). https://doi.org/10.1016/0022-247X(76)90243-2

    Article  MathSciNet  Google Scholar 

  48. Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 60 (2019). https://doi.org/10.1186/s40537-019-0197-0

    Article  Google Scholar 

  49. Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: AutoAugment: learning augmentation strategies from data. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 113–123 (2019). https://doi.org/10.1109/CVPR.2019.00020

  50. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  51. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. http://arxiv.org/abs/1707.06347. Accessed 14 Jan 2023

  52. Lim, S., Kim, I., Kim, T., Kim, C., Kim, S.: ‘Fast AutoAugment’. http://arxiv.org/abs/1905.00397. Accessed 14 Jan 2023

  53. Shahrokh Esfahani, M., Dougherty, E.R.: Effect of separate sampling on classification accuracy. Bioinformatics 30(2), 242–250 (2014). https://doi.org/10.1093/bioinformatics/btt662

  54. Jones, D.R.: A Taxonomy of Global Optimization Methods Based on Response Surfaces, pp. 345–383. Springer (2001)

    Google Scholar 

  55. Moritz, P., et al.: Ray: a distributed framework for emerging AI applications. http://arxiv.org/abs/1712.05889. Accessed 14 Jan 2023

  56. Hataya, R., Zdenek, J., Yoshizoe, K., Nakayama, H.: Faster AutoAugment: Learning Augmentation Strategies using Backpropagation. http://arxiv.org/abs/1911.06987. Accessed 14 Jan 2023

  57. Ho, D., Liang, E., Stoica, I., Abbeel, P., Chen, X.: Population based augmentation: efficient learning of augmentation policy schedules. http://arxiv.org/abs/1905.05393. Accessed 14 Jan 2023

  58. Zhang, X., Wang, Q., Zhang, J., Zhong, Z.: Adversarial AutoAugment. http://arxiv.org/abs/1912.11188. Accessed 14 Jan 2023

  59. Inoue, H.: Data augmentation by pairing samples for images classification. http://arxiv.org/abs/1801.02929. Accessed 14 Jan 2023

  60. DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. http://arxiv.org/abs/1708.04552. Accessed 14 Jan 2023

  61. Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: practical automated data augmentation with a reduced search space. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA, pp. 3008–3017 (2020). https://doi.org/10.1109/CVPRW50498.2020.00359

  62. Zoph, B., et al.: Vedaldi, A., Bischof, H., Brox, T., J.-M. Frahm (Eds.) Springer International Publishing, Cham, vol. 12372, pp. 566–583 (2020). https://doi.org/10.1007/978-3-030-58583-9_34

  63. Cubuk, E.D., Zoph, B., Schoenholz, S.S., Le, Q.V.: Intriguing properties of adversarial examples. http://arxiv.org/abs/1711.02846. Accessed 14 Jan 2023

  64. Muller, S.G., Hutter, F.: TrivialAugment: tuning-free yet state-of-the-art data augmentation. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, pp. 754–762 (2021). https://doi.org/10.1109/ICCV48922.2021.00081

  65. LingChen, T.C., Khonsari, A., Lashkari, A., Nazari, M.R., Sambee, J.S., Nascimento, M.A.: UniformAugment: a search-free probabilistic data augmentation approach. http://arxiv.org/abs/2003.14348. Accessed 14 Jan 2023

  66. Negassi, M., Wagner, D., Reiterer, A.: Smart(Sampling)Augment: optimal and efficient data augmentation for semantic segmentation. Algorithms 15(5), 165 (2022). https://doi.org/10.3390/a15050165

    Article  Google Scholar 

  67. Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: AugMix: a simple data processing method to improve robustness and uncertainty. http://arxiv.org/abs/1912.02781. Accessed 14 Jan 2023

  68. Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: regularization strategy to train strong classifiers with localizable features. http://arxiv.org/abs/1905.04899. Accessed 14 Jan 2023

  69. Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: beyond empirical risk minimization. http://arxiv.org/abs/1710.09412. Accessed 14 Jan 2023

  70. Zheng, Y., Zhang, Z., Yan, S., Zhang, M.: Deep AutoAugment. http://arxiv.org/abs/2203.06172. Accessed 14 Jan 2023

  71. Li, Y., Hu, G., Wang, Y., Hospedales, T., Robertson, N.M., Yang, Y.: DADA: differentiable automatic data augmentation. http://arxiv.org/abs/2003.03780. Accessed 14 Jan 2023

  72. Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. http://arxiv.org/abs/1611.01144. Accessed 14 Jan 2023

  73. Grathwohl, W., Choi, D., Wu, Y., Roeder, G., Duvenaud, D.: Backpropagation through the void: optimizing control variates for black-box gradient estimation. http://arxiv.org/abs/1711.00123. Accessed 14 Jan 2023

  74. Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradients through stochastic neurons for conditional computation. http://arxiv.org/abs/1308.3432. Accessed 14 Jan 2023

  75. Liu, A., Huang, Z., Huang, Z., Wang, N.: Direct differentiable augmentation search. http://arxiv.org/abs/2104.04282. Accessed 14 Jan 2023

  76. Lin, C., et al.: Online hyper-parameter learning for auto-augmentation strategy. http://arxiv.org/abs/1905.07373. Accessed 14 Jan 2023

  77. Chen, C.-Y., Chang, C.-H.: Hypernetwork-based augmentation. http://arxiv.org/abs/2006.06320. Accessed 14 Jan 2023

Download references

Acknowledgments

This work was co-financed by the project PRiiMe - Piston Rings intelligent inspection Machine (N° POCI-01-0247-FEDER-047062), financed by Portugal 2020, under the Competitiveness and Internationalization Operational Program (POCI), and by the European Regional Development Fund (ERDF). Authors would like to acknowledge the TEXP@CT Portugal Project, co-financed by the RRP - Recovery and Resilience Plan and the European NextGeneration EU Funds, within the scope of the Mobilizing Agendas for Reindustrialization, under reference C644915249-00000025. Finally, the authors would, also, like to acknowledge the Vine&Wine Portugal Project, co-financed by the RRP - Recovery and Resilience Plan and the European NextGeneration EU Funds, within the scope of the Mobilizing Agendas for Reindustrialization, under reference C644866286-00000011''.

Author information

Authors and Affiliations

  1. Centre for Computer Graphics (CCG), Campus de Azurém, Edifício 14, 4800-058, Guimarães, Portugal

    Somayeh Shahrabadi & Telmo Adão

  2. ALGORITMI Research Centre, University of Minho, Guimarães, Portugal

    Somayeh Shahrabadi, Telmo Adão, Victor Alves & Luís G.Magalhães

Authors
  1. Somayeh Shahrabadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Telmo Adão
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Victor Alves
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Luís G.Magalhães
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Somayeh Shahrabadi .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shahrabadi, S., Adão, T., Alves, V., G.Magalhães, L. (2024). Automatic Optimization-Based Methods in Machine Learning: A Systematic Review. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2023. Lecture Notes in Networks and Systems, vol 823. Springer, Cham. https://doi.org/10.1007/978-3-031-47724-9_21

Download citation

Publish with us

Policies and ethics

Search

Navigation