Skip to main content
SpringerLink
Log in

Hyperparameter Optimization in a Convolutional Neural Network Using Metaheuristic Algorithms

  • Chapter
  • First Online:
Metaheuristics in Machine Learning: Theory and Applications

Part of the book series: Studies in Computational Intelligence ((SCI,volume 967))

Abstract

The hyperparameters of a convolutional neural network always have been important, because the performance of the convolutional neural network depends largely on them. To find manually the optimal value requires a lot of work, experience, and time. Metaheuristic algorithms are approximate general-purpose search and optimization algorithms, they are used to find a good solution into search space. In the book chapter, the hyperparameters of a convolutional neural network are optimized. A metaheuristic algorithm is used to finds the best solution, that is, the best hyperparameter value for higher accuracy of the convolutional neural network (CNN). The results of four metaheuristic algorithms that have the same objective are compared, which are PSO (Particle Swarm Optimization), ABC (Artificial Bee Colony), ALO (Ant Lion Optimization), and BA (Bat Algorithm). Each algorithm uses the same convolutional neural network architecture as well as the same dataset (MNIST database) also each algorithm is run 30 times to perform the corresponding statistics. Finally, the results of the algorithms are shown.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Teuwen, J., Moriakov, N., Convolutional neural networks, in Handbook of Medical Image Computing and Computer Assisted Intervention (Elsevier, 2019), pp. 481–501

    Google Scholar 

  2. Lopez Pinaya, W.H., Vieira, S., Garcia-Dias, R., Mechelli, A., Convolutional neural networks, in Machine Learning (Elsevier, 2020), pp. 173–191

    Google Scholar 

  3. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 1–46 (1998)

    Google Scholar 

  4. Krizhevsky A., Sutskever I., Hinton G., ImageNet classification with deep convolutional neural networks. Commun. ACM. 60, 84–90 (2017)

    Google Scholar 

  5. Simonyan, K., Zisserman, A., Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)

    Google Scholar 

  6. Deepan, P., Sudha, L.R., Object classification of remote sensing image using deep convolutional neural network, in The Cognitive Approach in Cloud Computing and Internet of Things Technologies for Surveillance Tracking Systems (Elsevier, 2020), pp. 107–120

    Google Scholar 

  7. H. Cui, J. Bai, A new hyperparameters optimization method for convolutional neural networks. Pattern Recogn. Lett. 125, 828–834 (2019). https://doi.org/10.1016/j.patrec.2019.02.009

    Article  Google Scholar 

  8. Loussaief, S., Abdelkrim, A., Convolutional neural network hyper-parameters optimization based on genetic algorithms. Int. J. Adv. Comput. Sci. Appl. 9 (2018)

    Google Scholar 

  9. Y. Wang, H. Zhang, G. Zhang, cPSO-CNN: an efficient PSO-based algorithm for fine-tuning hyper-parameters of convolutional neural networks. Swarm Evol. Comput. 49, 114–123 (2019). https://doi.org/10.1016/j.swevo.2019.06.002

    Article  Google Scholar 

  10. Lorenzo, P.R., Nalepa, J., Kawulok, M., Ramos, L.S., Pastor, J.R., Particle swarm optimization for hyper-parameter selection in deep neural networks, in GECCO 2017—Proceedings of the 2017 Genetic and Evolutionary Computation Conference (Association for Computing Machinery, Inc, 2017), pp 481–488

    Google Scholar 

  11. Dewancker, I., McCourt, M., Clark, S., Bayesian optimization for machine learning : a practical guidebook. ArXiv abs/1612.04858 (2016)

    Google Scholar 

  12. Bergstra, J., Bengio, Y., Random search for hyper-parameter optimization. J. Machine Learning Res.13, 281–305 (2012)

    Google Scholar 

  13. Frazier, P.I., A tutorial on bayesian optimization. ArXiv abs/1807.02811 (2018)

    Google Scholar 

  14. Cuevas, E., Díaz, M. A method for estimating view transformations from image correspondences based on the harmony search algorithm. Comput. Intell. Neurosci. (2015). https://doi.org/10.1155/2015/434263

  15. D. Oliva, S. Hinojosa, E. Cuevas, G. Pajares, O. Avalos, J. Gálvez, Cross entropy based thresholding for magnetic resonance brain images using Crow Search Algorithm. Expert Syst. Appl. 79, 164–180 (2017). https://doi.org/10.1016/j.eswa.2017.02.042

    Article  Google Scholar 

  16. D. Zaldivar, E. Cuevas, O. Maciel, A. Valdivia, E. Chavolla, D. Oliva, Learning classical and metaheuristic optimization techniques by using an educational platform based on LEGO robots. Int. J. Electr. Eng. Educ. (2019). https://doi.org/10.1177/0020720918822738

    Article  Google Scholar 

  17. Cuevas, E., Barocio Espejo, E., Conde Enríquez, A., Clustering representative electricity load data using a particle swarm optimization algorithm. Stud. Comput. Intell. 187–210 (2019. (Springer)

    Google Scholar 

  18. Avalos, O., Cuevas, E., Gálvez, J. Induction motor parameter identification using a gravitational search algorithm. Computers 5 (2016). https://doi.org/10.3390/computers5020006

  19. E. Cuevas, Artificial Bee Colony (ABC) algorithm and its use in digital image processing. Intel. Artif. 18, 50 (2015). https://doi.org/10.4114/intartif.vol18,issue55,pp.50-68

    Article  Google Scholar 

  20. D. Oliva, E. Cuevas, G. Pajares, Parameter identification of solar cells using artificial bee colony optimization. Energy 72, 93–102 (2014). https://doi.org/10.1016/j.energy.2014.05.011

    Article  Google Scholar 

  21. M. Elhoseny, D. Oliva, V. Osuna-Enciso, A.E. Hassanien, M. Gunasekaran, Parameter identification of two dimensional digital filters using electro-magnetism optimization. Multimedia Tools Appl. 79, 5005–5022 (2020). https://doi.org/10.1007/s11042-018-6095-1

    Article  Google Scholar 

  22. Oliva, D., Hinojosa, S., Demeshko, M.V., Engineering applications of metaheuristics: an introduction. J. Phys. Conf. Ser. (2017). [Institute of Physics Publishing]

    Google Scholar 

  23. Avalos, O., Cuevas, E., Valdivia-González, A., Gálvez, J., Hinojosa, S., Zaldívar, D., Oliva, D., A comparative study of evolutionary computation techniques for solar cells parameter estimation. Computacion y Sistemas 23, 231–256 (2019). https://doi.org/10.13053/CyS-23-1-2881

  24. Dokeroglu, T., Sevinc, E., Kucukyilmaz, T., Cosar, A., A survey on new generation metaheuristic algorithms. Comput. Ind. Eng. 137 (2019). https://doi.org/10.1016/j.cie.2019.106040

  25. Roy, A., Dutta, D., Choudhury, K (2013) Training artificial neural network using particle swarm optimization algorithm. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 3(3) (2013)

    Google Scholar 

  26. Garro, B.A., Vázquez, R.A., Designing artificial neural networks using particle swarm optimization algorithms. Comput. Intell. Neurosci. (2015). https://doi.org/10.1155/2015/369298

  27. W. Gómez-Flores, J. Hernández-López, Automatic adjustment of the pulse-coupled neural network hyperparameters based on differential evolution and cluster validity index for image segmentation. Appl. Soft Comput. J. (2019). https://doi.org/10.1016/j.asoc.2019.105547

    Article  Google Scholar 

  28. W.Y. Lee, S.M. Park, K.B. Sim, Optimal hyperparameter tuning of convolutional neural networks based on the parameter-setting-free harmony search algorithm. Optik 172, 359–367 (2018). https://doi.org/10.1016/j.ijleo.2018.07.044

    Article  Google Scholar 

  29. Aszemi, N.M., Dominic, P.D.D., Hyperparameter optimization in convolutional neural network using genetic algorithms. Int. J. Adv. Comput. Sci. Appl. 10 (2019)

    Google Scholar 

  30. Y. Yoo, Hyperparameter optimization of deep neural network using univariate dynamic encoding algorithm for searches ✩. Knowl. Based Syst. 178, 74–83 (2019). https://doi.org/10.1016/j.knosys

    Article  Google Scholar 

  31. Y. Bengio, Learning deep architectures for AI. Found. Trends Mach. Learn. 2, 1–27 (2009). https://doi.org/10.1561/2200000006

    Article  MATH  Google Scholar 

  32. S. Mirjalili, The ant lion optimizer. Adv. Eng. Softw. 83, 80–98 (2015). https://doi.org/10.1016/j.advengsoft.2015.01.010

    Article  Google Scholar 

  33. H. Garg, Solving structural engineering design optimization problems using an artificial bee colony algorithm. J. Ind. Manag. Optim. 10, 777–794 (2014). https://doi.org/10.3934/jimo.2014.10.777

    Article  MathSciNet  MATH  Google Scholar 

  34. Yang, X.S., A new metaheuristic bat-inspired algorithm, in Studies in Computational Intelligence (2010), pp 65–74

    Google Scholar 

  35. Solano, R., Meléndez, N., Garcia, C., C-PSO: Particle Swarm Optimization by Embedding Coulomb’s Law. Res. Comput. Sci. 147, 311–322 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. División de Electrónica y Computación, Universidad de Guadalajara, CUCEI, Av. Revolución 1500, C.P. 44100, Guadalajara, Jalisco, México

    Angel Gaspar, Diego Oliva, Erik Cuevas, Daniel Zaldívar & Marco Pérez

  2. Departamento de Ingeniería del Software e Inteligencia Artificial, Facultad Informática, Universidad Complutense de Madrid, 28040, Madrid, España

    Gonzalo Pajares

Authors
  1. Angel Gaspar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diego Oliva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Erik Cuevas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Zaldívar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marco Pérez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gonzalo Pajares
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Angel Gaspar .

Editor information

Editors and Affiliations

  1. Computer Sciences Department, CUCEI, University of Guadalajara, Guadajalara, Jalisco, Mexico

    Diego Oliva

  2. Department of Computer Science, Faculty of Computers and Information, Minia University, Minia, Egypt

    Essam H. Houssein

  3. Computer Sciences Department, CUCEI, University of Guadalajara, Guadajalara, Jalisco, Mexico

    Salvador Hinojosa

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gaspar, A., Oliva, D., Cuevas, E., Zaldívar, D., Pérez, M., Pajares, G. (2021). Hyperparameter Optimization in a Convolutional Neural Network Using Metaheuristic Algorithms. In: Oliva, D., Houssein, E.H., Hinojosa, S. (eds) Metaheuristics in Machine Learning: Theory and Applications. Studies in Computational Intelligence, vol 967. Springer, Cham. https://doi.org/10.1007/978-3-030-70542-8_2

Download citation

Publish with us

Policies and ethics

Search

Navigation