Skip to main content
SpringerLink
Log in

A Q-Learning Hyperheuristic Binarization Framework to Balance Exploration and Exploitation

  • Conference paper
  • First Online:
Applied Informatics (ICAI 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1277))

Included in the following conference series:

Abstract

Many Metaheuristics solve optimization problems in the continuous domain, so it is necessary to apply binarization schemes to solve binary problems, this selection that is not trivial since it impacts the heart of the search strategy: its ability to explore. This paper proposes a Hyperheuristic Binarization Framework based on a Machine Learning technique of Reinforcement Learning to select the appropriate binarization strategy, which is applied in a Low Level Metaheuristic. The proposed implementation is composed of a High Level Metaheuristic, Ant Colony Optimization, using Q-Learning replacing the pheromone trace component. In the Low Level Metaheuristic, we use a Grey Wolf Optimizer to solve the binary problem with binarization scheme fixed by ants. This framework allowing a better balance between exploration and exploitation, and can be applied selecting others low level components.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Beasley, J.E.: Or-library: distributing test problems by electronic mail. J. Oper. Res. Soc. 41(11), 1069–1072 (1990). http://www.jstor.org/stable/2582903

    Article  Google Scholar 

  2. Bishop, C.M.: Pattern Recoginiton and Machine Learning (2006)

    Google Scholar 

  3. Blum, C., Roli, A.: Metaheuristics in combinatorial optimization: overview and conceptual comparison. ACM Comput. Surv. 35(3), 268–308 (2003). https://doi.org/10.1145/937503.937505

    Article  Google Scholar 

  4. Book, R.V.: Book review: computers and intractability: a guide to the theory of NP-completeness. Bull. Am. Math. Soc. 3(2), 898–905 (1980). https://doi.org/10.1090/s0273-0979-1980-14848-x

    Article  MathSciNet  Google Scholar 

  5. Burke, E., Kendall, G., Newall, J., Hart, E., Ross, P., Schulenburg, S.: Hyper-heuristics: an emerging direction in modern search technology. In: Glover, F., Kochenberger, G.A. (eds) Handbook of Metaheuristics. Springer, Boston(2006). https://doi.org/10.1007/0-306-48056-5_16

  6. Celebi, M.E., Aydin, K. (eds.): Unsupervised Learning Algorithms. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-24211-8

    Book  Google Scholar 

  7. Choong, S.S., Wong, L.P., Lim, C.P.: Automatic design of hyper-heuristic based on reinforcement learning. Inf. Sci. (NY). (2018). https://doi.org/10.1016/j.ins.2018.01.005

    Article  Google Scholar 

  8. Cowling, P., Kendall, G., Soubeiga, E.: A hyperheuristic approach to scheduling a sales summit. In: Burke, E., Erben, W. (eds.) PATAT 2000. LNCS, vol. 2079, pp. 176–190. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44629-X_11

    Chapter  Google Scholar 

  9. Crawford, B., Soto, R., Astorga, G., García, J.: Constructive metaheuristics for the set covering problem. In: Korošec, P., Melab, N., Talbi, E.-G. (eds.) BIOMA 2018. LNCS, vol. 10835, pp. 88–99. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91641-5_8

    Chapter  Google Scholar 

  10. Crawford, B., Soto, R., Astorga, G., García, J., Castro, C., Paredes, F.: Putting continuous metaheuristics to work in binary search spaces (2017). https://doi.org/10.1155/2017/8404231

  11. Crawford, B., Soto, R., Olivares, R., Riquelme, L., Astorga, G., Johnson, F., Cortés, E., Castro, C., Paredes, F.: A self-adaptive biogeography-based algorithm to solve the set covering problem. RAIRO - Oper. Res. 53(3), 1033–1059 (2019). https://doi.org/10.1051/ro/2019039

    Article  MathSciNet  MATH  Google Scholar 

  12. Dorigo, M., Birattari, M., Stützle, T.: Ant colony optimization artificial ants as a computational intelligence technique. IEEE Comput. Intell. Mag. (2006). https://doi.org/10.1109/CI-M.2006.248054

    Article  Google Scholar 

  13. Dorigo, M., Gambardella, L.M.: Ant colony system: A cooperative learning approach to the traveling salesman problem. IEEE Trans. Evol. Comput. (1997). https://doi.org/10.1109/4235.585892

    Article  Google Scholar 

  14. Dorigo, M., Maniezzo, V., Colorni, A., Dorigo, M.: Positive Feedback as a Search Strategy. Technical report, 91-016 (1991)

    Google Scholar 

  15. Eusuff, M., Lansey, K., Pasha, F.: Shuffled frog-leaping algorithm: A memetic meta-heuristic for discrete optimization. Eng. Optim. (2006). https://doi.org/10.1080/03052150500384759

    Article  MathSciNet  Google Scholar 

  16. Feo, T.A., Resende, M.G.: A probabilistic heuristic for a computationally difficult set covering problem. Oper. Res. Lett. (1989). https://doi.org/10.1016/0167-6377(89)90002-3

    Article  MathSciNet  MATH  Google Scholar 

  17. Glover, F.: Tabu search-Part II. ORSA J. Comput. (1990). https://doi.org/10.1287/ijoc.2.1.4

    Article  MATH  Google Scholar 

  18. Holland, J.H.: Genetic algorithms. Sci. Am. (1992). https://doi.org/10.1038/scientificamerican0792-66

    Article  Google Scholar 

  19. Hussain, K., Mohd Salleh, M.N., Cheng, S., Shi, Y.: Metaheuristic research: a comprehensive survey. Artif. Intell. Rev. 52(4), 2191–2233 (2018). https://doi.org/10.1007/s10462-017-9605-z

    Article  Google Scholar 

  20. Khamassi, I., Hammami, M., Ghédira, K.: Ant-Q hyper-heuristic approach for solving 2-dimensional Cutting Stock Problem. In: IEEE SSCI 2011 - Symposium Series Computing Intelligent - SIS 2011 2011 IEEE Symposium Swarm Intelligent (2011). https://doi.org/10.1109/SIS.2011.5952530

  21. Kotsiantis, S.B.: Supervised machine learning: a review of classification techniques. In: Proceedings of the 2007 Conference on Emerging Artificial Intelligence Applications in Computer Engineering: Real Word AI Systems with Applications in EHealth, HCI, Information Retrieval and Pervasive Technologies, pp. 3–24. IOS Press, NLD (2007). https://doi.org/10.5555/1566770.1566773

  22. Leguizamon, G., Michalewicz, Z.: A new version of ant system for subset problems. In: Proceedings of the 1999 Congress on Evolutionary Computation, CEC 1999 (1999). https://doi.org/10.1109/CEC.1999.782655

  23. Lones, M.: Sean Luke: essentials of metaheuristics. Genet. Program Evolvable Mach. (2011). https://doi.org/10.1007/s10710-011-9139-0

    Article  Google Scholar 

  24. Mafarja, M., Eleyan, D., Abdullah, S., Mirjalili, S.: S-shaped vs. V-shaped transfer functions for ant lion optimization algorithm in feature selection problem. In: ACM International Conference Proceedings Series (2017). https://doi.org/10.1145/3102304.3102325

  25. Mirjalili, S., Mirjalili, S.M., Lewis, A.: Grey Wolf Optimizer. Adv. Eng. Softw. (2014). https://doi.org/10.1016/j.advengsoft.2013.12.007

    Article  Google Scholar 

  26. Mirjalili, S., Song Dong, J., Lewis, A.: Nature-Inspired Optimizers (2020).https://doi.org/10.1007/978-3-030-12127-3

  27. Müller, A.C., Guido, S.: Introduction to Machine Learning with Python: a guide for data scientists (2016). https://doi.org/10.1017/CBO9781107415324.004

  28. Muncie, H.L., Sobal, J., DeForge, B.: Research methodologies (1989). https://doi.org/10.5040/9781350004900.0008

  29. Solnon, C.: Ants can solve constraint satisfaction problems. IEEE Trans. Evol. Comput. (2002). https://doi.org/10.1109/TEVC.2002.802449

    Article  Google Scholar 

  30. Song, H., Triguero, I., Özcan, E.: A review on the self and dual interactions between machine learning and optimisation. Progress Artif. Intell 8(2), 143–165 (2019). https://doi.org/10.1007/s13748-019-00185-z

    Article  Google Scholar 

  31. Stützle, T., Hoos, H.H.: MAX-MIN ant system. Futur. Gener. Comput. Syst. (2000). https://doi.org/10.1016/S0167-739X(00)00043-1

    Article  MATH  Google Scholar 

  32. Sutton, R.S., Barto, A.G.: Reinforcement learning: an introduction 2018. Technical report (2017). https://doi.org/10.1109/TNN.1998.712192

  33. Talbi, E.G.: Metaheuristics: From Design to Implementation (2009). https://doi.org/10.1002/9780470496916

  34. Talbi, E.G.: Machine learning into metaheuristics: a survey and taxonomy of data-driven metaheuristics, June 2020. https://hal.inria.fr/hal-02745295, working paper or preprint

  35. Watkins, C.J., Dayan, P.: Technical note: Q-learning. Mach. Learn. (1992). https://doi.org/10.1023/A:1022676722315

    Article  MATH  Google Scholar 

Download references

Acknowledgements

Felipe Cisternas-Caneo is supported by Grant DI Investigación Interdisciplinaria del Pregrado/VRIEA/PUCV/039.324/2020. Broderick Crawford is supported by Grant CONICYT/FONDECYT/REGULAR/1171243. Ricardo Soto is supported by Grant CONICYT/FONDECYT/REGULAR/1190129. José Lemus-Romani is supported by National Agency for Research and Development (ANID)/Scholarship Program/DOCTORADO NACIONAL/2019 - 21191692. José García is supported by the Grant CONICYT/FONDECYT/INICIACION/11180056.

Author information

Authors and Affiliations

  1. Pontificia Universidad Católica de Valparaíso, Valparaíso, Chile

    Diego Tapia, Broderick Crawford, Ricardo Soto, Felipe Cisternas-Caneo, José Lemus-Romani, Mauricio Castillo, José García & Wenceslao Palma

  2. Universidad Diego Portales, Santiago, Chile

    Fernando Paredes

  3. Covenant University, Ota, Nigeria

    Sanjay Misra

Authors
  1. Diego Tapia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Broderick Crawford
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ricardo Soto
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Felipe Cisternas-Caneo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. José Lemus-Romani
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mauricio Castillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. José García
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Wenceslao Palma
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Fernando Paredes
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Sanjay Misra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diego Tapia .

Editor information

Editors and Affiliations

  1. Universidad Distrital Francisco Jose de Caldas, Bogota, Colombia

    Hector Florez

  2. Covenant University, Ota, Nigeria

    Sanjay Misra

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tapia, D. et al. (2020). A Q-Learning Hyperheuristic Binarization Framework to Balance Exploration and Exploitation. In: Florez, H., Misra, S. (eds) Applied Informatics. ICAI 2020. Communications in Computer and Information Science, vol 1277. Springer, Cham. https://doi.org/10.1007/978-3-030-61702-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61702-8_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61701-1

  • Online ISBN: 978-3-030-61702-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation