Abstract
In this work we propose an architecture which learns to select subgoals with Deep Q-Learning in order to decrease the load of a planner when faced with scenarios with tight time restrictions, such as online execution systems. We have trained this architecture on a video game environment used as a standard testbed for intelligent systems applications. We experiment with different values of the discount rate \(\gamma \) and show the importance of long-term thinking when selecting subgoals. We also compare our approach against a classical planner and show how it is able to greatly reduce time requirements, although obtaining plans with 25% more actions on average. We conclude our approach is competitive with a classical planner and presents better generalization properties than most Reinforcement Learning algorithms when applied to new levels of the same game.
This work has been partially supported by Spanish Government Project MINECO RTI2018-098460-B-I00 and UE FEDER.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bonanno, D., Roberts, M., Smith, L., Aha, D.W.: Selecting subgoals using deep learning in minecraft: a preliminary report. In: IJCAI Workshop on Deep Learning for Artificial Intelligence (2016)
Cox, M.T.: Perpetual self-aware cognitive agents. AI Mag. 28(1), 32–45 (2007)
Fox, M., Long, D.: PDDL2: 1: an extension to PDDL for expressing temporal planning domains. J. Artif. Intell. Res. 20, 61–124 (2003)
Ghallab, M., Nau, D., Traverso, P.: Automated Planning and Acting. Cambridge University Press, New York (2016)
Hoffmann, J.: FF: the fast-forward planning system. AI Mag. 22(3), 57 (2001)
Jaidee, U., Muñoz-Avila, H., Aha, D.W.: Learning and reusing goal-specific policies for goal-driven autonomy. In: Agudo, B.D., Watson, I. (eds.) ICCBR 2012. LNCS (LNAI), vol. 7466, pp. 182–195. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32986-9_15
Klenk, M., Molineaux, M., Aha, D.W.: Goal-driven autonomy for responding to unexpected events in strategy simulations. Comput. Intell. 29(2), 187–206 (2013)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Mnih, V., et al.: Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Mukadam, M., Cosgun, A., Nakhaei, A., Fujimura, K.: Tactical decision making for lane changing with deep reinforcement learning (2017)
Niemueller, T., Hofmann, T., Lakemeyer, G.: Goal reasoning in the clips executive for integrated planning and execution. In: Proceedings of the International Conference on Automated Planning and Scheduling, vol. 29, pp. 754–763 (2019)
Patra, S., Ghallab, M., Nau, D., Traverso, P.: Acting and planning using operational models. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 7691–7698 (2019)
Perez-Liebana, D., et al.: The 2014 general video game playing competition. IEEE Trans. Comput. Intell. AI Games 8(3), 229–243 (2015)
Shen, Y., Zhao, N., Xia, M., Du, X.: A deep q-learning network for ship stowage planning problem. Pol. Marit. Res. 24(s3), 102–109 (2017)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Tamar, A., Wu, Y., Thomas, G., Levine, S., Abbeel, P.: Value iteration networks. In: Advances in Neural Information Processing Systems, pp. 2154–2162 (2016)
Torrado, R.R., Bontrager, P., Togelius, J., Liu, J., Perez-Liebana, D.: Deep reinforcement learning for general video game AI. In: 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1–8. IEEE (2018)
Toyer, S., Trevizan, F., Thiébaux, S., Xie, L.: Action schema networks: generalised policies with deep learning. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Watkins, C.J.C.H.: Learning from delayed rewards (1989)
Zhang, C., Vinyals, O., Munos, R., Bengio, S.: A study on overfitting in deep reinforcement learning (2018). arXiv preprint arXiv:1804.06893
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Núñez-Molina, C., Vellido, I., Nikolov-Vasilev, V., Pérez, R., Fdez-Olivares, J. (2021). A Proposal to Integrate Deep Q-Learning with Automated Planning to Improve the Performance of a Planning-Based Agent. In: Alba, E., et al. Advances in Artificial Intelligence. CAEPIA 2021. Lecture Notes in Computer Science(), vol 12882. Springer, Cham. https://doi.org/10.1007/978-3-030-85713-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-85713-4_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85712-7
Online ISBN: 978-3-030-85713-4
eBook Packages: Computer ScienceComputer Science (R0)