Play Ms. Pac-Man Using an Advanced Reinforcement Learning Agent

  • Nikolaos Tziortziotis
  • Konstantinos Tziortziotis
  • Konstantinos Blekas
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8445)


Reinforcement Learning (RL) algorithms have been promising methods for designing intelligent agents in games. Although their capability of learning in real time has been already proved, the high dimensionality of state spaces in most game domains can be seen as a significant barrier. This paper studies the popular arcade video game Ms. Pac-Man and outlines an approach to deal with its large dynamical environment. Our motivation is to demonstrate that an abstract but informative state space description plays a key role in the design of efficient RL agents. Thus, we can speed up the learning process without the necessity of Q-function approximation. Several experiments were made using the multiagent MASON platform where we measured the ability of the approach to reach optimum generic policies which enhances its generalization abilities.


Intelligent Agents Reinforcement Learning Ms. Pac-Man 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Galway, L., Charles, D., Black, M.: Machine learning in digital games: A survey. Artificial Intelligence Review 29, 123–161 (2008)CrossRefGoogle Scholar
  2. 2.
    Sutton, R.: Learning to predict by the method of temporal differences. Machine Learning 3(1), 9–44 (1988)Google Scholar
  3. 3.
    Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of Artificial Inteligence Research 4, 237–285 (1996)Google Scholar
  4. 4.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  5. 5.
    Szita, I.: Reinforcement learning in games. In: Reinforcement Learning, pp. 539–577 (2012)Google Scholar
  6. 6.
    Szita, I., Lorincz, A.: Learning to play using low-complexity rule-based policies: Illustrations through ms. pac-man. Journal of Artificial Intelligence Research 30, 659–684 (2007)zbMATHGoogle Scholar
  7. 7.
    Lucas, S.M.: Evolving a neural network location evaluator to play ms. pac-man. In: Proc. of IEEE Symposium on Computational Intelligence and Games (CIG 2005), pp. 203–210 (2005)Google Scholar
  8. 8.
    Bom, L., Henken, R., Wiering, M.A.: Reinforcement learning to train ms. pac-man using higher-order action-relative inputs. In: Proc. of IEEE Intern. Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 156–163 (2013)Google Scholar
  9. 9.
    Alhejali, A.M., Lucas, S.M.: Evolving diverse ms. pac-man playing agents using genetic programming. In: Proc. of IEEE Symposium on Computational Intelligence and Games (CIG 2010), pp. 53–60 (2010)Google Scholar
  10. 10.
    Samothrakis, S., Robles, D., Lucas, S.: Fast approximate max-n monte-carlo tree search for ms. pac-man. IEEE Trans. on Computational Intelligence and AI in Games 3(2), 142–154 (2011)CrossRefGoogle Scholar
  11. 11.
    Nguyen, K.Q., Thawonmas, R.: Monte carlo tree search for collaboration control of ghosts in ms. pac-man. IEEE Trans. on Computational Intelligence and AI in Games 5(1), 57–68 (2013)CrossRefGoogle Scholar
  12. 12.
    Torrey, L., Taylor, M.: Teaching on a budget: Agents advising agents in reinforcement learning. In: Intern. Conferecene on Autonomous Agents and Multi-agent Systems (AAMAS), pp. 1053–1060 (2013)Google Scholar
  13. 13.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley (2005)Google Scholar
  14. 14.
    Singh, S., Sutton, R.S., Kaelbling, P.: Reinforcement learning with replacing eligibility traces, pp. 123–158 (1996)Google Scholar
  15. 15.
    Hart, P.E., Nilsson, N.J., Raphael, B.: A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems, Science, and Cybernetics SSC-4(2), 100–107 (1968)CrossRefGoogle Scholar
  16. 16.
    Luke, S., Cioffi-Revilla, C., Panait, L., Sullivan, K., Balan, G.: Mason: A multiagent simulation environment. Simulation 81(7), 517–527 (2005)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Nikolaos Tziortziotis
    • 1
  • Konstantinos Tziortziotis
    • 1
  • Konstantinos Blekas
    • 1
  1. 1.Department of Computer Science & EngineeringUniversity of IoanninaIoanninaGreece

Personalised recommendations