Optimality and Equilibrium of Exploration Ratio for Multiagent Learning in Nonstationary Environments
I investigate relations between total performance of agent societies and relative performance of individual agents with respect to exploration ratio of multiagent learning. The exploration ratio is a key parameter to determine features of multiagent learning in two aspects: as a speed controller of learning in individual agents, and as a reciprocal noise factor for other agents. The investigation figures out trade-off of the two aspects and shows existence of single optimal value of the ratio to minimize the learning errors. I also carried out experiments to compare the performances of agents who use different exploration ratios. The results of the experiments tells existence of equilibrium points to choose the ratio by individual agents. Finally, we discuss the relationship between optimal and equilibrium values of the exploration ratio, which might bring dilemma of selection of the exploration ratio in an evolutionary way.
This work was supported by JST CREST and JSPS KAKENHI 24300064.
- 1.Kaisers, M., Tuyls, K.: Frequency adjusted multi-agent q-learning. In: Proceedings of 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2010), pp. 309–315, May 2010Google Scholar
- 2.Noda, I.: Limitations of simultaneous multiagent learning in nonstationary environments. In: Proceedings of 2013 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT 2013), pp. 309–314. IEEE, November 2013Google Scholar
- 3.Noda, I.: Robustness of optimality of exploration ratio against agent population in multiagent learning for nonstationary environments. In: Multiagent Interaction Without Prior Coordination (Technical report WS-14-09), pp. 28–34. AAAI, July 2014Google Scholar
- 4.Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
- 7.Wunder, M., Littman, M.L., Babes, M.: Classes of multiagent q-learning dynamicswith epsilon-greedy exploration. In: Frnkranz, J., Joachims, T. (eds.) Proceedings of the 27th International Conference on Machine Learning (ICML 2010), pp. 1167–1174. Omnipress (2010). http://www.icml2010.org/papers/191.pdf