Skip to main content
SpringerLink
Log in

Enhancing Learning Efficiency in FACL: A Novel Fuzzy Rule Transfer Method for Transfer Learning

  • Published:
International Journal of Fuzzy Systems Aims and scope Submit manuscript

Abstract

The concept of leveraging knowledge from previous experience to accelerate learning forms the crux of transfer learning. Within the realm of reinforcement learning (RL), the agent typically requires protracted interaction with the environment, which can be time-consuming and can lead to slow convergence. Transfer learning offers a promising solution in such settings. In this paper, we investigate the application of transfer learning in the fuzzy reinforcement learning domain, specifically within the context of differential games. We introduce a novel approach for knowledge transfer across analogous tasks, employing fuzzy logic controllers as function approximators, notably within the Fuzzy Actor-Critic Learning (FACL) algorithm. Specifically, we propose a strategy for fuzzy rule transfer aimed at mapping fuzzy rules between the source and target tasks. The target task is assumed to be related to the source task yet it contains more complex states. Our approach has been implemented and tested within the domain of differential games in which all state space and action space are continuous. The simulation outcomes demonstrate that the application of knowledge transfer enables RL agents to learn faster and achieve asymptotic performance more rapidly in the target task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT, Cambridge (1998))

    Google Scholar 

  2. Schwartz, H.M.: Multi-Agent Machine Learning: A Reinforcement Approach. Wiley, Hoboken (2014)

  3. Jouffe, L.: Actor-critic learning based on fuzzy inference system. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, vol. 1, pp. 339–344 (1996)

  4. Wang, L.-X.: A Course in Fuzzy Systems and Control. Prentice Hall, Upper Saddle River (1997)

    Google Scholar 

  5. Andrecut, M., Ali, M.K.: Fuzzy reinforcement learning. Int. J. Mod. Phys. C 13(05), 659–674 (2002)

    Article  Google Scholar 

  6. Kuo, Ping-Huan., Jun, Hu., Lin, Ssu-Ting., Hsu, Po-Wei.: Fuzzy deep deterministic policy gradient-based motion controller for humanoid robot. Int. J. Fuzzy Syst. 24(5), 2476–2492 (2022)

    Article  Google Scholar 

  7. Tsai, C.-C., Chen, H.-Y., Chen, S.-C., Tai, F.-C., Chen, G.-M.: Adaptive reinforcement learning formation control using ORFBLS for omnidirectional mobile multi-robots. Int. J. Fuzzy Syst. 25(9), 1–14 (2023)

  8. Wang, X., Bin, X., Guo, Y.: Fuzzy logic system-based robust adaptive control of AUV with target tracking. Int. J. Fuzzy Syst. 25(1), 338–346 (2023)

    Article  Google Scholar 

  9. Taylor, M.E., Stone, P.: Behavior transfer for value-function-based reinforcement learning. In Proceedings of the 4th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 53–59 (2005)

  10. Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. arXiv preprint (2015). arXiv:1511.05952

  11. Ng, A.Y., Harada, D., Russell, S.: Policy invariance under reward transformations: theory and application to reward shaping. In: ICML, vol. 99, pp. 278–287 (1999)

  12. Awheda, M.D., Schwartz, H.M.: A residual gradient fuzzy reinforcement learning algorithm for differential games. Int. J. Fuzzy Syst. 19, 1058–1076 (2017)

  13. Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10(Jul), 1633–1685 (2009)

  14. Taylor, M.E., Stone, P., Liu, Y.: Transfer learning via inter-task mappings for temporal difference learning. J. Mach. Learn. Res. 8(Sep), 2125–2167 (2007)

  15. Da Silva, F.L., Glatt, R., Costa, A.H.R.: Simultaneously learning and advising in multiagent reinforcement learning. In: Proceedings of the 16th Conference on Autonomous Agents and Multiagent Systems, pp. 1100–1108 (2017)

  16. Ramon, J., Driessens, K., Croonenborghs, T.: Transfer learning in reinforcement learning problems through partial policy recycling. In: European Conference on Machine Learning, pp. 699–707 (2007)

  17. Taylor, M.E., Stone, P.: Cross-domain transfer for reinforcement learning. In: Proceedings of the 24th International Conference on Machine Learning, pp. 879–886 (2007)

  18. Taylor, M.E., Whiteson, S., Stone, P.: Transfer via inter-task mappings in policy search reinforcement learning. In: Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, p. 37 (2007)

  19. Razavi, R., Klein, S., Claussen, H.: Self-optimization of capacity and coverage in lTE networks using a fuzzy reinforcement learning approach. In: 21st Annual IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, pp. 1865–1870 (2010)

  20. Jouffe, L.: Fuzzy inference system learning by reinforcement methods. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 28(3), 338–355 (1998)

  21. Van Buijtenen, W.M., Schram, G., Babuska, R., Verbruggen, H.B.: Adaptive fuzzy control of satellite attitude by reinforcement learning. IEEE Trans. Fuzzy Syst. 6(2), 185–194 (1998)

  22. Takagi, T., Sugeno, M.: Fuzzy identification of systems and its applications to modeling and control. IEEE Trans. Syst. Man Cybern. 1, 116–132 (1985)

    Article  Google Scholar 

  23. Givigi, S.N., Schwartz, H.M., Lu, X.: A reinforcement learning adaptive fuzzy controller for differential games. J. Intell. Robot. Syst. 59(1), 3–30 (2010)

  24. Desouky, Sameh F., Schwartz, Howard M.: Self-learning fuzzy logic controllers for pursuit-evasion differential games. Robot. Auton. Syst. 59(1), 22–33 (2011)

    Article  Google Scholar 

  25. Dai, X., Li, C.-K., Rad, A.B.: An approach to tune fuzzy controllers based on reinforcement learning for autonomous vehicle control. IEEE Trans. Intell. Transp. Syst. 6(3), 285–293 (2005)

  26. Awheda, M.D., Schwartz, H.: A decentralized fuzzy learning algorithm for pursuit-evasion differential games with superior evaders. J. Intell. Robot. Syst. 83, 35–53 (2015)

  27. Schwartz, H.: An object oriented approach to fuzzy actor-critic learning for multi-agent differential games. In: 2019 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 183–190 (2019)

  28. Givigi, S.N., Schwartz, H.M., Lu, X.: An experimental adaptive fuzzy controller for differential games. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pp. 3017–3023 (2009)

  29. Randløv, J,, Alstrøm, P.: Learning to drive a bicycle using reinforcement learning and shaping. In: ICML, vol. 98, pp. 463–471 (1998)

  30. Gullapalli, V., Barto, A.G.: Shaping as a method for accelerating reinforcement learning. In: Proceedings of the 1992 IEEE International Symposium on Intelligent Control, pp. 554–559 (1992)

  31. Isaacs, R.: Differential Games: A Mathematical Theory with Applications to Warfare and Pursuit, Control and Optimization. Wiley, New York (1965)

    Google Scholar 

  32. Analikwu, C.V., Schwartz, H.M.: Multi-agent learning in the game of guarding a territory. Int. J. Innov. Comput. Inf. Control 13(6), 1855–1872 (2017)

Download references

Author information

Authors and Affiliations

  1. Global Artificial Intelligence Accelerator, Ericsson, Montreal, Canada

    Dawei Ni

  2. Department of System and Computer Engineering, Carleton University, Ottawa, Canada

    Howard M. Schwartz

Authors
  1. Dawei Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Howard M. Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dawei Ni.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ni, D., Schwartz, H.M. Enhancing Learning Efficiency in FACL: A Novel Fuzzy Rule Transfer Method for Transfer Learning. Int. J. Fuzzy Syst. 26, 1215–1232 (2024). https://doi.org/10.1007/s40815-023-01662-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40815-023-01662-3

Keywords

Search

Navigation