Skip to main content

Advertisement

SpringerLink
Log in

Combining NEAT and PSO for learning tactical human behavior

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This article presents and discusses a machine learning algorithm called PIGEON used to build agents capable of displaying tactical behavior in various domains. Such tactical behavior can be relevant in military simulations and video games, as well as in everyday tasks in the physical world, such as driving an automobile. Furthermore, PIGEON displays good performance across two different approaches to learning (observational and experiential) and across multiple domains. PIGEON is a hybrid algorithm, combining NEAT and PSO in two different manners. The investigation described in this paper compares the performance of the two versions of PIGEON to each other as well as to NEAT and to PSO individually. These four machine learning algorithms are applied in two different approaches to learning—through observation of human performance and through experience, as well as in three distinct domain testbeds. The criteria used to compare them were high proficiency in task completion and rapid learning. Results indicate that overall, PIGEON worked best when NEAT and PSO are applied in an alternating manner. This combination was called PIGEON-Alternate, or simply Alternate. The two versions of the PIGEON algorithm, the tests conducted, the results obtained and the conclusions are described in detail.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Gonzalez AJ, Georgiopoulos M, DeMara RF, Henninger AE, Gerber W (1998) Automating the CGF model development and refinement process by observing expert behavior in a simulation. In: Proceedings of the computer generated forces conference

  2. Gonzalez AJ, Gerber W, Castro J (2002) Automated acquisition of tactical knowledge through contextualization. In: Proceedings of the 2002 conference on computer generated forces and behavior representation. Orlando, FL, May 2002

  3. Fernlund H, Gonzalez AJ (2002) An Approach towards building human behavior models automatically by observation. The First Swedish-American Workshop on Modeling and Simulation, January 2002

  4. Dejong G, Mooney R (1986) Explanation-based learning: an alternative view. Mach Learn 1(2):145–176

    Google Scholar 

  5. Yang B, Asada H (1992) Hybrid linguistic/numeric control of deburring robots based on human skills. In: Proceedings of the IEEE international conference on robotics and automation, pp 1467–1474

  6. Lee S, Shimoji S (1991) Machine acquisition of skills by neural networks. In: International joint conference on neural networks, vol 2

  7. Lee S, Chen J (1994) Skill learning from observations. In: Proceedings of the IEEE international conference on robotics and automation, pp 3245–3250

  8. Kaiser M, Dillmann R (1996) Building elementary robot skills from human demonstration. Proc IEEE Int Conf Robot Autom 3:2700–2705

    Article  Google Scholar 

  9. Atkeson C, Hale J, Pollick F, Riley M, Kotosaka S, Schaul S, Shibata T, Tevatia G, Ude A, Vijayakumar S et al (2000) Using humanoid robots to study human behavior. IEEE Conf Intell Syst Appl 15(4):46–56

    Article  Google Scholar 

  10. Morales E, Sammut C (2004) Learning to fly by combining reinforcement learning with behavioural cloning. In: ACM international conference proceeding series

  11. Baird L, Deacon S, Holland P (2000) From action learning to learning from action: Implementing the after action review. In: Cross R, Israelit S (eds) Strategic learning in a knowledge economy: individual, collective and organizational learning process. Butterworth-Heinemann, Woburn, pp 185–202

  12. Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of the IEEE international conference on neural networks, vol 4

  13. Stanley K, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evolut Comput 10(2):99–127

    Article  Google Scholar 

  14. Stein G, Gonzalez AJ (2011) Building high-performing human-like tactical agents through observation and experience. IEEE Trans Syst Man Cybern Part B Cybern 31(3):792–804

    Article  Google Scholar 

  15. Sammut C, Hurst S, Kedzier D, Michie D (1992) Learning to fly. In: Proceedings of the ninth international conference on machine learning

  16. Isaac A, Sammut C (2003) Goal-directed learning to fly. In: Proceedings of the 20th international conference of machine learning, pp 258–265

  17. Harries M, Sammut C, Horn K (1998) Extracting hidden context. Mach Learn 32(2):101–126

    Article  MATH  Google Scholar 

  18. Fix E (1990) Neural network based human performance modeling. In: Proceedings of the IEEE 1990 national aerospace and electronics conference, pp 1162–1165

  19. Sidani T, Gonzalez AJ (2000) A framework for learning implicit expert knowledge through observation. Trans Soc Comput Simul 17(2):54–72

    Google Scholar 

  20. Stanley K, Kohl N, Sherony R, Miikkulainen R (2005) Neuroevolution of an automobile crash warning system. In: Proceedings of the 2005 conference on genetic and evolutionary computation, pp 1977–1984

  21. Pomerleau D (1991) Efficient training of artificial neural networks for autonomous navigation. Neural Comput 3(1):88–97

    Article  Google Scholar 

  22. Fernlund H, Gonzalez A, Georgiopoulos M, DeMara RF (2006) Learning tactical human behavior through observation of human performance. IEEE Trans Syst Man Cybern Part B Cybern 36(1):128–140

    Article  Google Scholar 

  23. Trinh V, Gonzalez AJ (2013) Discovering contexts from observed performance. IEEE Trans Hum Mach Syst 439(4):359–370

    Article  Google Scholar 

  24. Watkins C, Dayan P (1992) Q-learning. Mach Learn 8(3):279–292

    MATH  Google Scholar 

  25. Sutton R (1988) Learning to predict by the methods of temporal differences. Mach Learn 3(1):9–44

    Google Scholar 

  26. Holland J (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor

    Google Scholar 

  27. Koza J (1989) Hierarchical genetic algorithms operating on populations of computer programs. In: Proceedings of the 11th international joint conference on artificial intelligence, vol 1. pp 768–774

  28. Dorigo M, Maniezzo V, Colorni A (1996) Ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern Part B 26(1):29–41

    Article  Google Scholar 

  29. Miller GF, Todd PM, Hegde SU (1989) Designing neural networks using genetic algorithms. In: Proceedings of the 3rd international conference on genetic algorithms and their applications, Morgan Kaufmann, San Mateo, CA, pp 379–384

  30. Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447

    Article  Google Scholar 

  31. Angeline P, Saunders G, Pollack J (1994) An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans Neural Netw 5(1):54–65

    Article  Google Scholar 

  32. Reynolds C (1987) Flocks, herds and schools: a distributed behavioral model. ACM SIGGRAPH Comput Graphics 21(4):25–34

    Article  Google Scholar 

  33. Potter M, De Jong K (1995) Evolving neural networks with collaborative species. In: Proceedings of the 1995 summer computer simulation conference, pp 340–345

  34. Gruau F, Whitley D, Pyeatt L (1996) A comparison between cellular encoding and direct encoding for genetic neural networks. In: Proceedings of the first annual conference on genetic programming. MIT Press, pp 81–89

  35. Gudise V, Venayagamoorthy GK (2003) Comparison of particle swarm optimization and backpropagation as training algorithms for neural networks. In: Proceedings of the 2003 IEEE swarm intelligence symposium, pp 110–117

  36. Zhang C, Shao H, Li Y (2000) Particle swarm optimisation for evolving artificial neural network. In: IEEE international conference on systems, man, and cybernetics. vol 4

  37. Chen Y, Yang B, Dong J (2004) Evolving Flexible neural networks using ant programming and PSO algorithm. Int Symp Neural Netw 3173:211–216

    Google Scholar 

  38. Yao X, Liu Y (1997) A new evolutionary system for evolving artificial neural networks. IEEE Trans Neural Netw 8(3):694–713

    Article  MathSciNet  Google Scholar 

  39. Conforth M, Meng Y (2008) Toward evolving neural networks using bio-inspired algorithms. In: Proceedings of the international conference on artificial intelligence, pp 413–419

  40. Ismail A, Jeng D-S (2012) Self-evolving neural networks based on PSO and JPSO algorithms. World Academy of Science, Engineering and Technology, vol 6, pp 9–22

  41. Ismail A, Jeng D-S (2013) SEANN: a self-evolving neural network based on PSO and JPSO algorithms. J Hybrid Technol (JHT) 1(1):17–29

  42. Geetha T (2013) A diverse versions and variations of particle swarm optimization (PSO) algorithm for real world complex optimization problems. Int J Emerg Technol Adv Eng 3(8):454–459

  43. Zhao J, Peng G (2011) NEAT versus PSO for evolving autonomous multi-agents coordination on pursuit-evasion problem. Inf Control Autom Robot 2(LNEE 133):711–717

  44. Stanley K (2004) Efficient evolution of neural networks through complexification. Ph.D. dissertation, Univ. of Texas, Austin

  45. Panksepp J (1998) Affective neuroscience: the foundations of human and animal emotions. Oxford University Press, Oxford

    Google Scholar 

  46. Koza J (1989) Hierarchical genetic algorithms operating on populations of computer programs. In: Proceedings of the 11th international joint conference on artificial intelligence, vol 1. pp 768–774

  47. Wieland A (1991) Evolving neural network controllers for unstable systems. In: International joint conference on neural networks, vol 2

  48. Suganthan P (1999) Particle swarm optimiser with neighbourhood operator. In: Proceedings of the 1999 congress on evolutionary computation, vol 3

  49. Angeline P (1998) Evolutionary optimization versus particle swarm optimization: philosophy and performance differences. Evolut Program 7:601–610

    Google Scholar 

  50. Yao X (1993) Review of evolutionary artificial neural networks. Int J Intell Syst 8(4):539–567

    Article  Google Scholar 

  51. Stein G (2009) FALCONET: force-feedback approach for learning from coaching and observation using natural and experiential training. PhD Dissertation. University of Central Florida

  52. Khaw J, Lim B, Lim L (1995) Optimal design of neural networks using the Taguchi method. Neurocomputing 7(3):225–245

    Article  MATH  Google Scholar 

  53. Paquet U, Engelbrecht AP (2003) A new particle swarm optimiser for linearly constrained optimisation. In: The 2003 IEEE congress on evolutionary computation, CEC'03, vol 1, pp 227–233

  54. Liang JJ, Suganthan PN (2005) Dynamic multi-swarm particle swarm optimizer. In: Proceedings of the 2005 IEEE swarm intelligence symposium

  55. Arumugam MS, Rao MVC, Chandramohan A (2008) A new and improved version of particle swarm optimization algorithm with global–local best parameters. Knowl Inf Syst 16(3):331–357

    Article  Google Scholar 

  56. Stanley K, Bryant B, Miikkulainen R (2005) Real-time neuroevolution in the NERO video game. IEEE Trans Evolut Comput 9(6):653–668

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Intelligent Systems Laboratory, Computer Science Division, University of Central Florida, Orlando, FL, USA

    Gary Stein, Avelino J. Gonzalez & Clayton Barham

Authors
  1. Gary Stein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Avelino J. Gonzalez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Clayton Barham
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Avelino J. Gonzalez.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stein, G., Gonzalez, A.J. & Barham, C. Combining NEAT and PSO for learning tactical human behavior. Neural Comput & Applic 26, 747–764 (2015). https://doi.org/10.1007/s00521-014-1761-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-014-1761-3

Keywords

Search

Navigation