Skip to main content

Advertisement

SpringerLink
Log in

A Cognitive Architecture Based on a Learning Classifier System with Spiking Classifiers

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Learning classifier systems (LCS) are population-based reinforcement learners that were originally designed to model various cognitive phenomena. This paper presents an explicitly cognitive LCS by using spiking neural networks as classifiers, providing each classifier with a measure of temporal dynamism. We employ a constructivist model of growth of both neurons and synaptic connections, which permits a genetic algorithm to automatically evolve sufficiently-complex neural structures. The spiking classifiers are coupled with a temporally-sensitive reinforcement learning algorithm, which allows the system to perform temporal state decomposition by appropriately rewarding “macro-actions”, created by chaining together multiple atomic actions. The combination of temporal reinforcement learning and neural information processing is shown to outperform benchmark neural classifier systems, and successfully solve a robotic navigation task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Beer RD (1995) On the dynamics of small continuous-time recurrent neural networks. Adapt Behav 3:469–509. doi:10.1177/105971239500300405

    Article  Google Scholar 

  2. Bonarini A (1998) Reinforcement distribution to fuzzy classifiers. In: Proceedings of the IEEE world congress on computational intelligence (WCCI)—evolutionary computation, IEEE Computer Press, pp 51–56

  3. Boyan JA, Moore AW (1995) Generalization in reinforcement learning: safely approximating the value function. In: Tesauro G, Touretzky DS, Leen TK (eds) Advances in neural information processing systems 7. The MIT Press, Cambridge, pp 369–376

    Google Scholar 

  4. Bull L (2002) On using constructivism in neural classifier systems. In: Merelo J, Adamidis P, Beyer HG, Fernandez-Villacanas JL, Schwefel HP (eds) Parallel problem solving from nature—PPSN VII. Springer, New York, pp 558–567

    Chapter  Google Scholar 

  5. Bull L, Hurst J (2003) A neural learning classifier system with self-adaptive constructivism. In: Proceedings of the IEEE congress on evolutionary computation, IEEE Press, pp 991–997

  6. Butz MV, Herbort O (2008) Context-dependent predictions and cognitive arm control with XCSF. In: Ryan C, Keijzer M (eds) Proceedings of the genetic and evolutionary computation conference, GECCO 2008, ACM, Atlanta, 12–16 July 2008, pp 1357–1364

  7. Butz MV, Lanzi PL, Wilson SW (2006) Hyper-ellipsoidal conditions in xcs: rotation, linear approximation, and solution structure. In: Proceedings of the 8th annual conference on genetic and evolutionary computation (GECCO ’06), ACM Press, New York, pp 1457–1464. doi:10.1145/1143997.1144237

  8. Buzsaki G (2006) Rhythms of the brain. Oxford University Press, Oxford

    Book  MATH  Google Scholar 

  9. Cazangi RR, Zuben FJV, Figueiredo M (2003) A classifier system in real applications for robot navigation. In: Proceedings of the IEEE congress on evolutionary computation, IEEE, pp 574–580

  10. Churchill AW, Fernando C (2014) An evolutionary cognitive architecture made of a bag of networks. Evol Intell 7(3):169–182

    Article  Google Scholar 

  11. Donnart JY, Meyer JA (1996) Learning reactive and planning rules in a motivationally autonomous animat. IEEE Trans Syst Man Cybern 26(3):381–395

    Article  Google Scholar 

  12. Dorigo M, Colombetti M (1994) Robot shaping: developing autonomous agents through learning. Artif Intell 71(2):321–370

    Article  Google Scholar 

  13. Fauer S, Schwenker F (2015) Neural network ensembles in reinforcement learning. Neural Process Lett 41(1):55–69. doi:10.1007/s11063-013-9334-5

    Article  Google Scholar 

  14. Fernando C (2011) Symbol manipulation and rule learning in spiking neuronal networks. J theor biol 275(1):29–41

    Article  MathSciNet  Google Scholar 

  15. Fernando C, Goldstein R, Szathmáry E (2010) The neuronal replicator hypothesis. Neural Comput 22(11):2809–2857

    Article  MathSciNet  MATH  Google Scholar 

  16. Floreano D, Mattiussi C (2001) Evolution of spiking neural controllers for autonomous vision-based robots. Lect Notes Comput Sci 2217:38–61

    Article  MATH  Google Scholar 

  17. Floreano D, Schoeni N, Caprari G, Blynel J (2002) Evolutionary bitsnspikes. In: Proceedings of the eight international conference on artificial life, MIT Press

  18. Gerstner W, Kistler W (2002) Spiking neuron models—single neurons, populations, plasticity. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  19. Hagras H, Sobh T (2002) Intelligent learning and control of autonomous robotic agents operating in unstructured environments. Inf Sci 145(1):1–12

    Article  MathSciNet  MATH  Google Scholar 

  20. He P, Jagannathan S (2007) Reinforcement learning neural-network-based controller for nonlinear discrete-time systems with input constraints. In: Proceedings of the IEEE transactions on systems, man, and cybernetics, Part B: cybernetics, 37(2):425–436

  21. Hodgkin AL, Huxley AF (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol 117(4):500

    Article  Google Scholar 

  22. Holland JH (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann Arbor

    Google Scholar 

  23. Holland JH (1976) Adaptation. In: Rosen R, Snell F (eds) Progress in theoretical biology. Academic Press, New York

    Google Scholar 

  24. Holland JH, Reitman JS (1978) Cognitive systems based on adaptive algorithms. In: Waterman DA, Hayes-Roth F (eds) Pattern-directed inference systems. Academic Press, Orlando, pp 313–329

    Google Scholar 

  25. Howard G, Bull L, Lanzi PL (2010) A spiking neural representation for xcsf. In: Proceedings of the IEEE congress on evolutionary computation (CEC), IEEE, pp 1–8

  26. Howard GD, Bull L (2008) On the effects of node duplication and connection-oriented constructivism in neural XCSF. In: Ryan C, Keijzer M (eds) In: Proceedings of the genetic and evolutionary computation conference, GECCO 2008, Atlanta, Companion Material, ACM, 12–16 July 2008, pp 1977–1984

  27. Howard GD, Bull L, Lanzi PL (2009) Towards continuous actions in continuous space and time using self-adaptive constructivism in neural XCSF. In: Proceedings of the 11th annual conference on genetic and evolutionary computation, GECCO ’09, ACM, New York, pp 1219–1226. doi:10.1145/1569901.1570065

  28. Hurst J, Bull L (2006) A neural learning classifier system with self-adaptive constructivism for mobile robot control. Artif Life 12(3):353–380

    Article  Google Scholar 

  29. Hurst J, Bull L, Melhuish C (2002) TCS learning classifier system controller on a real robot. Lect Notes Comput Sci 2439:588–600

    Article  Google Scholar 

  30. Kistler WM (2002) Spike-timing dependent synaptic plasticity: a phenomenological framework. Biol Cybern 87(5–6):416–427. doi:10.1007/s00422-002-0359-5

    Article  MATH  Google Scholar 

  31. Lanzi P, Loiacono D (2006) Xcsf with neural prediction. In: Yen GG, Lucas SM, Fogel G, Kendall G, Salomon R, Zhang BT, Coello CAC, Runarsson TP (eds) Proceedings of the 2006 IEEE congress on evolutionary computation, IEEE Press, Vancouver, Canada, pp 2270–2276. URL http://ieeexplore.ieee.org/servlet/opac?punumber=11108

  32. Lanzi PL, Loiacono D, Wilson SW, Goldberg DE (2005) XCS with computed prediction in continuous multistep environments. In: Proceedings of the IEEE congress on evolutionary computation, IEEE, pp 2032–2039

  33. Lanzi PL, Loiacono D, Wilson SW, Goldberg DE (2006) Classifier prediction based on tile coding. In: Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO ’06, ACM, New York, pp 1497–1504

  34. Maass W (1997) Networks of spiking neurons: the third generation of neural network models. Neural Netw 10(9):1659–1671

    Article  Google Scholar 

  35. Michel O (2004) WebotsTM: professional mobile robot simulation. Int J Adv Robot Syst 1(1):39–42

    Google Scholar 

  36. Moioli RC, Vargas PA, Zuben FJV (2007) Analysing learning classifier systems in reactive and non-reactive robotic tasks. In: Bacardit J, Bernadó-Mansilla E, Butz MV, Kovacs T, Llorà X, Takadama K (eds) International workshop on learning classifier systems IWLCS, lecture notes in computer science, Springer, New York, vol 4998, pp 286–305

  37. Pipe AG, Carse B (2002) First results from experiments in fuzzy classifier system architectures for mobile robotics. Lect Notes Comput Sci 2439:578–587

    Article  Google Scholar 

  38. Preen R, Bull L (2014) Discrete and fuzzy dynamical genetic programming in the xcsf learning classifier system. Soft Comput 18(1):153–167. doi:10.1007/s00500-013-1044-4

    Article  Google Scholar 

  39. Quartz SR, Sejnowski TJ (1997) The neural basis of cognitive development: a constructivist manifesto. Behav Brain Sci 20(04):537–556

    Google Scholar 

  40. Rechenberg I (1973) Evolutionsstrategie: optimierung technischer systeme nach prinzipien der biologischen evolution. Frommann-Holzboog, Stuttgart

    Google Scholar 

  41. Rumelhart D, McClelland J (1986) Parallel distributed processing, vol 1 & 2. MIT Press, Cambridge

    Google Scholar 

  42. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80(1):1–27

    Google Scholar 

  43. Shouval H, Gavornik J (2011) A single spiking neuron that can represent interval timing: analysis, plasticity and multi-stability. J Comput Neurosci 30(2):489–499

    Article  MathSciNet  Google Scholar 

  44. Stolzmann W (1999) Latent learning in khepera robots with anticipatory classifier systems. In: Lanzi PL, Stolzmann W, Wilson SW (eds) 2nd international workshop on learning classifier systems. Orlando, pp 290–297

  45. Studley M, Bull L (2005) X-TCS: accuracy-based learning classifier system robotics. In: Proceedings of the IEEE congress on evolutionary computation, IEEE, pp 2099–2106

  46. Sutton RS (1996) Generalization in reinforcement learning: successful examples using sparse coarse coding. In: Touretzky DS, Mozer MC, Hasselmo ME (eds) Advances in neural information processing systems 8. MIT Press, Cambridge, pp 1038–1044

    Google Scholar 

  47. Sutton RS, Precup D, Singh S (1999) Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artif Intell 112(1–2):181–211

    Article  MathSciNet  MATH  Google Scholar 

  48. Watkins C (1989) Learning from delayed rewards. PhD thesis, Cambridge University, Psychology Department, Cambridge

  49. Webb A, Hart E, Ross P, Lawson A (2003) Controlling a simulated khepera with an XCS classifier system with memory. In: Banzhaf W, Christaller T, Dittrich P, Kim JT, Ziegler J (eds) Proceedings of the advances in artificial life, 7th European conference, ECAL 2003, lecture notes in computer science, vol 2801, Springer, Dortmund, pp 885–892, 14–17 Sept 2003

  50. Wilson SW (2000) Get real! xcs with continuous-valued inputs. In: Lanzi PL, Stolzmann W, Wilson SW (eds) Learning classifier systems, from foundations to applications, LNAI-1813. Springer, New York, pp 209–219

    Google Scholar 

  51. Wilson SW (2001) Function approximation with a classifier system. In: Spector L, Goodman ED, Wu A, Langdon WB, Voigt HM, Gen M, Sen S, Dorigo M, Pezeshk S, Garzon MH, Burke E (eds) Proceedings of the genetic and evolutionary computation conference (GECCO-2001). Morgan Kaufmann, San Francisco, pp 974–981

  52. Wilson SW (2001b) Mining oblique data with XCS. In: Lanzi PL, Stolzmann W, Wilson SW (eds) Advances in learning classifier systems, third international workshop, IWLCS 2000, LNCS, vol 1996, Springer, Heidelberg, pp 158–176

Download references

Author information

Authors and Affiliations

  1. Autonomous Systems Program, Queensland Centre for Advanced Technology, Pullenvale, Australia

    David Howard

  2. Faculty of Environment and Technology, University of the West of England, Bristol, UK

    Larry Bull

  3. Dipartimento di Elettronica e Informazione, Politecnico di Milano, Milano, Italy

    Pier-Luca Lanzi

Authors
  1. David Howard
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Larry Bull
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pier-Luca Lanzi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Howard.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Howard, D., Bull, L. & Lanzi, PL. A Cognitive Architecture Based on a Learning Classifier System with Spiking Classifiers. Neural Process Lett 44, 125–147 (2016). https://doi.org/10.1007/s11063-015-9451-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-015-9451-4

Keywords

Search

Navigation