Neural Processing Letters

, Volume 44, Issue 1, pp 125–147 | Cite as

A Cognitive Architecture Based on a Learning Classifier System with Spiking Classifiers

  • David Howard
  • Larry Bull
  • Pier-Luca Lanzi


Learning classifier systems (LCS) are population-based reinforcement learners that were originally designed to model various cognitive phenomena. This paper presents an explicitly cognitive LCS by using spiking neural networks as classifiers, providing each classifier with a measure of temporal dynamism. We employ a constructivist model of growth of both neurons and synaptic connections, which permits a genetic algorithm to automatically evolve sufficiently-complex neural structures. The spiking classifiers are coupled with a temporally-sensitive reinforcement learning algorithm, which allows the system to perform temporal state decomposition by appropriately rewarding “macro-actions”, created by chaining together multiple atomic actions. The combination of temporal reinforcement learning and neural information processing is shown to outperform benchmark neural classifier systems, and successfully solve a robotic navigation task.


Learning classifier systems Spiking neural networks  Self-adaptation Semi-MDP 


  1. 1.
    Beer RD (1995) On the dynamics of small continuous-time recurrent neural networks. Adapt Behav 3:469–509. doi: 10.1177/105971239500300405 CrossRefGoogle Scholar
  2. 2.
    Bonarini A (1998) Reinforcement distribution to fuzzy classifiers. In: Proceedings of the IEEE world congress on computational intelligence (WCCI)—evolutionary computation, IEEE Computer Press, pp 51–56Google Scholar
  3. 3.
    Boyan JA, Moore AW (1995) Generalization in reinforcement learning: safely approximating the value function. In: Tesauro G, Touretzky DS, Leen TK (eds) Advances in neural information processing systems 7. The MIT Press, Cambridge, pp 369–376Google Scholar
  4. 4.
    Bull L (2002) On using constructivism in neural classifier systems. In: Merelo J, Adamidis P, Beyer HG, Fernandez-Villacanas JL, Schwefel HP (eds) Parallel problem solving from nature—PPSN VII. Springer, New York, pp 558–567CrossRefGoogle Scholar
  5. 5.
    Bull L, Hurst J (2003) A neural learning classifier system with self-adaptive constructivism. In: Proceedings of the IEEE congress on evolutionary computation, IEEE Press, pp 991–997Google Scholar
  6. 6.
    Butz MV, Herbort O (2008) Context-dependent predictions and cognitive arm control with XCSF. In: Ryan C, Keijzer M (eds) Proceedings of the genetic and evolutionary computation conference, GECCO 2008, ACM, Atlanta, 12–16 July 2008, pp 1357–1364Google Scholar
  7. 7.
    Butz MV, Lanzi PL, Wilson SW (2006) Hyper-ellipsoidal conditions in xcs: rotation, linear approximation, and solution structure. In: Proceedings of the 8th annual conference on genetic and evolutionary computation (GECCO ’06), ACM Press, New York, pp 1457–1464. doi: 10.1145/1143997.1144237
  8. 8.
    Buzsaki G (2006) Rhythms of the brain. Oxford University Press, OxfordCrossRefzbMATHGoogle Scholar
  9. 9.
    Cazangi RR, Zuben FJV, Figueiredo M (2003) A classifier system in real applications for robot navigation. In: Proceedings of the IEEE congress on evolutionary computation, IEEE, pp 574–580Google Scholar
  10. 10.
    Churchill AW, Fernando C (2014) An evolutionary cognitive architecture made of a bag of networks. Evol Intell 7(3):169–182CrossRefGoogle Scholar
  11. 11.
    Donnart JY, Meyer JA (1996) Learning reactive and planning rules in a motivationally autonomous animat. IEEE Trans Syst Man Cybern 26(3):381–395CrossRefGoogle Scholar
  12. 12.
    Dorigo M, Colombetti M (1994) Robot shaping: developing autonomous agents through learning. Artif Intell 71(2):321–370CrossRefGoogle Scholar
  13. 13.
    Fauer S, Schwenker F (2015) Neural network ensembles in reinforcement learning. Neural Process Lett 41(1):55–69. doi: 10.1007/s11063-013-9334-5 CrossRefGoogle Scholar
  14. 14.
    Fernando C (2011) Symbol manipulation and rule learning in spiking neuronal networks. J theor biol 275(1):29–41MathSciNetCrossRefGoogle Scholar
  15. 15.
    Fernando C, Goldstein R, Szathmáry E (2010) The neuronal replicator hypothesis. Neural Comput 22(11):2809–2857MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Floreano D, Mattiussi C (2001) Evolution of spiking neural controllers for autonomous vision-based robots. Lect Notes Comput Sci 2217:38–61CrossRefzbMATHGoogle Scholar
  17. 17.
    Floreano D, Schoeni N, Caprari G, Blynel J (2002) Evolutionary bitsnspikes. In: Proceedings of the eight international conference on artificial life, MIT PressGoogle Scholar
  18. 18.
    Gerstner W, Kistler W (2002) Spiking neuron models—single neurons, populations, plasticity. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
  19. 19.
    Hagras H, Sobh T (2002) Intelligent learning and control of autonomous robotic agents operating in unstructured environments. Inf Sci 145(1):1–12MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    He P, Jagannathan S (2007) Reinforcement learning neural-network-based controller for nonlinear discrete-time systems with input constraints. In: Proceedings of the IEEE transactions on systems, man, and cybernetics, Part B: cybernetics, 37(2):425–436Google Scholar
  21. 21.
    Hodgkin AL, Huxley AF (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol 117(4):500CrossRefGoogle Scholar
  22. 22.
    Holland JH (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann ArborGoogle Scholar
  23. 23.
    Holland JH (1976) Adaptation. In: Rosen R, Snell F (eds) Progress in theoretical biology. Academic Press, New YorkGoogle Scholar
  24. 24.
    Holland JH, Reitman JS (1978) Cognitive systems based on adaptive algorithms. In: Waterman DA, Hayes-Roth F (eds) Pattern-directed inference systems. Academic Press, Orlando, pp 313–329Google Scholar
  25. 25.
    Howard G, Bull L, Lanzi PL (2010) A spiking neural representation for xcsf. In: Proceedings of the IEEE congress on evolutionary computation (CEC), IEEE, pp 1–8Google Scholar
  26. 26.
    Howard GD, Bull L (2008) On the effects of node duplication and connection-oriented constructivism in neural XCSF. In: Ryan C, Keijzer M (eds) In: Proceedings of the genetic and evolutionary computation conference, GECCO 2008, Atlanta, Companion Material, ACM, 12–16 July 2008, pp 1977–1984Google Scholar
  27. 27.
    Howard GD, Bull L, Lanzi PL (2009) Towards continuous actions in continuous space and time using self-adaptive constructivism in neural XCSF. In: Proceedings of the 11th annual conference on genetic and evolutionary computation, GECCO ’09, ACM, New York, pp 1219–1226. doi: 10.1145/1569901.1570065
  28. 28.
    Hurst J, Bull L (2006) A neural learning classifier system with self-adaptive constructivism for mobile robot control. Artif Life 12(3):353–380CrossRefGoogle Scholar
  29. 29.
    Hurst J, Bull L, Melhuish C (2002) TCS learning classifier system controller on a real robot. Lect Notes Comput Sci 2439:588–600CrossRefGoogle Scholar
  30. 30.
    Kistler WM (2002) Spike-timing dependent synaptic plasticity: a phenomenological framework. Biol Cybern 87(5–6):416–427. doi: 10.1007/s00422-002-0359-5 CrossRefzbMATHGoogle Scholar
  31. 31.
    Lanzi P, Loiacono D (2006) Xcsf with neural prediction. In: Yen GG, Lucas SM, Fogel G, Kendall G, Salomon R, Zhang BT, Coello CAC, Runarsson TP (eds) Proceedings of the 2006 IEEE congress on evolutionary computation, IEEE Press, Vancouver, Canada, pp 2270–2276. URL
  32. 32.
    Lanzi PL, Loiacono D, Wilson SW, Goldberg DE (2005) XCS with computed prediction in continuous multistep environments. In: Proceedings of the IEEE congress on evolutionary computation, IEEE, pp 2032–2039Google Scholar
  33. 33.
    Lanzi PL, Loiacono D, Wilson SW, Goldberg DE (2006) Classifier prediction based on tile coding. In: Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO ’06, ACM, New York, pp 1497–1504Google Scholar
  34. 34.
    Maass W (1997) Networks of spiking neurons: the third generation of neural network models. Neural Netw 10(9):1659–1671CrossRefGoogle Scholar
  35. 35.
    Michel O (2004) WebotsTM: professional mobile robot simulation. Int J Adv Robot Syst 1(1):39–42Google Scholar
  36. 36.
    Moioli RC, Vargas PA, Zuben FJV (2007) Analysing learning classifier systems in reactive and non-reactive robotic tasks. In: Bacardit J, Bernadó-Mansilla E, Butz MV, Kovacs T, Llorà X, Takadama K (eds) International workshop on learning classifier systems IWLCS, lecture notes in computer science, Springer, New York, vol 4998, pp 286–305Google Scholar
  37. 37.
    Pipe AG, Carse B (2002) First results from experiments in fuzzy classifier system architectures for mobile robotics. Lect Notes Comput Sci 2439:578–587CrossRefGoogle Scholar
  38. 38.
    Preen R, Bull L (2014) Discrete and fuzzy dynamical genetic programming in the xcsf learning classifier system. Soft Comput 18(1):153–167. doi: 10.1007/s00500-013-1044-4 CrossRefGoogle Scholar
  39. 39.
    Quartz SR, Sejnowski TJ (1997) The neural basis of cognitive development: a constructivist manifesto. Behav Brain Sci 20(04):537–556Google Scholar
  40. 40.
    Rechenberg I (1973) Evolutionsstrategie: optimierung technischer systeme nach prinzipien der biologischen evolution. Frommann-Holzboog, StuttgartGoogle Scholar
  41. 41.
    Rumelhart D, McClelland J (1986) Parallel distributed processing, vol 1 & 2. MIT Press, CambridgeGoogle Scholar
  42. 42.
    Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80(1):1–27Google Scholar
  43. 43.
    Shouval H, Gavornik J (2011) A single spiking neuron that can represent interval timing: analysis, plasticity and multi-stability. J Comput Neurosci 30(2):489–499MathSciNetCrossRefGoogle Scholar
  44. 44.
    Stolzmann W (1999) Latent learning in khepera robots with anticipatory classifier systems. In: Lanzi PL, Stolzmann W, Wilson SW (eds) 2nd international workshop on learning classifier systems. Orlando, pp 290–297Google Scholar
  45. 45.
    Studley M, Bull L (2005) X-TCS: accuracy-based learning classifier system robotics. In: Proceedings of the IEEE congress on evolutionary computation, IEEE, pp 2099–2106Google Scholar
  46. 46.
    Sutton RS (1996) Generalization in reinforcement learning: successful examples using sparse coarse coding. In: Touretzky DS, Mozer MC, Hasselmo ME (eds) Advances in neural information processing systems 8. MIT Press, Cambridge, pp 1038–1044Google Scholar
  47. 47.
    Sutton RS, Precup D, Singh S (1999) Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artif Intell 112(1–2):181–211MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Watkins C (1989) Learning from delayed rewards. PhD thesis, Cambridge University, Psychology Department, CambridgeGoogle Scholar
  49. 49.
    Webb A, Hart E, Ross P, Lawson A (2003) Controlling a simulated khepera with an XCS classifier system with memory. In: Banzhaf W, Christaller T, Dittrich P, Kim JT, Ziegler J (eds) Proceedings of the advances in artificial life, 7th European conference, ECAL 2003, lecture notes in computer science, vol 2801, Springer, Dortmund, pp 885–892, 14–17 Sept 2003Google Scholar
  50. 50.
    Wilson SW (2000) Get real! xcs with continuous-valued inputs. In: Lanzi PL, Stolzmann W, Wilson SW (eds) Learning classifier systems, from foundations to applications, LNAI-1813. Springer, New York, pp 209–219Google Scholar
  51. 51.
    Wilson SW (2001) Function approximation with a classifier system. In: Spector L, Goodman ED, Wu A, Langdon WB, Voigt HM, Gen M, Sen S, Dorigo M, Pezeshk S, Garzon MH, Burke E (eds) Proceedings of the genetic and evolutionary computation conference (GECCO-2001). Morgan Kaufmann, San Francisco, pp 974–981Google Scholar
  52. 52.
    Wilson SW (2001b) Mining oblique data with XCS. In: Lanzi PL, Stolzmann W, Wilson SW (eds) Advances in learning classifier systems, third international workshop, IWLCS 2000, LNCS, vol 1996, Springer, Heidelberg, pp 158–176Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Autonomous Systems ProgramQueensland Centre for Advanced TechnologyPullenvaleAustralia
  2. 2.Faculty of Environment and TechnologyUniversity of the West of EnglandBristolUK
  3. 3.Dipartimento di Elettronica e InformazionePolitecnico di MilanoMilanoItaly

Personalised recommendations