Skip to main content
SpringerLink
Log in
Book cover

International Workshop on Learning Classifier Systems

International Workshop on Learning Classifier Systems

IWLCS 2006, IWLCS 2007: Learning Classifier Systems pp 169–188Cite as

A Learning Classifier System Approach to Relational Reinforcement Learning

  • Conference paper

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4998))

Abstract

This article describes a learning classifier system (LCS) approach to relational reinforcement learning (RRL). The system, Foxcs-2, is a derivative of Xcs that learns rules expressed as definite clauses over first-order logic. By adopting the LCS approach, Foxcs-2, unlike many RRL systems, is a general, model-free and “tabula rasa” system. The change in representation from bit-strings in Xcs to first-order logic in Foxcs-2 necessitates modifications, described within, to support matching, covering, mutation and several other functions. Evaluation on inductive logic programming (ILP) and RRL tasks shows that the performance of Foxcs-2 is comparable to other systems. Further evaluation on RRL tasks highlights a significant advantage of Foxcs-2’s rule language: in some environments it is able to represent policies that are genuinely scalable; that is, policies that are independent of the size of the environment.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bernadó, E., Llorà, X., Garrel, J.M.: XCS and GALE: A comparative study of two learning classifier systems on data mining. In: Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds.) IWLCS 2001. LNCS (LNAI), vol. 2321, pp. 115–132. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  2. Bernadó-Mansilla, E., Garrell-Guiu, J.M.: Accuracy-based learning classifier systems: Models, analysis and applications to classification tasks. Evolutionary Computation 11(3), 209–238 (2003)

    Article  Google Scholar 

  3. Beyer, H.-G., O’Reilly, U.-M. (eds.): Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2005. ACM Press, New York (2005)

    Google Scholar 

  4. Blockeel, H., Raedt, L.D.: Top-down induction of first-order logical decision trees. Artificial Intelligence 101(1–2), 285–297 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  5. Blockeel, H., Džeroski, S., Kompare, B., Kramer, S., Pfahringer, B., Laer, W.V.: Experiments in predicting biodegradability. Applied Artificial Intelligence 18(2), 157–181 (2004)

    Article  Google Scholar 

  6. Bull, L., O’Hara, T.: Accuracy-based neuro and neuro-fuzzy classifier systems. In: Langdon, et al. (eds.) [28], pp. 905–911

    Google Scholar 

  7. Butz, M.V.: Rule-based Evolutionary Online Learning Systems: Learning Bounds, Classification, and Prediction. PhD thesis, University of Illinois at Urbana-Champaign, 104 S. Mathews Avenue, Urbana, IL 61801, U.S.A (2004)

    Google Scholar 

  8. Martin, V.: Kernel-based, ellipsoidal conditions in the real-valued XCS classifier system. In: Beyer, O’Reilly (eds.) [3], pp. 1835–1842

    Google Scholar 

  9. Butz, M.V., Kovacs, T., Lanzi, P.L., Wilson, S.W.: Toward a theory of generalization and learning in XCS. IEEE Transactions on Evolutionary Computation 8(1), 28–46 (2004)

    Article  Google Scholar 

  10. Butz, M.V., Pelikan, M.: Analyzing the evolutionary pressures in XCS. In: Spector, et al. (eds.) [40], pp. 935–942

    Google Scholar 

  11. Butz, M.V., Sastry, K., Goldberg, D.E.: Strong, stable, and reliable fitness pressure in XCS due to tournament selection. Genetic Programming and Evolvable Machines 6(1), 53–77 (2005)

    Article  Google Scholar 

  12. Butz, M.V., Wilson, S.W.: An algorithmic description of XCS. In: Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds.) Advances in Learning Classifier Systems. Third International Workshop (IWLCS-2000), pp. 253–272. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  13. Casillas, J., Carse, B., Bull, L.: Fuzzy XCS: an accuracy-based fuzzy classifier system. In: Proceedings of the XII Congreso Espanol sobre Tecnologia y Logica Fuzzy (ESTYLF 2004), pp. 369–376 (2004)

    Google Scholar 

  14. Cole, J., Lloyd, J., Ng, K.S.: Symbolic learning for adaptive agents. In: Proceedings of the Annual Partner Conference, Smart Internet Technology Cooperative Research Centre (2003), http://users.rsise.anu.edu.au/~jwl/crc_paper.pdf

  15. Raedt, L.D., Laer, W.V.: Inductive constraint logic. In: Jantke, K.P., Shinohara, T., Zeugmann, T. (eds.) ALT 1995. LNCS, vol. 997, pp. 80–94. Springer, Heidelberg (1995)

    Chapter  Google Scholar 

  16. Divina, F.: Hybrid Genetic Relational Search for Inductive Learning. PhD thesis, Department of Computer Science, Vrije Universiteit, Amsterdam, The Netherlands (2004)

    Google Scholar 

  17. Divina, F., Marchiori, E.: Evolutionary concept learning. In: Langdon, et al. (eds.) [28], pp. 343–350

    Google Scholar 

  18. Driessens, K., Džeroski, S.: Combining model-based and instance-based learning for first order regression. In: Raedt, L.D., Wrobel, S. (eds.) Proceedings of the Twenty-Second International Conference on Machine Learning (ICML 2005). ACM International Conference Proceeding Series, vol. 119, pp. 193–200. ACM Press, New York (2005)

    Google Scholar 

  19. Driessens, K., Ramon, J.: Relational instance based regression for relational reinforcement learning. In: Fawcett, T., Mishra, N. (eds.) Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), pp. 123–130. AAAI Press, Menlo Park (2003)

    Google Scholar 

  20. Driessens, K., Ramon, J., Blockeel, H.: Speeding up relational reinforcement learning through the use of an incremental first order decision tree learner. In: Raedt, L.D., Flach, P. (eds.) Proceedings of the 12th European Conference on Machine Learning, pp. 97–108. Springer, Heidelberg (2001)

    Google Scholar 

  21. Džeroski, S., Raedt, L.D., Driessens, K.: Relational reinforcement learning. Machine Learning 43(1–2), 7–52 (2001)

    Article  MATH  Google Scholar 

  22. Džeroski, S., Jacobs, N., Molina, M., Moure, C., Muggleton, S., van Laer, W.: Detecting traffic problems with ILP. In: Page, D.L. (ed.) ILP 1998, vol. 1446, pp. 281–290. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  23. Gärtner, T., Driessens, K., Ramon, J.: Graph kernels and Gaussian processes for relational reinforcement learning. In: Horváth, T., Yamamoto, A. (eds.) ILP 2003. LNCS (LNAI), vol. 2835, pp. 146–163. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  24. Genesereth, M.R., Nilsson, N.J.: Logical Foundations of Artificial Intelligence. Morgan Kaufmann, San Francisco (1987)

    MATH  Google Scholar 

  25. Holland, J.H.: Adaptation. In: Rosen, R., Snell, F.M. (eds.) Progress in Theoretical Biology, vol. 4. Plenum, NY (1976)

    Google Scholar 

  26. Kovacs, T.: Towards a theory of strong overgeneral classifiers. In: Martin, W., Spears, W. (eds.) Foundations of Genetic Algorithms 6, pp. 165–184. Morgan Kaufmann, San Francisco (2001)

    Chapter  Google Scholar 

  27. Kovacs, T.: A Comparison of Strength and Accuracy-Based Fitness in Learning Classifier Systems. PhD thesis, School of Computer Science, University of Birmingham, UK (2002)

    Google Scholar 

  28. Langdon, W.B., Cantú-Paz, E., Mathias, K.E., Roy, R., Davis, D., Poli, R., Balakrishnan, K., Honavar, V., Rudolph, G., Wegener, J., Bull, L., Potter, M.A., Schultz, A.C., Miller, J.F., Burke, E.K., Jonoska, N.: GECCO 2002: Proceedings of the Genetic and Evolutionary Computation Conference, 9-13 July 2002. Morgan Kaufmann, San Francisco (2002)

    Google Scholar 

  29. Lanzi, P.L.: Extending the representation of classifer conditions, part II: From messy codings to S-expressions. In: Banzhaf, W., Daida, J., Eiben, A.E., Garzon, M.H., Honavar, V., Jakiela, M., Smith, R.E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 1999), pp. 345–352. Morgan Kaufmann, San Francisco (1999)

    Google Scholar 

  30. Lanzi, P.L.: Mining interesting knowledge from data with the XCS classifier system. In: Spector, et al. (eds.) [40], pp. 958–965

    Google Scholar 

  31. Lanzi, P.L., Loiacono, D., Wilson, S.W., Goldberg, D.E.: XCS with computed prediction in multistep environments. In: Beyer, O’Reilly (eds.) [3], pp. 1859–1866

    Google Scholar 

  32. Mellor, D.: A first order logic classifier system. In: Beyer, O’Reilly (eds.) [3], pp. 1819–1826

    Google Scholar 

  33. Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)

    MATH  Google Scholar 

  34. Muggleton, S.: Inductive Logic Programming. In: The MIT Encyclopedia of the Cognitive Sciences (MITECS). Academic Press, London (1992)

    Google Scholar 

  35. Muggleton, S.: Inverse entailment and Progol. New Generation Computing, Special issue on Inductive Logic Programming 13(3–4), 245–286 (1995)

    Article  Google Scholar 

  36. Nienhuys-Cheng, S.-H., de Wolf, R.: Foundations of Inductive Logic Programming. LNCS, vol. 1228. Springer, Heidelberg (1997)

    MATH  Google Scholar 

  37. Plotkin, G.D.: Automatic Methods of Inductive Inference. PhD thesis, Edinburgh University (1971)

    Google Scholar 

  38. Quinlan, J.R.: Learning logical definition from relations. Machine Learning 5(3), 239–266 (1990)

    Google Scholar 

  39. Slaney, J., Thiébaux, S.: Blocks World revisited. Artificial Intelligence 125, 119–153 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  40. Spector, L., Goodman, E.D., Wu, A., Langdon, W.B., Voigt, H.-M., Gen, M., Sen, S., Dorigo, M., Pezeshk, S., Garzon, M.H., Burke, E. (eds.): Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2001), July 7-11 2001. Morgan Kaufmann, San Francisco (2001)

    Google Scholar 

  41. Srinivasan, A., Muggleton, S., De King, R.: Comparing the use of background knowledge by inductive logic programming systems. In: Raedt, L.D. (ed.) Proceedings of the Fifth International Inductive Logic Programming Workshop, Katholieke Universteit, Leuven (1995); Withdrawn from publication and replaced by [42]

    Google Scholar 

  42. Srinivasan, A., King, R.D., Muggleton, S.: The role of background knowledge: using a problem from chemistry to examine the performance of an ILP program. Technical Report PRG-TR-08-99, Oxford University Computing Laboratory, Oxford, UK (1999)

    Google Scholar 

  43. Srinivasan, A., Muggleton, S., Sternberg, M.J.E., King, R.D.: Theories for mutagenicity: A study in first-order and feature-based induction. Artificial Intelligence 85(1-2), 277–299 (1996)

    Article  Google Scholar 

  44. Stone, C., Bull, L.: For real! XCS with continuous-valued inputs. Evolutionary Computation 11(3), 299–336 (2003)

    Article  Google Scholar 

  45. Tadepalli, P., Givan, R., Driessens, K.: Relational reinforcement learning: an overview. In: Tadepalli, P., Givan, R., Driessens, K. (eds.) Proceedings of the ICML2004 Workshop on Relational Reinforcement Learning, pp. 1–9 (2004), http://eecs.oregonstate.edu/research/rrl/index.html

  46. Van Laer, W.: From Propositional to First Order Logic in Machine Learning and Data Mining. PhD thesis, Katholieke Universiteit Leuven, Belgium (2002)

    Google Scholar 

  47. Van Laer, W., De Raedt, L.: How to upgrade propositional learners to first order logic: A case study. In: Paliouras, G., Karkaletsis, V., Spyropoulos, C.D. (eds.) ACAI 1999. LNCS (LNAI), vol. 2049, pp. 102–126. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  48. van Otterlo, M.: A survey of reinforcement learning in relational domains. Technical Report TR-CTIT-05-31, University of Twente, The Netherlands (2005)

    Google Scholar 

  49. Wilson, S.W.: Classifier fitness based on accuracy. Evolutionary Computation 3(2), 149–175 (1995)

    Article  Google Scholar 

  50. Wilson, S.W.: Generalization in the XCS classifier system. In: Koza, J.R., Banzhaf, W., Chellapilla, K., Deb, K., Dorigo, M., Fogel, D.B., Garzon, M.H., Goldberg, D.E., Iba, H., Riolo, R. (eds.) Genetic Programming 1998: Proceedings of the Third Annual Conference, University of Wisconsin, Madison, Wisconsin, USA, pp. 665–674. Morgan Kaufmann, San Francisco (1998)

    Google Scholar 

  51. Wilson, S.W.: Get real! XCS with continuous-valued inputs. In: Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds.) IWLCS 1999. LNCS (LNAI), vol. 1813, pp. 209–222. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical Engineering and Computer Science, The University of Newcastle, Callaghan, 2308, Australia

    Drew Mellor

Authors
  1. Drew Mellor
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Nottingham, School of Computer Science, ASAP research group, Jubilee Campus, Nottingham, NG8 1BB, and Multidisciplinary Centre for Integrative Biology, School of Biosciences, Sutton Bonington, LE12 5RD, UK

    Jaume Bacardit

  2. Enginyeria i Arquitectura La Salle, Gruß de Recerca en Sistemes Intel.ligents, Quatre Camins 2, Universitat Ramon Llull, 08022, Barcelona, Spain

    Ester Bernadó-Mansilla

  3. Department of Psychology, University of Würzburg, Röntgenring 11, 97070, Würzburg, Germany

    Martin V. Butz

  4. Department of Computer Science, University of Bristol, Merchant Venturers Building, Woodland Road, BS8 1UB, Bristol, UK

    Tim Kovacs

  5. Department of Industrial and Enterprise Systems Engineering, Illinois Genetic Algorithms Lab (IlliGAL), University of Illinois at Urbana-Champaign, 104 S. Mathews Avenue, 61801-2996, Urbana, IL, USA

    Xavier Llorà

  6. Tokyo Institute of Technology, 152-8550, tokyo, Japan

    Keiki Takadama

Rights and permissions

Reprints and permissions

Copyright information

© 2008 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Mellor, D. (2008). A Learning Classifier System Approach to Relational Reinforcement Learning. In: Bacardit, J., Bernadó-Mansilla, E., Butz, M.V., Kovacs, T., Llorà, X., Takadama, K. (eds) Learning Classifier Systems. IWLCS IWLCS 2006 2007. Lecture Notes in Computer Science(), vol 4998. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-88138-4_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-88138-4_10

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-88137-7

  • Online ISBN: 978-3-540-88138-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation