Multi-objective Learning Classifier Systems

  • Ester Bernadó-Mansilla
  • Xavier Llorà
  • Ivan Traus
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 16)

Abstract

Learning concept descriptions from data is a complex multiobjective task. The model induced by the learner should be accurate so that it can represent precisely the data instances, complete, which means it can be generalizable to new instances, and minimum, or easily readable. Learning Classifier Systems (LCSs) are a family of learners whose primary search mechanism is a genetic algorithm. Along the intense history of the field, the efforts of the community have been centered on the design of LCSs that solved these goals efficiently, resulting in the proposal of multiple systems. This paper revises the main LCS approaches and focuses on the analysis of the different mechanisms designed to fulfill the learning goals. Some of these mechanisms include implicit multiobjective learning mechanisms, while others use explicit multiobjective evolutionary algorithms. The paper analyses the advantages of using multiobjective evolutionary algorithms, especially in Pittsburgh LCSs, such as controlling the so-called bloat effect, and offering the human expert a set of concept description alternatives.

Keywords

Genetic Algorithm Pareto Front Multiobjective Optimization Target Concept Multiobjective Evolutionary Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    D. Aha and D. Kibler. Instance-based learning algorithms. Machine Learning, 6:37–66, 1991.Google Scholar
  2. [2]
    J. Bacardit and M. V. Butz. Data Mining in Learning classifier Systems: Comparing XCS with GAssist. In Seventh International Workshop on Learning Classifier Systems (IWLCS-2004). LNAI, Springer (in press), 2004.Google Scholar
  3. [3]
    J. Bacardit and J. M. Garrell. Métodos de generalización para sistemas clasificadores de Pittsburgh. In Primer Congreso Espanol de Algoritmos Evolutivos y Bioinspirados (AEB’02), pages 486–493, 2002.Google Scholar
  4. [4]
    J. Bacardit and J. M. Garrell. Bloat Control and Generalization Pressure using the Minimum Description Length Principle for a Pittsburgh Approach Learning Classifier System. In Proceedings of the 6th International Workshop on Learning Classifier Systems. LNAI, Springer (in press), 2003.Google Scholar
  5. [5]
    J. Bacardit and J. M. Garrell. Evolving Multiple Discretizations with Adaptive Intervals for a Pittsburgh Rule-Based Learning Classifier System. In Proceedings of the Genetic and Evolutionary Computation Conference, volume 2724 of LNCS, pages 1818–1831. Springer, 2003.Google Scholar
  6. [6]
    T. Bäck. Generalized convergence models for tournament-and (μ, λ)-selection. Proceedings of the Sixth International Conference on Genetic Algorithms, pages 2–8, 1995.Google Scholar
  7. [7]
    J. E. Baker. Reducing bias and inefficieny in the selection algorithm. In Genetic Algorithms and their Applications: Proceedings of the Second International Conference on Genetic Algorithms, pages 14–21, 1987.Google Scholar
  8. [8]
    W. Banzhaf and W. B. Langdon. Some Considerations on the Reason for Bloat. Genetic Programming and Evolvable Hardware, 3(1):81–91, 2002.MATHCrossRefGoogle Scholar
  9. [9]
    J. K. Bassett and K. A. De Jong. Evolving Behaviors for Cooperating Agents. In Foundations of Intelligent Systems: 12th International Symposium, volume 1932 of LNAI, pages 157–165. Springer-Verlag Berlin Heidelberg, 2000.Google Scholar
  10. [10]
    E. Bernadó-Mansilla and J. M. Garrell. MOLeCS: Using Multiobjective Evolutionary Algorithms for Learning. In Evolutionary Multi-Criterion Optimization, First International Conference, EMO 2001, volume 1993 of LNCS, pages 696–710. Springer Verlag, 2001.Google Scholar
  11. [11]
    E. Bernadó-Mansilla and J. M. Garrell. Accuracy-Based Learning Classifier Systems: Models, Analysis and Applications to Classification Tasks. Evolutionary Computation, 11(3):209–238, 2003.CrossRefGoogle Scholar
  12. [12]
    E. Bernadó-Mansilla and T. K. Ho. Domain of Competence of XCS Classifier System in Complexity Measurement Space. IEEE Transactions on Evolutionary Computation, 9(1):82–104, 2005.CrossRefGoogle Scholar
  13. [13]
    E. Bernadó-Mansilla, X. Llorà, and J. M. Garrell. XCS and GALE: a Comparative Study of Two Learning Classifier Systems on Data Mining. In Advances in Learning Classifier Systems, 4th International Workshop, volume 2321 of LNAI, pages 115–132. Springer, 2002.Google Scholar
  14. [14]
    E. Bernadó-Mansilla, A. Mekaouche, and J. M. Garrell. A Study of a Genetic Classifier System Based on the Pittsburgh Approach on a Medical Domain. In 12th International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, IEA/AIE-99, pages 175–184, 1999.Google Scholar
  15. [15]
    S. Bleuler, M. Brack, L. Thiele, and E. Zitzler. Multiobjective genetic programming: Reducing bloat using SPEA2. In Proceedings of the 2001 Congress on Evolutionary Computation CEC 2001, pages 536–543. IEEE Press, 2001.Google Scholar
  16. [16]
    P. Bonelli, A. Parodi, S. Sen, and S. W. Wilson. NEWBOOLE: A fast GBML System. In Seventh International Conference on Machine Learning, pages 153–159. Morgan Kaufmann, 1990.Google Scholar
  17. [17]
    L. Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996.MATHMathSciNetGoogle Scholar
  18. [18]
    L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification and Regression Trees. Wadsworth International Group, 1984.Google Scholar
  19. [19]
    M. V. Butz. Rule-based Evolutionary Online Learning Systems: Learning Bounds, Classification, and Prediction. PhD thesis, University of Illinois, 2004.Google Scholar
  20. [20]
    M. V. Butz, P. L. Lanzi, X. Llorà, and D. E. Goldberg. Knowledge Extraction and Problem Structure Identification in XCS. In Parallel Problem Solving from Nature (PPSN-2004), volume 3242 of LNCS, pages 1051–1060. Springer, 2004.Google Scholar
  21. [21]
    M. V. Butz and M. Pelikan. Analyzing the Evolutionary Pressures in XCS. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’2001), pages 935–942. San Francisco, CA: Morgan Kaufmann, 2001.Google Scholar
  22. [22]
    P Clark and T. Niblett. The CN2 induction algorithm. Machine Learning, 3(4):261–283, 1989.Google Scholar
  23. [23]
    K. A. De Jong and W. M. Spears. Learning Concept Classification Rules Using Genetic Algorithms. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 651–656. Sidney, Australia, 1991.Google Scholar
  24. [24]
    P. W. Dixon, D. W. Corne, and M. J. Oates. A Preliminary Investigation of Modified XCS as a Generic Data Mining Tool. In Advances in Learning Classifier Systems, 4th International Workshop, volume 2321 of LNCS, pages 133–150. Springer, 2002.Google Scholar
  25. [25]
    I. W. Flockhart. GA-MINER: Parallel Data Mining with Hierarchical Genetic Algorithms. Technical Report EPCC-AIKMS-GA-MINER-REPORT 1.0, University of Edinburgh, 1995.Google Scholar
  26. [26]
    E. Frank and I. H. Witten. Generating Accurate Rule Sets Without Global Optimization. In Machine Learning: Proceedings of the Fifteenth International Conference, pages 144–151. Morgan Kaufmann, 1998.Google Scholar
  27. [27]
    J. M Garrell, E. Golobardes, E. Bernadó-Mansilla, and X. Llorà. Automatic Diagnosis with Genetic Algorithms and Case-Based Reasoning. Artificial Intelligence in Engineering, 13:367–372, 1999.CrossRefGoogle Scholar
  28. [28]
    A. Giordana and F. Neri. Search-Intensive Concept Induction. Evolutionary Computation, 3(4):375–416, 1995.Google Scholar
  29. [29]
    D. E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley Publishing Company, Inc., 1989.Google Scholar
  30. [30]
    D. E. Goldberg and J. Richardson. Genetic algorithms with sharing for multimodal function optimization. In Proceedings of the Second International Conference on Genetic Algorithms, pages 41–49, 1987.Google Scholar
  31. [31]
    D. P. Greene and S. F. Smith. Competition-based induction of decision models from examples. Machine Learning, 13:229–257, 1993.CrossRefGoogle Scholar
  32. [32]
    J. H. Holland. Processing and processors for schemata. In Associative Information Processing, pages 127–146, 1971.Google Scholar
  33. [33]
    J. H. Holland. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control and Artificial Intelligence. MIT Press/ Bradford Books edition, 1975.Google Scholar
  34. [34]
    J. H. Holland and J. S. Reitman. Cognitive systems based on adaptive algorithms. Pattern Directed Inference Systems, pages 313–329, 1978.Google Scholar
  35. [35]
    J. H. Holmes. Discovering Risk of Disease with a Learning Classifier System. In Proceedings of the Seventh International Conference of Genetic Algorithms (ICGA97), pages 426–433. Morgan Kaufmann, 1997.Google Scholar
  36. [36]
    C. Janikow. Inductive Learning of Decision Rules in Attribute-Based Examples: a Knowledge-Intensive Genetic Algorithm Approach. PhD thesis, University of North Carolina at Chapel Hill, July 1991.Google Scholar
  37. [37]
    T. Kovacs. XCS Classifier System Reliably Evolves Accurate, Complete and Minimal Representations for Boolean Functions. In Soft Computing in Engineering Design and Manufacturing, pages 59–68. Springer-Verlag, 1997.Google Scholar
  38. [38]
    W. B. Langdon and R. Poli. Fitness causes bloat: Mutation. Genetic Programming: First European Conference, pages 37–48, 1998.Google Scholar
  39. [39]
    X. Llorà. Genetic Based Machine Learning using Fine-grained Parallelism for Data Mining. PhD thesis, Enginyeria i Arquitectura La Salle. Ramon Llull University, Barcelona, Catalonia, European Union, February, 2002.Google Scholar
  40. [40]
    X. Llorà, J. Bacardit, I. Traus, and E. Bernadó-Mansilla. Where to go once you evolved a bunch of promising hypotheses? In Learning Classifier Systems, 6th International Workshop, IWLCS 2003, LNAI. Springer (in press), 2005.Google Scholar
  41. [41]
    X. Llorà and J. M. Garrell. Automatic Classification and Artificial Life Models. In Proceedings of Learning00 Workshop. IEEE and Univesidad Carlos III, 2000.Google Scholar
  42. [42]
    X. Llorà and J. M. Garrell. Evolving Partially-Defined Instances with Evolutionary Algorithms. In Proceedings of the 18th International Conference on Machine Learning (ICML’2001), pages 337–344. Morgan Kaufmann Publishers, 2001.Google Scholar
  43. [43]
    X. Llorà and J. M. Garrell. Knowledge-Independent Data Mining with Fine-Grained Parallel Evolutionary Algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’2001), pages 461–468. Morgan Kaufmann Publishers, 2001.Google Scholar
  44. [44]
    X. Llorà and D. E. Goldberg. Bounding the effect of noise in Multiobjective Learning Classifier Systems. Evolutionary Computation, 11(3):279–298, 2003.CrossRefGoogle Scholar
  45. [45]
    X. Llorà, D. E. Goldberg, I. Traus, and E. Bernadó-Mansilla. Accuracy, Parsimony, and Generality in Evolutionary Learning Systems via Multiobjective Selection. In Learning Classifier Systems, 5th International Workshop, IWLCS 2002, volume 2661 of LNAI, pages 118–142. Springer, 2003.Google Scholar
  46. [46]
    X. Llorà, K. Sastry, and D. E. Goldberg. The Compact Classifier System: Scalability Analysis and First Results. In Proceedings of the IEEE Conference on Evolutionary Computation. IEEE press (in press), 2005.Google Scholar
  47. [47]
    C. J. Merz and P. M. Murphy. UCI Repository for Machine Learning Data-Bases [http://www.ics.uci.edu/~mlearn/MLRepository.html]. Irvine, CA: University of California, Department of Information and Computer Science, 1998.Google Scholar
  48. [48]
    T. M. Mitchell. Machine Learning. McGraw-Hill, 1997.Google Scholar
  49. [49]
    P. Nordin and W. Banzhaf. Complexity Compression and Evolution. In Proceedings of the Sixth International Conference on Genetic Algorithms, 1995.Google Scholar
  50. [50]
    C. K. Oei, D. E. Goldberg, and S. J. Chang. Tournament selection, niching, and the preservation of diversity. IlliGAL Report No. 91011, University of Illinois at Urbana-Champaign, Urbana, IL, 1991.Google Scholar
  51. [51]
    A. Orriols Puig and E. Bernadó-Mansilla. Analysis of Reduction Algorithms in XCS Classifier System. In Recent Advances in Artificial Intelligence Research and Development, volume 113 of Frontiers in Artificial Intelligence and Applications, pages 383–390. IOS Press, 2004.Google Scholar
  52. [52]
    A. Orriols Puig and E. Bernadó-Mansilla. The Class Imbalance Problem in Learning Classifier Systems: A Preliminary Study. In Learning Classifier Systems, 7th International Workshop, IWLCS 2005, LNAI. Springer (in press), 2005.Google Scholar
  53. [53]
    V. Pareto. Cours d'Economie Politique, volume I and II. F. Rouge, Lausanne, 1896.Google Scholar
  54. [54]
    R. Quinlan. Induction of decision trees. Machine Learning, 1(1):81–106, 1986.Google Scholar
  55. [55]
    R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, 1993.Google Scholar
  56. [56]
    J. Rissanen. Modeling by shortest data description. Automatica, vol. 14:465–471, 1978.MATHCrossRefGoogle Scholar
  57. [57]
    D.E. Rumelhart, J.L. McClelland, and the PDP Research Group. Parallel Distributed Processing, Vol. I, II. The MIT Press, 1986.Google Scholar
  58. [58]
    S. F. Smith. Flexible Learning of Problem Solving Heuristics through Adaptive Search. In Proceedings of the 8th International Joint Conference on Artificial Intelligence, pages 422–425, 1983.Google Scholar
  59. [59]
    T. Soule and J. Foster. Effects of code growth and parsimony pressure on populations in genetic programming. Evolutionary Computation, 6(4):293–309, Winter 1998.Google Scholar
  60. [60]
    W. A. Tackett. Recombination, selection, and the genetic construction of computer programs. Unpublished doctoral dissertation, University of Southern California, 1994.Google Scholar
  61. [61]
    D. A. Van Veldhuizen and G. B. Lamont. Multiobjective evolutionary algorithms: Analyzing the state-of-the-art. Evolutionary Computation, 8(2):125–147, 2000.CrossRefGoogle Scholar
  62. [62]
    S. W. Wilson. Classifier System Learning of a Boolean Function. Technical Report RIS 27r, The Rowland Institute for Science, 1986.Google Scholar
  63. [63]
    S. W. Wilson. Classifier Fitness Based on Accuracy. Evolutionary Computation, 3(2):149–175, 1995.Google Scholar
  64. [64]
    S. W. Wilson. Generalization in the XCS Classifier System. In Genetic Programming: Proceedings of the Third Annual Conference, pages 665–674. San Francisco, CA: Morgan Kaufmann, 1998.Google Scholar
  65. [65]
    S. W. Wilson. Mining Oblique Data with XCS. In Advances in Learning Classifier Systems: Proceedings of the Third International Workshop, volume 1996 of LNAI, pages 158–176. Springer-Verlag Berlin Heidelberg, 2001.Google Scholar
  66. [66]
    S. W. Wilson. Compact Rulesets from XCSI. In Advances in Learning Classifier Systems, 4th International Workshop, volume 2321 of LNAI, pages 197–210. Springer, 2002.Google Scholar
  67. [67]
    I. H. Witten and F. Eibe. Data Mining. Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 2003.Google Scholar
  68. [68]
    E. Zitzler. Evolutionary Algorithms for Multiobjective Optimization: Methods and Applications. PhD thesis, Swiss Federal Institute of Technology (ETH) Zurich, 1999.Google Scholar
  69. [69]
    E. Zitzler. SPEA2: Improving the Strength Pareto Evolutionary Algorithm. Technical report 103, Swiss Federal Institute of Technology (ETH) Zurich, Gloriastrasse 35, CH-8092 Zurich, May, 2001.Google Scholar
  70. [70]
    E. Zitzler, K. Deb, and L. Thiele. Comparison of Multiobjective Evolutionary Algorithms: Empirical Results. Evolutionary Computation, 8(2):173–195, 2000.CrossRefGoogle Scholar

Copyright information

© Springer 2006

Authors and Affiliations

  • Ester Bernadó-Mansilla
    • 1
  • Xavier Llorà
    • 2
  • Ivan Traus
    • 3
  1. 1.Department of Computer Engineering, Enginyeria i Arquitectura La SalleUniversitat Ramon Llull. Quatre CaminsBarcelonaSpain
  2. 2.Illinois Genetic Algorithms LabUniversity of Illinois at Urbana-ChampaignUrbanaUSA
  3. 3.Conducive Corp.New YorkUSA

Personalised recommendations