Learning with Configurable Operators and RL-Based Heuristics

  • Fernando Martínez-Plumed
  • Cèsar Ferri
  • José Hernández-Orallo
  • María José Ramírez-Quintana
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7765)


In this paper, we push forward the idea of machine learning systems for which the operators can be modified and finetuned for each problem. This allows us to propose a learning paradigm where users can write (or adapt) their operators, according to the problem, data representation and the way the information should be navigated. To achieve this goal, data instances, background knowledge, rules, programs and operators are all written in the same functional language, Erlang. Since changing operators affect how the search space needs to be explored, heuristics are learnt as a result of a decision process based on reinforcement learning where each action is defined as a choice of operator and rule. As a result, the architecture can be seen as a ‘system for writing machine learning systems’ or to explore new operators.


machine learning operators complex data heuristics inductive programming reinforcement learning Erlang 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Armstrong, J.: A history of erlang. In: Proceedings of the Third ACM SIGPLAN Conf. on History of Programming Languages, HOPL III, pp. 1–26. ACM (2007)Google Scholar
  2. 2.
    Brazdil, P., Giraud-Carrier: Metalearning: Concepts and systems. In: Metalearning. Cognitive Technologies, pp. 1–10. Springer, Heidelberg (2009)Google Scholar
  3. 3.
    Daumé III, H., Langford, J.: Search-based structured prediction (2009)Google Scholar
  4. 4.
    Dietterich, T., Domingos, P., Getoor, L., Muggleton, S., Tadepalli, P.: Structured machine learning: the next ten years. Machine Learning 73, 3–23 (2008)CrossRefGoogle Scholar
  5. 5.
    Dietterich, T.G., Lathrop, R., Lozano-Perez, T.: Solving the multiple-instance problem with axis-parallel rectangles. Artificial Intelligence 89, 31–71 (1997)CrossRefzbMATHGoogle Scholar
  6. 6.
    Džeroski, S.: Towards a general framework for data mining. In: Džeroski, S., Struyf, J. (eds.) KDID 2006. LNCS, vol. 4747, pp. 259–300. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  7. 7.
    Dzeroski, S., De Raedt, L., Driessens, K.: Relational reinforcement learning. Machine Learning 43, 7–52 (2001), 10.1023/A:1007694015589CrossRefzbMATHGoogle Scholar
  8. 8.
    Dzeroski, S., Lavrac, N. (eds.): Relational Data Mining. Springer (2001)Google Scholar
  9. 9.
    Estruch, V., Ferri, C., Hernández-Orallo, J., Ramírez-Quintana, M.J.: Similarity functions for structured data. an application to decision trees. Inteligencia Artificial, Revista Iberoamericana de Inteligencia Artificial 10(29), 109–121 (2006)Google Scholar
  10. 10.
    Estruch, V., Ferri, C., Hernández-Orallo, J., Ramírez-Quintana, M.J.: Web categorisation using distance-based decision trees. ENTCS 157(2), 35–40 (2006)Google Scholar
  11. 11.
    Estruch, V., Ferri, C., Hernández-Orallo, J., Ramírez-Quintana, M.J.: Bridging the Gap between Distance and Generalisation. Computational Intelligence (2012)Google Scholar
  12. 12.
    Ferri-Ramírez, C., Hernández-Orallo, J., Ramírez-Quintana, M.J.: Incremental learning of functional logic programs. In: Kuchen, H., Ueda, K. (eds.) FLOPS 2001. LNCS, vol. 2024, pp. 233–247. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  13. 13.
    Gärtner, T.: Kernels for Structured Data. PhD thesis, Universitat Bonn (2005)Google Scholar
  14. 14.
    Holland, J.H., Booker, L.B., Colombetti, M., Dorigo, M., Goldberg, D.E., Forrest, S., Riolo, R.L., Smith, R.E., Lanzi, P.L., Stolzmann, W., Wilson, S.W.: What is a learning classifier system? In: Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds.) IWLCS 1999. LNCS (LNAI), vol. 1813, pp. 3–32. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  15. 15.
    Holmes, J.H., Lanzi, P., Stolzmann, W.: Learning classifier systems: New models, successful applications. Information Processing Letters (2002)Google Scholar
  16. 16.
    Kitzelmann, E.: Inductive programming: A survey of program synthesis techniques. In: Schmid, U., Kitzelmann, E., Plasmeijer, R. (eds.) AAIP 2009. LNCS, vol. 5812, pp. 50–73. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  17. 17.
    Koller, D., Sahami, M.: Hierarchically classifying documents using very few words. In: Proceedings of the Fourteenth International Conference on Machine Learning, ICML 1997, pp. 170–178. Morgan Kaufmann Publishers Inc., San Francisco (1997)Google Scholar
  18. 18.
    Lafferty, J., McCallum, A.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: ICML 2001, pp. 282–289 (2001)Google Scholar
  19. 19.
    Lloyd, J.W.: Knowledge representation, computation, and learning in higher-order logic (2001)Google Scholar
  20. 20.
    Maes, F., Denoyer, L., Gallinari, P.: Structured prediction with reinforcement learning. Machine Learning Journal 77(2-3), 271–301 (2009)CrossRefGoogle Scholar
  21. 21.
    Martínez-Plumed, F., Estruch, V., Ferri, C., Hernández-Orallo, J., Ramírez-Quintana, M.J.: Newton trees. In: Li, J. (ed.) AI 2010. LNCS, vol. 6464, pp. 174–183. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  22. 22.
    Muggleton, S.: Inverse entailment and Progol. New Generation Computing (1995)Google Scholar
  23. 23.
    Muggleton, S.H.: Inductive logic programming: Issues, results, and the challenge of learning language in logic. Artificial Intelligence 114(1-2), 283–296 (1999)CrossRefzbMATHGoogle Scholar
  24. 24.
    Plotkin, G.: A note on inductive generalization. Machine Intelligence 5 (1970)Google Scholar
  25. 25.
    Schmidhuber, J.: Optimal ordered problem solver. Maching Learning 54(3), 211–254 (2004)CrossRefzbMATHGoogle Scholar
  26. 26.
    Srinivasan, A.: The Aleph Manual (2004)Google Scholar
  27. 27.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (1998)Google Scholar
  28. 28.
    Tadepalli, P., Givan, R., Driessens, K.: Relational reinforcement learning: An overview. In: Proc. of the Workshop on Relational Reinforcement Learning (2004)Google Scholar
  29. 29.
    Tamaddoni-Nezhad, A., Muggleton, S.: A genetic algorithms approach to ILP. In: Matwin, S., Sammut, C. (eds.) ILP 2002. LNCS (LNAI), vol. 2583, pp. 285–300. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  30. 30.
    Tsochantaridis, I., Hofmann, T., Joachims, T., Altun, Y.: Support vector machine learning for interdependent and structured output spaces. In: ICML (2004)Google Scholar
  31. 31.
    Wallace, C.S., Dowe, D.L.: Refinements of MDL and MML coding. Comput. J. 42(4), 330–337 (1999)CrossRefzbMATHGoogle Scholar
  32. 32.
    Watkins, C., Dayan, P.: Q-learning. Machine Learning 8, 279–292 (1992)zbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Fernando Martínez-Plumed
    • 1
  • Cèsar Ferri
    • 1
  • José Hernández-Orallo
    • 1
  • María José Ramírez-Quintana
    • 1
  1. 1.DSICUniversitat Politècnica de ValènciaValènciaSpain

Personalised recommendations