Recursive Least Squares and Quadratic Prediction in Continuous Multistep Problems

  • Daniele Loiacono
  • Pier Luca Lanzi
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6471)

Abstract

XCS with computed prediction, namely XCSF, has been recently extended in several ways. In particular, a novel prediction update algorithm based on recursive least squares and the extension to polynomial prediction led to significant improvements of XCSF. However, these extensions have been studied so far only on single step problems and it is currently not clear if these findings might be extended also to multistep problems. In this paper we investigate this issue by analyzing the performance of XCSF with recursive least squares and with quadratic prediction on continuous multistep problems. Our results show that both these extensions improve the convergence speed of XCSF toward an optimal performance. As showed by the analysis reported in this paper, these improvements are due to the capabilities of recursive least squares and of polynomial prediction to provide a more accurate approximation of the problem value function after the first few learning problems.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Boyan, J.A., Moore, A.W.: Generalization in reinforcement learning: Safely approximating the value function. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems 7, pp. 369–376. The MIT Press, Cambridge (1995)Google Scholar
  2. 2.
    Butz, M.V., Pelikan, M.: Analyzing the evolutionary pressures in xcs. In: Spector, L., Goodman, E.D., Wu, A., Langdon, W.B., Voigt, H.-M., Gen, M., Sen, S., Dorigo, M., Pezeshk, S., Garzon, M.H., Burke, E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2001), July 7-11, pp. 935–942. Morgan Kaufmann, San Francisco (2001)Google Scholar
  3. 3.
    Butz, M.V., Wilson, S.W.: An algorithmic description of xcs. Journal of Soft Computing 6(3-4), 144–153 (2002)CrossRefMATHGoogle Scholar
  4. 4.
    Goodwin, G.C., Sin, K.S.: Adaptive Filtering: Prediction and Control, Prentice-Hall information and system sciences series (March 1984)Google Scholar
  5. 5.
    Haykin, S.: Adaptive Filter Theory, 4th edn. Prentice-Hall, Englewood Cliffs (2001)MATHGoogle Scholar
  6. 6.
    Lanzi, P.L., Loiacono, D.: Xcsf with neural prediction. In: IEEE Congress on Evolutionary Computation, CEC 2006, pp. 2270–2276 (2006)Google Scholar
  7. 7.
    Lanzi, P.L., Loiacono, D., Wilson, S.W., Goldberg, D.E.: Extending XCSF beyond linear approximation. In: Genetic and Evolutionary Computation – GECCO-2005, Washington DC, USA, pp. 1859–1866. ACM Press, New York (2005)Google Scholar
  8. 8.
    Lanzi, P.L., Loiacono, D., Wilson, S.W., Goldberg, D.E.: XCS with computed prediction for the learning of boolean functions. In: Proceedings of the IEEE Congress on Evolutionary Computation – CEC 2005, Edinburgh, UK, pp. 588–595. IEEE, Los Alamitos (September 2005)CrossRefGoogle Scholar
  9. 9.
    Lanzi, P.L., Loiacono, D., Wilson, S.W., Goldberg, D.E.: XCS with computed prediction in continuous multistep environments. In: Proceedings of the IEEE Congress on Evolutionary Computation – CEC 2005, Edinburgh, UK, pp. 2032–2039. IEEE, Los Alamitos (September 2005)CrossRefGoogle Scholar
  10. 10.
    Lanzi, P.L., Loiacono, D., Wilson, S.W., Goldberg, D.E.: Prediction update algorithms for XCSF: RLS, kalman filter, and gain adaptation. In: GECCO 2006: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 1505–1512. ACM Press, New York (2006)Google Scholar
  11. 11.
    Lanzi, P.L., Loiacono, D., Wilson, S.W., Goldberg, D.E.: Generalization in the XCSF classifier system: Analysis, improvement, and extension. Evolutionary Computation 15(2), 133–168 (2007)CrossRefGoogle Scholar
  12. 12.
    Loiacono, D., Marelli, A., Lanzi, P.L.: Support vector regression for classifier prediction. In: GECCO 2007: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, pp. 1806–1813. ACM Press, New York (2007)Google Scholar
  13. 13.
    Watkins, C.J.C.H.: Learning from delayed reward. PhD thesis (1989)Google Scholar
  14. 14.
    Watkins, C.J.C.H., Dayan, P.: Technical note: Q-Learning. Machine Learning 8, 279–292 (1992)MATHGoogle Scholar
  15. 15.
    Widrow, B., Hoff, M.E.: Neurocomputing: Foundation of Research. In: Adaptive Switching Circuits, pp. 126–134. The MIT Press, Cambridge (1988)Google Scholar
  16. 16.
    Wilson, S.W.: Classifier Fitness Based on Accuracy. Evolutionary Computation 3(2), 149–175 (1995), http://prediction-dynamics.com/ CrossRefGoogle Scholar
  17. 17.
    Wilson, S.W.: Mining Oblique Data with XCS. In: Lanzi, P.L., Stolzmann, W., Wilson, S.W. (workshop organisers): Proceedings of the International Workshop on Learning Classifier Systems (IWLCS-2000), in the Joint Workshops of SAB 2000 and PPSN 2000, pp. 158–174 (2000)Google Scholar
  18. 18.
    Wilson, S.W.: Classifiers that approximate functions. Journal of Natural Computing 1(2-3), 211–234 (2002)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Wilson, S.W.: Classifier systems for continuous payoff environments. In: Deb, K., Poli, R., Banzhaf, W., Beyer, H.-G., Burke, E., Darwen, P., Dasgupta, D., Floreano, D., Foster, J., Harman, M., Holland, O., Lanzi, P.L., Spector, L., Tettamanzi, A., Thierens, D., Tyrrell, A. (eds.) GECCO 2004. LNCS, vol. 3103, pp. 824–835. Springer, Heidelberg (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Daniele Loiacono
    • 1
  • Pier Luca Lanzi
    • 1
  1. 1.Dipartimento di Elettronica e InformazionePolitecnico di MilanoMilanoItaly

Personalised recommendations