Skip to main content

Nonlinear Least Squares Optimization of Constants in Symbolic Regression

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 8111)

Abstract

In this publication a constant optimization approach for symbolic regression by genetic programming is presented. The Levenberg-Marquardt algorithm, a nonlinear, least-squares method, tunes numerical values of constants in symbolic expression trees to improve their fit to observed data. The necessary gradient information for the algorithm is obtained by automatic programming, which efficiently calculates the partial derivatives of symbolic expression trees.

The performance of the methodology is tested for standard and offspring selection genetic programming on four well-known benchmark datasets. Although constant optimization includes an overhead regarding the algorithm runtime, the achievable quality increases significantly compared to the standard algorithms. For example, the average coefficient of determination on the Poly-10 problem changes from 0.537 without constant optimization to over 0.8 with constant optimization enabled. In addition to the experimental results, the effect of different parameter settings like the number of individuals to be optimized is detailed.

Keywords

  • Constant Optimization
  • Symbolic Regression
  • Genetic Programming
  • Levenberg-Marquard Algorithm
  • Automatic Differentiation

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-53856-8_53
  • Chapter length: 8 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   64.99
Price excludes VAT (USA)
  • ISBN: 978-3-642-53856-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   84.99
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Affenzeller, M., Winkler, S., Wagner, S., Beham, A.: Genetic Algorithms and Genetic Programming - Modern Concepts and Practical Applications, Numerical Insights, vol. 6. CRC Press, Chapman & Hall (2009)

    Google Scholar 

  2. Alonso, C.L., Montaña, J.L., Borges, C.E.: Evolution strategies for constants optimization in genetic programming. In: ICTAI, pp. 703–707. IEEE Computer Society (2009), http://dblp.uni-trier.de/db/conf/ictai/ictai2009.html#AlonsoMB09

  3. Bochkanov, S., Bystritsky, V.: Alglib, http://www.alglib.net/

  4. Friedman, J.H.: Multivariate adaptive regression splines. The Annals of Statistics, 1–67 (1991)

    Google Scholar 

  5. Keijzer, M.: Improving symbolic regression with interval arithmetic and linear scaling. In: Ryan, C., Soule, T., Keijzer, M., Tsang, E.P.K., Poli, R., Costa, E. (eds.) EuroGP 2003. LNCS, vol. 2610, pp. 70–82. Springer, Heidelberg (2003)

    CrossRef  Google Scholar 

  6. Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992)

    MATH  Google Scholar 

  7. Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Quarterly Journal of Applied Mathmatics II(2), 164–168 (1944)

    MathSciNet  Google Scholar 

  8. Mukherjee, S., Eppstein, M.J.: Differential evolution of constants in genetic programming improves efficacy and bloat. In: Proceedings of the Fourteenth International Conference on Genetic and Evolutionary Computation Conference Companion, GECCO Companion 2012, pp. 625–626. ACM, New York (2012), http://doi.acm.org/10.1145/2330784.2330891

    CrossRef  Google Scholar 

  9. Poli, R.: A simple but theoretically-motivated method to control bloat in genetic programming. In: Ryan, C., Soule, T., Keijzer, M., Tsang, E.P.K., Poli, R., Costa, E. (eds.) EuroGP 2003. LNCS, vol. 2610, pp. 204–217. Springer, Heidelberg (2003), http://dl.acm.org/citation.cfm?id=1762668.1762688

    CrossRef  Google Scholar 

  10. Rall, L.B.: Automatic Differentiation: Techniques and Applications. LNCS, vol. 120. Springer, Heidelberg (1981)

    CrossRef  MATH  Google Scholar 

  11. Shtof, A.: Autodiff, http://autodiff.codeplex.com/

  12. Topchy, A., Punch, W.F.: Faster genetic programming based on local gradient search of numeric leaf values. In: Spector, L., Goodman, E.D., Wu, A., Langdon, W.B., Voigt, H.M., Gen, M., Sen, S., Dorigo, M., Pezeshk, S., Garzon, M.H., Burke, E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), July 7-11, pp. 155–162. Morgan Kaufmann, San Francisco (2001), http://www.cs.bham.ac.uk/~wbl/biblio/gecco2001/d01.pdf

    Google Scholar 

  13. Wagner, S.: Heuristic Optimization Software Systems - Modeling of Heuristic Optimization Algorithms in the HeuristicLab Software Environment. Ph.D. thesis, Institute for Formal Models and Verification, Johannes Kepler University, Linz, Austria (2009)

    Google Scholar 

  14. White, D.R., McDermott, J., Castelli, M., Manzoni, L., Goldman, B.W., Kronberger, G., Jaskowski, W., O’Reilly, U.M., Luke, S.: Better GP benchmarks: community survey results and proposals. Genetic Programming and Evolvable Machines 14(1), 3–29 (2013)

    CrossRef  Google Scholar 

  15. Zhang, Q., Zhou, C., Xiao, W., Nelson, P.C.: Improving gene expression programming performance by using differential evolution. In: Proceedings of the Sixth International Conference on Machine Learning and Applications, ICMLA 2007, pp. 31–37. IEEE Computer Society, Washington, DC (2007), http://dx.doi.org/10.1109/ICMLA.2007.55

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Kommenda, M., Affenzeller, M., Kronberger, G., Winkler, S.M. (2013). Nonlinear Least Squares Optimization of Constants in Symbolic Regression. In: Moreno-Díaz, R., Pichler, F., Quesada-Arencibia, A. (eds) Computer Aided Systems Theory - EUROCAST 2013. EUROCAST 2013. Lecture Notes in Computer Science, vol 8111. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-53856-8_53

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-53856-8_53

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-53855-1

  • Online ISBN: 978-3-642-53856-8

  • eBook Packages: Computer ScienceComputer Science (R0)