Nonlinear Least Squares Optimization of Constants in Symbolic Regression

  • Michael Kommenda
  • Michael Affenzeller
  • Gabriel Kronberger
  • Stephan M. Winkler
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8111)


In this publication a constant optimization approach for symbolic regression by genetic programming is presented. The Levenberg-Marquardt algorithm, a nonlinear, least-squares method, tunes numerical values of constants in symbolic expression trees to improve their fit to observed data. The necessary gradient information for the algorithm is obtained by automatic programming, which efficiently calculates the partial derivatives of symbolic expression trees.

The performance of the methodology is tested for standard and offspring selection genetic programming on four well-known benchmark datasets. Although constant optimization includes an overhead regarding the algorithm runtime, the achievable quality increases significantly compared to the standard algorithms. For example, the average coefficient of determination on the Poly-10 problem changes from 0.537 without constant optimization to over 0.8 with constant optimization enabled. In addition to the experimental results, the effect of different parameter settings like the number of individuals to be optimized is detailed.


Constant Optimization Symbolic Regression Genetic Programming Levenberg-Marquard Algorithm Automatic Differentiation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Affenzeller, M., Winkler, S., Wagner, S., Beham, A.: Genetic Algorithms and Genetic Programming - Modern Concepts and Practical Applications, Numerical Insights, vol. 6. CRC Press, Chapman & Hall (2009)Google Scholar
  2. 2.
    Alonso, C.L., Montaña, J.L., Borges, C.E.: Evolution strategies for constants optimization in genetic programming. In: ICTAI, pp. 703–707. IEEE Computer Society (2009),
  3. 3.
    Bochkanov, S., Bystritsky, V.: Alglib,
  4. 4.
    Friedman, J.H.: Multivariate adaptive regression splines. The Annals of Statistics, 1–67 (1991)Google Scholar
  5. 5.
    Keijzer, M.: Improving symbolic regression with interval arithmetic and linear scaling. In: Ryan, C., Soule, T., Keijzer, M., Tsang, E.P.K., Poli, R., Costa, E. (eds.) EuroGP 2003. LNCS, vol. 2610, pp. 70–82. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  6. 6.
    Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992)zbMATHGoogle Scholar
  7. 7.
    Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Quarterly Journal of Applied Mathmatics II(2), 164–168 (1944)MathSciNetGoogle Scholar
  8. 8.
    Mukherjee, S., Eppstein, M.J.: Differential evolution of constants in genetic programming improves efficacy and bloat. In: Proceedings of the Fourteenth International Conference on Genetic and Evolutionary Computation Conference Companion, GECCO Companion 2012, pp. 625–626. ACM, New York (2012), CrossRefGoogle Scholar
  9. 9.
    Poli, R.: A simple but theoretically-motivated method to control bloat in genetic programming. In: Ryan, C., Soule, T., Keijzer, M., Tsang, E.P.K., Poli, R., Costa, E. (eds.) EuroGP 2003. LNCS, vol. 2610, pp. 204–217. Springer, Heidelberg (2003), CrossRefGoogle Scholar
  10. 10.
    Rall, L.B.: Automatic Differentiation: Techniques and Applications. LNCS, vol. 120. Springer, Heidelberg (1981)CrossRefzbMATHGoogle Scholar
  11. 11.
    Shtof, A.: Autodiff,
  12. 12.
    Topchy, A., Punch, W.F.: Faster genetic programming based on local gradient search of numeric leaf values. In: Spector, L., Goodman, E.D., Wu, A., Langdon, W.B., Voigt, H.M., Gen, M., Sen, S., Dorigo, M., Pezeshk, S., Garzon, M.H., Burke, E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), July 7-11, pp. 155–162. Morgan Kaufmann, San Francisco (2001), Google Scholar
  13. 13.
    Wagner, S.: Heuristic Optimization Software Systems - Modeling of Heuristic Optimization Algorithms in the HeuristicLab Software Environment. Ph.D. thesis, Institute for Formal Models and Verification, Johannes Kepler University, Linz, Austria (2009)Google Scholar
  14. 14.
    White, D.R., McDermott, J., Castelli, M., Manzoni, L., Goldman, B.W., Kronberger, G., Jaskowski, W., O’Reilly, U.M., Luke, S.: Better GP benchmarks: community survey results and proposals. Genetic Programming and Evolvable Machines 14(1), 3–29 (2013)CrossRefGoogle Scholar
  15. 15.
    Zhang, Q., Zhou, C., Xiao, W., Nelson, P.C.: Improving gene expression programming performance by using differential evolution. In: Proceedings of the Sixth International Conference on Machine Learning and Applications, ICMLA 2007, pp. 31–37. IEEE Computer Society, Washington, DC (2007), Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Michael Kommenda
    • 1
  • Michael Affenzeller
    • 1
  • Gabriel Kronberger
    • 1
  • Stephan M. Winkler
    • 1
  1. 1.Heuristic and Evolutionary Algorithms Laboratory, School of Informatics, Communications and MediaUniversity of Applied Sciences Upper AustriaHagenbergAustria

Personalised recommendations