A Novel Neural Network-Based Symbolic Regression Method: Neuro-Encoded Expression Programming

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11728)


Neuro-encoded expression programming (NEEP) that aims to offer a novel continuous representation of combinatorial encoding for genetic programming methods is proposed in this paper. Genetic programming with linear representation uses nature-inspired operators (e.g., crossover, mutation) to tune expressions and finally search out the best explicit function to simulate data. The encoding mechanism is essential for genetic programmings to find a desirable solution efficiently. However, the linear representation methods manipulate the expression tree in discrete solution space, where a small change of the input can cause a large change of the output. The unsmooth landscapes destroy the local information and make difficulty in searching. The neuro-encoded expression programming constructs the gene string with recurrent neural network (RNN) and the weights of the network are optimized by powerful continuous evolutionary algorithms. The neural network mappings smoothen the sharp fitness landscape and provide rich neighborhood information to find the best expression. The experiments indicate that the novel approach improves training efficiency and reduces test errors on several well-known symbolic regression problems.


Genetic programming Symbolic regression Neural network Gene expression programming Evolutionary algorithm 



This work was supported by National Natural Science Foundation of China under Grant No. 61573166, No. 61572230, No. 61872419, No. 61873324, No. 81671785, No. 61672262. Project of Shandong Province Higher Educational Science and Technology Program under Grant No. J16LN07. Shandong Provincial Natural Science Foundation No. ZR2019MF040, No. ZR2018LF005. Shandong Provincial Key R&D Program under Grant No. 2019GGX101041, No. 2018GGX101048, No. 2016ZDJS01A12, No. 2016GGX101001, No. 2017CXZC1206. Taishan Scholar Project of Shandong Province, China.


  1. 1.
    Auger, A., Hansen, N.: A restart CMA evolution strategy with increasing population size. In: 2005 IEEE Congress on Evolutionary Computation, vol. 2, pp. 1769–1776 (2005).
  2. 2.
    Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., Bengio, S.: Generating sentences from a continuous space. In: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10–21 (2016).
  3. 3.
    Brameier, M.F., Banzhaf, W.: Basic concepts of linear genetic programming. In: Linear Genetic Programming, pp. 13–34. Springer, Boston (2007).
  4. 4.
    Chatzarakis, G.E., Li, T.: Oscillation criteria for delay and advanced differential equations with nonmonotone arguments. Complexity 2018, 1–18 (2018). Scholar
  5. 5.
    Chen, C.L.P., Zhang, T., Chen, L., Tam, S.C.: I-Ching divination evolutionary algorithm and its convergence analysis. IEEE Trans. Cybern. 47(1), 2–13 (2017). Scholar
  6. 6.
    Dheeru, D., Karra Taniskidou, E.: UCI machine learning repository (2017)Google Scholar
  7. 7.
    Dick, G.: Sensitivity-like analysis for feature selection in genetic programming. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 401–408. ACM (2017).
  8. 8.
    Fan, P., Xin, W., Ouyang, Y.: Approximation of discrete spatial data for continuous facility location design. Integr. Comput. Aided Eng. 21(4), 311–320 (2014). Scholar
  9. 9.
    Ferreira, C.: Automatically defined functions in gene expression programming. In: Nedjah, N., Mourelle, L.M., Abraham, A. (eds.) Genetic Systems Programming. SCI, vol. 13, pp. 21–56. Springer, Heidelberg (2006). Scholar
  10. 10.
    Ferreira, C.: Gene expression programming: a new adaptive algorithm for solving problems. Complex Syst. 13(2), 87–129 (2001). Scholar
  11. 11.
    Holland, J.H.: Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. MIT Press, Cambridge (1992)CrossRefGoogle Scholar
  12. 12.
    Huang, W., Oh, S.K., Pedrycz, W.: Hybrid fuzzy wavelet neural networks architecture based on polynomial neural networks and fuzzy set/relation inference-based wavelet neurons. IEEE Trans. Neural Netw. Learn. Syst. 29(8), 3452–3462 (2018). Scholar
  13. 13.
    Kataoka, Y., Matsubara, T., Uehara, K.: Image generation using generative adversarial networks and attention mechanism. In: IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS), pp. 1–6 (2016).
  14. 14.
    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the International Conference on Neural Networks, ICNN 1995, vol. 4, pp. 1942–1948 (1995).
  15. 15.
    Koza, J.R., Andre, D., Bennett III, F.H., Keane, M.A.: Use of automatically defined functions and architecture-altering operations in automated circuit synthesis with genetic programming. In: Proceedings of the First Annual Conference on Genetic Programming, pp. 132–140 (1996)Google Scholar
  16. 16.
    Langdon, W.B., Poli, R., McPhee, N.F., Koza, J.R.: Genetic programming: an introduction and tutorial, with a survey of techniques and applications. In: Fulcher, J., Jain, L.C. (eds.) Computational Intelligence: A Compendium. SCI, vol. 115, pp. 927–1028. Springer, Heidelberg (2008). Scholar
  17. 17.
    Levy, O., Goldberg, Y.: Neural word embedding as implicit matrix factorization. In: Advances in Neural Information Processing Systems, pp. 2177–2185 (2014)Google Scholar
  18. 18.
    Liskowski, P., Bładek, I., Krawiec, K.: Neuro-guided genetic programming: prioritizing evolutionary search with neural networks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1143–1150 (2018).
  19. 19.
    Maehara, T., Marumo, N., Murota, K.: Continuous relaxation for discrete DC programming. Math. Programm. 169(1), 199–219 (2018). Scholar
  20. 20.
    Manjunath, G., Jaeger, H.: Echo state property linked to an input: exploring a fundamental characteristic of recurrent neural networks. Neural Comput. 25(3), 671–696 (2013). Scholar
  21. 21.
    McConaghy, T.: FFX: fast, scalable, deterministic symbolic regression technology. In: Riolo, R., Vladislavleva, E., Moore, J. (eds.) Genetic Programming Theory and Practice IX. Genetic and Evolutionary Computation, pp. 235–260. Springer, New York (2011). Scholar
  22. 22.
    McDermott, J., et al.: Genetic programming needs better benchmarks. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 791–798 (2012).
  23. 23.
    Mogren, O.: C-RNN-GAN: continuous recurrent neural networks with adversarial training (2016). arXiv preprint: arXiv:1611.09904
  24. 24.
    Nicolau, M., Agapitos, A., O’Neill, M., Brabazon, A.: Guidelines for defining benchmark problems in genetic programming. In: 2015 IEEE Congress on Evolutionary Computation (CEC), pp. 1152–1159 (2015).
  25. 25.
    Oltean, M., Grosan, C.: Evolving evolutionary algorithms using multi expression programming. In: The 7th European Conference on Artificial Life, vol. 2801, pp. 651–658 (2003). Scholar
  26. 26.
    O’Neill, M., Ryan, C.: Under the hood of grammatical evolution. In: Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation, vol. 2, pp. 1143–1148 (1999)Google Scholar
  27. 27.
    Orzechowski, P., La Cava, W., Moore, J.H.: Where are we now?: A large benchmark study of recent symbolic regression methods. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1183–1190 (2018).
  28. 28.
    Pardalos, P.M., Prokopyev, O.A., Busygin, S.: Continuous approaches for solving discrete optimization problems. In: Appa, G., Pitsoulis, L., Williams, H.P. (eds.) Handbook on Modelling for Discrete Optimization. International Series in Operations Research & Management Science, vol. 88, pp. 39–60. Springer, Boston (2006). Scholar
  29. 29.
    Poli, R.: A simple but theoretically-motivated method to control bloat in genetic programming. In: Ryan, C., Soule, T., Keijzer, M., Tsang, E., Poli, R., Costa, E. (eds.) EuroGP 2003. LNCS, vol. 2610, pp. 204–217. Springer, Heidelberg (2003). Scholar
  30. 30.
    Rothlauf, F.: Analysis and design of representations for trees. In: Representations for Genetic and Evolutionary Algorithms, pp. 141–215. Springer, Heidelberg (2006).
  31. 31.
    Sussillo, D., Abbott, L.F.: Generating coherent patterns of activity from chaotic neural networks. Neuron 63(4), 544–557 (2009). Scholar
  32. 32.
    Wang, H., Qin, Z., Wan, T.: Text generation based on generative adversarial nets with latent variables. In: Phung, D., Tseng, V.S., Webb, G.I., Ho, B., Ganji, M., Rashidi, L. (eds.) 2018 Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 92–103 (2018). Scholar
  33. 33.
    Wang, L., Orchard, J.: Investigating the evolution of a neuroplasticity network for learning. IEEE Trans. Syst. Man Cybern. Syst. (2017).
  34. 34.
    Wang, L., Yang, B., Chen, Y., Zhang, X., Orchard, J.: Improving neural-network classifiers using nearest neighbor partitioning. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2255–2267 (2017). Scholar
  35. 35.
    Wang, L., Yang, B., Orchard, J.: Particle swarm optimization using dynamic tournament topology. Appl. Soft Comput. 48, 584–596 (2016). Scholar
  36. 36.
    Wang, L., Yang, B., Wang, S., Liang, Z.: Building image feature kinetics for cement hydration using gene expression programming with similarity weight tournament selection. IEEE Trans. Evol. Comput. 19(5), 679–693 (2015). Scholar
  37. 37.
    Xin, L., Chi, Z., Weimin, X., Peter, C.N.: Prefix gene expression programming. In: Late Breaking Paper at Genetic and Evolutionary Computation Conference, pp. 25–29 (2005)Google Scholar
  38. 38.
    Yin, J., Meng, Y., Jin, Y.: A developmental approach to structural self-organization in reservoir computing. IEEE Trans. Auton. Mental Dev. 4(4), 273–289 (2012). Scholar
  39. 39.
    Zelinka, I., Oplatkova, Z., Nolle, L.: Analytic programming-symbolic regression by means of arbitrary evolutionary algorithms. Int. J. Simul. Syst. Sci. Technol. 6(9), 44–56 (2005)Google Scholar
  40. 40.
    Zhong, J., Feng, L., Ong, Y.S.: Gene expression programming: a survey. IEEE Comput. Intell. Mag. 12(3), 54–72 (2017). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Shandong Provincial Key Laboratory of Network Based Intelligent ComputingUniversity of JinanJinanChina
  2. 2.David R. Cheriton School of Computer ScienceUniversity of WaterlooWaterlooCanada

Personalised recommendations