Skip to main content
Book cover

Black Box Optimization, Machine Learning, and No-Free Lunch Theorems pp 221–271Cite as

Variable Neighborhood Programming as a Tool of Machine Learning

  • 1162 Accesses

Part of the Springer Optimization and Its Applications book series (SOIA,volume 170)

Abstract

Automatic programming is an efficient technique that has contributed to an important development in the artificial intelligence and machine learning fields. In this chapter, we introduce the technique called Variable Neighborhood Programming (VNP) that was inspired by the principle of the Variable Neighborhood Search (VNS) algorithm. VNP starts from a single solution presented by a program, and the search for a good quality global solution (program) continues by exploring different neighborhoods. The goal of our algorithm is to generate a good representative program adequate to a selected problem. VNP takes the advantages of the systematic change of neighborhood structures randomly or within a local search algorithm to diversify or intensify search through the solution space. To show its efficiency and usefulness, the VNP method is applied first for solving the symbolic regression problem (VNP-SRP) and tested and compared on usual test instances from the literature. In addition, the VNP-SRP method is tested in finding formulas for life expectancy as a function of some health care economic factors in 18 Russian districts. Finally, the VNP is implemented on prediction and classification problems and tested on real-life maintenance railway problems from the US railway system.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-66515-9_9
  • Chapter length: 51 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-66515-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   139.99
Price excludes VAT (USA)
Hardcover Book
USD   139.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

References

  1. Andersson, M.: Strategic planning of track maintenance – state of the art. Technical Report 02-035 (2002)

    Google Scholar 

  2. Andrade, A.R., Teixeira, P.F.: Hierarchical Bayesian modelling of rail track geometry degradation. Proc. Inst. Mech. Eng. F J. Rail Rapid Transit 227(4), 364–375 (2013)

    CrossRef  Google Scholar 

  3. Andrade, A., Teixeira, P.: Statistical modelling of railway track geometry degradation using Hierarchical Bayesian models. Reliab. Eng. Syst. Saf. 142, 169–183 (2015)

    CrossRef  Google Scholar 

  4. Arnaldo, I., Krawiec, K., O’Reilly, U.-M.: Multiple regression genetic programming. In: Proceedings of the 2014 Conference on Genetic and Evolutionary Computation – GECCO ’14, pp. 879–886. ACM Press, New York (2014)

    Google Scholar 

  5. Association of American Railroads (AAR). https://www.aar.org/todays-railroads (2015)

  6. Bouaziz, S., Dhahri, H., Alimi, A.M., Abraham, A.: A hybrid learning algorithm for evolving Flexible Beta Basis Function Neural Tree Model. Neurocomputing 117, 107–117 (2013)

    CrossRef  Google Scholar 

  7. Brimberg, J., Mladenović, N., Todosijević, R., Urošević, D.: Less is more: solving the Max-Mean diversity problem with variable neighborhood search. Inf. Sci. 382–383, 179–200 (2017)

    CrossRef  Google Scholar 

  8. Brown, B.M., Chen, S.X.: Beta-Bernstein smoothing for regression curves with compact support. Scand. J. Stat. 26(1), 47–59 (1999)

    MathSciNet  MATH  CrossRef  Google Scholar 

  9. Cai, W., Pacheco-Vega, A., Sen, M., Yang, K.: Heat transfer correlations by symbolic regression. Int. J. Heat Mass Trans. 49(23–24), 4352–4359 (2006)

    MATH  CrossRef  Google Scholar 

  10. Cannon, D.F., Edel, K.-O., Grassie, S.L., Sawley, K.: Rail defects: an overview. Fatigue Fract. Eng. Mater. Struct. 26(10), 865–886 (2003)

    CrossRef  Google Scholar 

  11. Castelli, M., Vanneschi, L., Silva, S.: Prediction of the unified Parkinson’s disease rating scale assessment using a genetic programming system with geometric semantic genetic operators. Expert Syst. Appl. 41(10), 4608–4616 (2014)

    CrossRef  Google Scholar 

  12. Castelli, M., Trujillo, L., Vanneschi, L.: Energy consumption forecasting using semantic-based genetic programming with local search optimizer. Comput. Intel. Neurosci. 2015, 971908 (2015)

    Google Scholar 

  13. Choi, W.-J., Choi, T.-S.: Genetic programming-based feature transform and classification for the automatic detection of pulmonary nodules on computed tomography images. Inf. Sci. 212, 57–78 (2012)

    CrossRef  Google Scholar 

  14. Gonçalves-de-Silva, K., Aloise, D., Xavier-de-Souza, S., Mladenovic, N.: Less is more: simplified Nelder-Mead method for large unconstrained optimization. Yugosl. J. Oper. Res. 28, 153–169 (2018)

    MathSciNet  MATH  CrossRef  Google Scholar 

  15. Costa, L.R., Aloise, D., Mladenović, N.: Less is more: basic variable neighborhood search heuristic for balanced minimum sum-of-squares clustering. Inf. Sci. 415-416, 247–253 (2017)

    Google Scholar 

  16. de Arruda Pereira, M., Davis Júnior, C.A., Gontijo Carrano, E., de Vasconcelos, J.A.A.: A niching genetic programming-based multi-objective algorithm for hybrid data classification. Neurocomputing 133, 342–357 (2014)

    CrossRef  Google Scholar 

  17. De Boor, C.: A Practical Guide to Splines: With 32 Figures. Springer, Berlin (2001)

    MATH  Google Scholar 

  18. Deklel, A.K., Saleh, M.A., Hamdy, A.M., Saad, E.M.: Transfer learning with long term artificial neural network memory (LTANN-MEM) and neural symbolization algorithm (NSA) for solving high dimensional multi-objective symbolic regression problems. In: 2017 34th National Radio Science Conference (NRSC), pp. 343–352. IEEE, Piscataway (2017)

    Google Scholar 

  19. Elleuch, S., Jarboui, B., Mladenovic, N.: Variable neighborhood programming – a new automatic programming method in artificial intelligence. Technical report, G-2016-92, GERAD, Montreal (2016)

    Google Scholar 

  20. Elleuch, S., Hansen, P., Jarboui, B., Mladenović, N.: New VNP for automatic programming. Elect. Notes Discrete Math. 58, 191–198 (2017)

    MATH  CrossRef  Google Scholar 

  21. Eurostat. https://ec.europa.eu/eurostat/data/database

  22. Fernandez de Canete, J., Del Saz-Orozco, P., Baratti, R., Mulas, M., Ruano, A., Garcia-Cerezo, A.: Soft-sensing estimation of plant effluent concentrations in a biological wastewater treatment plant using an optimal neural network. Expert Syst. Appl. 63, 8–19 (2016)

    Google Scholar 

  23. Friedman, J. H.: Multivariate adaptive regression splines. Annal. Stat. 19(1), 1–67 (1991)

    MathSciNet  MATH  Google Scholar 

  24. Galar, M., Fernández, A., Barrenechea, E., Bustince, H., Herrera, F.: An overview of ensemble methods for binary classifiers in multi-class problems: experimental study on one-vs-one and one-vs-all schemes. Patt. Recog. 44(8), 1761–1776 (2011)

    CrossRef  Google Scholar 

  25. García-Torres, M., Gómez-Vela, F., Melián-Batista, B., Moreno-Vega, J.M.: High-dimensional feature selection via feature grouping: a variable neighborhood search approach. Inf. Sci. 326, 102–118 (2016)

    MathSciNet  CrossRef  Google Scholar 

  26. Ghaddar, B., Sakr, N., Asiedu, Y.: Spare parts stocking analysis using genetic programming. Europ. J. Oper. Res. 252(1), 136–144 (2016)

    CrossRef  Google Scholar 

  27. Graupe, D.: Principles of Artificial Neural Networks. World Scientific, Singapore (2007)

    MATH  CrossRef  Google Scholar 

  28. Guler, H.: Prediction of railway track geometry deterioration using artificial neural networks: a case study for Turkish state railways. Struct. Infrastruct. Eng. 10(5), 614–626 (2014)

    CrossRef  Google Scholar 

  29. Gustavsson, E., Patriksson, M., Strömberg, A.-B., Wojciechowski, A., Önnheim, M.: Preventive maintenance scheduling of multi-component systems with interval costs. Comput. Ind. Eng. 76, 390–400 (2014)

    CrossRef  Google Scholar 

  30. Hansen, P., Mladenović, N.: Variable neighborhood search. In: Search Methodologies, pp. 211–238. Springer, Boston (2005)

    Google Scholar 

  31. Hansen, P., Mladenović, N., Pérez, JAM.: Variable neighbourhood search: methods and applications. Ann. Oper. Res. 175(1), 367–407

    Google Scholar 

  32. He, Q., Li, H., Bhattacharjya, D., Parikh, D.P., Hampapur, A.: Track geometry defect rectification based on track deterioration modelling and derailment risk assessment. J. Oper. Res. Soc. 66(3), 392–404 (2015)

    CrossRef  Google Scholar 

  33. Healthcare in Russia. Stat. book./Rosstat (2006)

    Google Scholar 

  34. Healthcare in Russia. Stat. book./Rosstat (2007)

    Google Scholar 

  35. Healthcare in Russia. Stat. book./Rosstat (2009)

    Google Scholar 

  36. Healthcare in Russia. Stat. book./Rosstat (2011)

    Google Scholar 

  37. Healthcare in Russia. Stat. book./Rosstat (2015)

    Google Scholar 

  38. Healthcare in Russia. Stat. book./Rosstat (2017)

    Google Scholar 

  39. Health at a Glance 2017: OECD indicators. http://dx.doi.org/10.1787/888933602215

  40. Health at a Glance 2017: OECD indicators. http://dx.doi.org/10.1787/888933602272

  41. Health care: current status and possible development scenarios. In: Dokl. to the 18th April International Scientific conference on the Problems of Economic and Social Development, Moscow, April 11–14, 2017. House of the Higher School of Economics (2017)

    Google Scholar 

  42. Hoai, N., McKay, R., Essam, D., Chau, R.: Solving the symbolic regression problem with tree-adjunct grammar guided genetic programming: the comparative results. In: Proceedings of the 2002 Congress on Evolutionary Computation. CEC’02 (Cat. No. 02TH8600), vol. 2, pp. 1326–1331. IEEE, Piscataway (2002)

    Google Scholar 

  43. Hoang, T.-H., Essam, D., McKay, B., Hoai, N.-X.: Building on success in genetic programming: adaptive variation and developmental evaluation. In: Advances in Computation and Intelligence, pp. 137–146. Springer, Berlin (2007)

    Google Scholar 

  44. Howard, D., Roberts, S., Brankin, R.: Target detection in SAR imagery by genetic programming. Adv. Eng. Softw. 30(5), 303–311 (1999)

    CrossRef  Google Scholar 

  45. Icke, I., Bongard, J.C.: Improving genetic programming based symbolic regression using deterministic machine learning. In: 2013 IEEE Congress on Evolutionary Computation, pp. 1763–1770. IEEE, Piscataway (2013)

    Google Scholar 

  46. Jaba, E., Balan, C.B., Robu, I.-B.: The relationship between life expectancy at birth and health expenditures estimated by a cross-country and time-series analysis. Proc. Eco. Finance 15, 108–114 (2014). Emerging Markets Queries in Finance and Business (EMQ 2013).

    Google Scholar 

  47. Jiaqiu, W., Ioannis, T., Chen, Z.: A space–time delay neural network model for travel time prediction. Eng. Appl. Artif. Intell. 52, 145–160 (2016)

    CrossRef  Google Scholar 

  48. Johnson, C.G.: Genetic Programming Crossover: Does It Cross over? pp. 97–108. Springer, Berlin (2009)

    Google Scholar 

  49. Kantardzic, M.: Data Mining Concepts, Models, Methods, and Algorithms. Wiley-IEEE Press, Hoboken (2011)

    MATH  CrossRef  Google Scholar 

  50. Karaboga, D., Ozturk, C., Karaboga, N., Gorkemli, B.: Artificial bee colony programming for symbolic regression. Inf. Sci. 209, 1–15 (2012)

    CrossRef  Google Scholar 

  51. Keijzer, M.: Improving Symbolic Regression with Interval Arithmetic and Linear Scaling, pp. 70–82. Springer, Berlin (2003)

    Google Scholar 

  52. Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992)

    MATH  Google Scholar 

  53. Koza, J.R.: Genetic Programming II: Automatic Discovery of Reusable Programs. MIT Press, Cambridge (1994)

    MATH  Google Scholar 

  54. Kristjanpoller, W., Minutolo, M.C.: Forecasting volatility of oil price using an artificial neural network-GARCH model. Expert Syst. Appl. 65, 233–241 (2016)

    CrossRef  Google Scholar 

  55. Lalonde, M.: A new perspective on the health of Canadians. Technical report (1994)

    Google Scholar 

  56. Lamson, S.T., Hastings, N.A.J., Willis, R.J.: Minimum cost maintenance in heavy haul rail track. J. Oper. Res. Soc. 34(3), 211 (1983)

    CrossRef  Google Scholar 

  57. Lane, F., Azad, R., Ryan, C.: On effective and inexpensive local search techniques in genetic programming regression. In: Parallel Problem Solving from Nature – PPSN XIII, vol. 8672. Lecture Notes in Computer Science. Springer International Publishing, Berlin (2014)

    Google Scholar 

  58. Lidén, T.: Railway infrastructure maintenance – a survey of planning problems and conducted research. Trans. Res. Proc. 10, 574–583 (2015)

    Google Scholar 

  59. Life expectancy increasing in Russia, experts claim. https://www.vedomosti.ru/economics/articles/2018/05/29/770996-rosta-prodolzhitelnosti-zhizni

  60. Lubitz, J., Cai, L., Kramarow, E., Lentzner, H.: Health, life expectancy, and health care spending among the elderly. N. Engl. J. Med. 349(11), 1048–1055 (2003). PMID: 12968089

    CrossRef  Google Scholar 

  61. Ly, D.L., Lipson, H.: Learning symbolic representations of hybrid dynamical systems. J. Mach. Learn. Res. 13(Dec), 3585–3618 (2012)

    MathSciNet  MATH  Google Scholar 

  62. Macchi, M., Garetti, M., Centrone, D., Fumagalli, L., Piero Pavirani, G.: Maintenance management of railway infrastructures based on reliability analysis. Reliab. Eng. Syst. Saf. 104, 71–83 (2012)

    CrossRef  Google Scholar 

  63. Mladenović, N., Hansen, P.: Variable neighborhood search. Comput. Oper. Res. 24(11), 1097–1100 (1997)

    MathSciNet  MATH  CrossRef  Google Scholar 

  64. Mladenović, N., Urošević, D.: Variable Neighborhood Search for the K-Cardinality Tree. Metaheuristics: Computer Decision-Making, Applied Optimization. Springer, Boston (2003)

    Google Scholar 

  65. Mladenović, N., Todosijević, R., Urošević, D.: Less is more: basic variable neighborhood search for minimum differential dispersion problem. Inf. Sci. 326, 160–171 (2016)

    CrossRef  Google Scholar 

  66. Mladenović, M., Delot, T., Laporte, G., Wilbaut, C.: The parking allocation problem for connected vehicles. J. Heuristics 26, 377–399 (2020)

    CrossRef  Google Scholar 

  67. Mladenović, N., Alkandari, A., Pei, J., Todosijević, R., Pardalos, P.M.: Less is more approach: basic variable neighborhood search for the obnoxious p-median problem. Int. Trans. Oper. Res. 27(1), 480–493 (2020)

    MathSciNet  CrossRef  Google Scholar 

  68. Muggleton, S., de Raedt, L.: Inductive logic programming: theory and methods. J. Logic Program. 19–20, 629–679 (1994)

    MathSciNet  MATH  CrossRef  Google Scholar 

  69. Musilek, P., Lau, A., Reformat, M., Wyardscott, L.: Immune programming. Inf. Sci. 176(8), 972–1002 (2006)

    MathSciNet  MATH  CrossRef  Google Scholar 

  70. Nguyen, S., Zhang, M., Member, S., Johnston, M., Tan, K.C.: Automatic programming via iterated local search for dynamic job shop scheduling. IEEE Trans. Cybern. 45(1), 1–14 (2015)

    CrossRef  Google Scholar 

  71. O’Neill, M., Brabazon, A.: Grammatical swarm. In: Genetic and Evolutionary Computation Conference (GECCO), pp. 163–174. Springer, Berlin (2004)

    Google Scholar 

  72. Pei, J., Mladenović, N., Urošević, D., Brimberg, J., Liu, X.: Solving the traveling repairman problem with profits: a novel variable neighborhood search approach. Inf. Sci. 507, 108–123 (2020)

    MathSciNet  MATH  CrossRef  Google Scholar 

  73. Peng, F., Kang, S., Li, X., Ouyang, Y., Somani, K., Acharya, D.: A heuristic approach to the railroad track maintenance scheduling problem. Comput. Aided Civ. Inf. Eng. 26(2), 129–145 (2011)

    CrossRef  Google Scholar 

  74. Peng, F., Ouyang, Y., Somani, K.: Optimal routing and scheduling of periodic inspections in large-scale railroad networks. J. Rail Transp. Plann. Manage. 3(4), 163–171 (2013)

    CrossRef  Google Scholar 

  75. Peng, Y., Yuan, C., Qin, X., Huang, J., Shi, Y.: An improved gene expression programming approach for symbolic regression problems. Neurocomputing 137, 293–301 (2014)

    CrossRef  Google Scholar 

  76. Rad, H.I., Feng, J., Iba, H.: GP-RVM: genetic programming-based symbolic regression using relevance vector machine. ArXiv: 1806,02502v (2018)

    Google Scholar 

  77. Roux, O., Cyril, F.: Ant programming: or how to use ants for automatic programming. In: International Conference on Swarm Intelligence, pp. 121–129 (2000)

    Google Scholar 

  78. Russian statistical yearbook. Rosstat (2018)

    Google Scholar 

  79. Shcherbakova, E.: Life expectancy and health care in OECD countries. Technical Report, Demoscope weekly (2018)

    Google Scholar 

  80. Stadtmüller, U.: Asymptotic properties of nonparametric curve estimates. Period. Math. Hung. 17(2), 83–108 (1986)

    MathSciNet  MATH  CrossRef  Google Scholar 

  81. Uy, N.Q., Hoai, N.X., O’Neill, M., McKay, R.I., Galván-López, E.: Semantically-based crossover in genetic programming: application to real-valued symbolic regression. Genet. Program Evolvable Mach. 12(2), 91–119 (2011)

    CrossRef  Google Scholar 

  82. Witten, I.H., Frank, E., Hall, M.A.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, Burlington (2011)

    Google Scholar 

  83. Wolpert, D., Macready, W.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)

    CrossRef  Google Scholar 

  84. Wong, P., Zhang, M.: SCHEME: caching subtrees in genetic programming. In: 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 2678–2685. IEEE, Piscataway (2008)

    Google Scholar 

  85. World health statistics 2017: monitoring health for the SDGs, Sustainable Development Goals. Technical Report, World Health Organization (2017)

    Google Scholar 

  86. Yao, X., Liu, Y., Lin, G.: Evolutionary programming made faster. IEEE Trans. Evol. Comput. 3(2), 82–102 (1999)

    CrossRef  Google Scholar 

  87. Yousefikia, M., Moridpour, S., Setunge, S., Mazloumi, E.: Modeling degradation of tracks for maintenance planning on a tram line. J. Traffic Logist. Eng. 2(2), 86–91 (2014)

    CrossRef  Google Scholar 

  88. Zhao, J., Chan, A.H.C., Stirling, A.B., Madelin, K.B.: Optimizing policies of railway ballast tamping and renewal. Trans. Res. Record J. Trans. Res. Board 1943(1), 50–56 (2006)

    CrossRef  Google Scholar 

  89. Zhao, J., Chan, A.H.C., Burrow, M.P.N.: Reliability analysis and maintenance decision for railway sleepers using track condition information. J. Oper. Res. Soc. 58(8), 1047–1055 (2007)

    MATH  CrossRef  Google Scholar 

Download references

Acknowledgements

This publication is partially supported by the Khalifa University of Science and Technology under Award No. RC2 DSO. This research is also partially supported by the framework of Grant BR05236839, development of information technologies and systems for stimulation of personality’s sustainable development as one of the bases of development of digital Kazakhstan.

Author information

Authors and Affiliations

  1. Department of Industrial and Systems Engineering, Research Center on Digital Supply Chain and Operations Management, Khalifa University, Abu Dhabi, UAE

    Nenad Mladenovic

  2. Higher Colleges of Technology, Abu Dhabi, UAE

    Bassem Jarboui

  3. Department of Management Information Systems, College of Business and Economics, Qassim University, Buraydah, Saudi Arabia

    Souhir Elleuch

  4. Institute of Information and Computational Technologies, Almata, Kazakhstan

    Rustam Mussabayev

  5. Leontief Centre, Saint Petersburg State University of Economics, Saint Petersburg, Russia

    Olga Rusetskaya

Authors
  1. Nenad Mladenovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bassem Jarboui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Souhir Elleuch
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rustam Mussabayev
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Olga Rusetskaya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nenad Mladenovic .

Editor information

Editors and Affiliations

  1. Department of Industrial & Systems Engineering, University of Florida, Gainesville, FL, USA

    Dr. Panos M. Pardalos

  2. Moscow Aviation Institute, Moscow, Russia

    Varvara Rasskazova

  3. Mathematics Department, University of Patras, Patras, Greece

    Michael N. Vrahatis

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Mladenovic, N., Jarboui, B., Elleuch, S., Mussabayev, R., Rusetskaya, O. (2021). Variable Neighborhood Programming as a Tool of Machine Learning. In: Pardalos, P.M., Rasskazova, V., Vrahatis, M.N. (eds) Black Box Optimization, Machine Learning, and No-Free Lunch Theorems. Springer Optimization and Its Applications, vol 170. Springer, Cham. https://doi.org/10.1007/978-3-030-66515-9_9

Download citation