Abstract
A Manhattan search algorithm to minimize artificial neural network error function is outlined in this paper. From an existing position in Cartesian coordinate, a search vector moves in orthogonal directions to locate minimum function value. The search algorithm computes optimized step length for rapid convergence. This step is performed when consecutive search is successful in minimizing function value. The optimized step length identifies favorable descent direction to minimize function value. The search method is suitable for complex error surface where derivative information is difficult to obtain or when the error surface is nearly flat. The rate of change in function value is almost negligible near the flat surface. Most of the derivative based training algorithm faces difficulty in such scenarios. This algorithm avoids derivative information of an error function. Therefore, it is an attractive search method when derivative based algorithm faces difficulty due to complex ridges and flat valleys. In case the algorithm gets into trapped minimum, the search vector takes steps to move out of a local minimum by exploring neighborhood descent search directions. The algorithm differs from the first and second order derivative based training methods. To measure the performance of the algorithm, estimation of electric energy generation model from Fiji Islands and “L-T” letter recognition problems are solved. Bootstrap analysis shows that the algorithm’s predictive and classification abilities are high. The algorithm is reliable when solution to a problem is unknown. Therefore, the algorithm identifies benchmark solution.
Similar content being viewed by others
References
Ahmed S (2010) Multi-directional search to optimize neural network error function. Kybernetes 39(7):1145–1166
Alrefaei MH, Andradóttir S (1999) A simulated annealing algorithm with constant temperature for discrete stochastic optimization. Manag Sci 45:748–764
Audet C, Orban D (2006) Finding optimal algorithmic parameters using derivative free optimization. SIAM J Optim 17(3):642–664
Carcangiu Sara, Carcangiu Alessandra Fanni, Augusto Montisci (2009) A constructive algorithm of neural approximation models for optimization problems. Int J Comput Math Electr Electron Eng 28(5):1276–1289
Conn AR, Scheinberg K, Vicente LN (2008) Geometry of interpolation sets in derivative free optimization. Math Program Ser B 111:141–172
Efron B, Tibshirani R (1993) An introduction to bootstrap. Chapman and Hall, New York
Erkmen Burcu, Yıldırım Tulay (2008) Improving classification performance of sonar targets by applying general regression neural network with PCA. Expert Syst Appl 35(1–2):472–475
Gerencsér L, Hill SD, Vágó Z (1999) Optimization over discrete sets via SPSA. In: Proceedings of the 38th conference on decision and, control, pp 1791–1795
Ghosh R, Ghosh M, Yearwood J, Bagirov A (2005) Determining regularization parameters for derivative free neural learning. In: Proceeding MLDM’05 Proceedings of the 4th international conference on machine learning and data mining in pattern recognition
Tawfeig H, Vijanth S (2011) Predicting flow rate of V-shape custom tank using derivative free recursive algorithm. J Appl Sci 11:1279–1284
Hecht-Nielsen R (1990) NeuroComputing. Addison-Wesley Publishing Company, Reading
Hesterberg T, Monaghan S, Moore DS, Clipson A, Epstein R (2003) Bootstrap methods and permutation tests. W.H. Freeman and Company, New York
Hong LJ, Nelson BL (2006) Discrete optimization via simulation using COMPASS. Oper Res 54:115–129
Hong LJ, Nelson BL (2007) Selecting the best system when systems are revealed sequentially. IIE Trans 39:723–734
Hooke R, Jeeves TA (1961) Direct search solution of numerical and statistical problems. J Assoc Comput Mach 8:212–229
Hush DR, Horne B, Salas JM (1992) Error surfaces for multilayer. IEEE Trans Syst Man Cybern 22:1152–1161
Moore JJ, Wild SM (2009) Benchmarking derivative free optimization algorithms. SIAM J Optim 20(1):172–191
Jacobs RA (1988) Increased rate of convergence through learning rate adaptation. Neural Netw 1:295–307
Kamarthi SV, Pittner S (1999) Accelerating neural network training using weight extrapolations. Neural Netw 12:1285–1299
Price RK, Spitznagel EL, Downey TJ, Meyer DJ, Risk NK, El-Ghazzawy OG (2000) Applying artificial neural network models to clinical decision making. Psychol Assess 12(1):40–51
Kleywegt A, Shapiro A, Homem-de-Mello T (2001) The sample average approximation method for stochastic discrete optimization. SIAM J Optim 12:479–502
Knoke JD, Anderson CM, Koch GG (2006) Analyzing repeated measures marginal models on sample surveys with resampling methods. J Stat Softw 15(8):1–13
Kordos M, Duch Kordos W (2008) Variable step search algorithm for feedforward networks, Neurocomputing. Corrected Proof, Available online 29 April 2008 (in press)
Krzyzak A, Dai W, Suen CY (1990) Classification of Large set of Handwritten Characters Using Modified Back Propagation Model. In: Proceedings of the international joint conference on neural networks III:225–232. IEEE Press, Piscataway, NJ
Takashi Kuremoto, Obayashi Kuremoto Masanao, Kobayashi Kunikazu (2009) Adaptive swarm behavior acquisition by a neuro-fuzzy system and reinforcement learning algorithm. Int J Intell Comput Cybern 2(4):724–744
Derong Liu, Zhang Huaguang (2008) Neural networks: algorithms and applications. Neurocomputing 71(4–6):471–473
Mirikitani Derrick T, Nikolaev Nikolay (2010) Efficient online recurrent connectionist learning with the ensemble Kalman filter. Neurocomputing 73(4–6):1024–1030
Mosteller F, Tukey (1968) Data analysis including statistics. In: Lindzey G, Aronson E (eds) Handbook of social psychology 2. Addision-Wesley, Reading Mass
Polak E, Ribiere G (1969) Note Sur la Convergence de Methods de Directions Conjures. Revue Francaise Information Recherche Operationnelle 16:35–43
Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representation by error propagation. In: Rumelhart DE, McClelland JL, PDP research group (eds) Parallel distributed processing: explorations in the microstructure of cognition, vol 1. foundations. MIT Press, Cambridge, MA, USA, pp 318–362
Saini Lalit Mohan (2008) Peak load forecasting using Bayesian regularization, resilient and adaptive back propagation learning based artificial neural networks. Electr Power Syst Res 78(7):1302–1310
Salmalian K, Soleimani M, Rouhi S (2012) Fatigue life modeling and prediction of GRP composites using multi-objective evolutionary optimized neural networks. Int J Math Models Methods Appl Sci 1(6):1–10
Salomon R, Van Hemmen L (1996) Accelerating back propagation through dynamic self-adaptation. Neural Netw 9(4):589–601
Shi L, Ólafsson S (2000) Nested partitions method for stochastic optimization. Methodol Comput Appl Probab 2:271–291
Snee RD (1977) Some aspects of nonorthogonal data analysis, Part I. Developing prediction equations. J Qual Technol 5:67–79 Springer, Berlin, Heidelberg
Stone M (1974) Cross-validation choice and assessment of statistical predictions (with discussions). J R Stat Soc Ser B 36:111–147
Torczon V (1997) On the convergence of pattern search algorithms. SIAM J Control Optim 7(1):1–25
Van Ooyen A, Nienhuis B (1992) Improving the convergence of the back propagation algorithm. Neural Netw 5:465–471
Vogl TP, Mangis JK, Rigler AK, Zink WT, Alkon DL (1988) Accelerating the convergence of the back-propagation method. Biol Cybern 59:257–263
Tai-Yue Wang, Chien-Yu Huang Wang (2008) Optimizing back-propagation networks via a calibrated heuristic algorithm with an orthogonal array. Expert Syst Appl 34(3):1630–1641
Yan D, Mukai H (1992) Stochastic discrete optimization. SIAM J Control Optim 30:594–612
Yang Xin-She, Benjamin Bronner, Leo Trottier, Nick Orbeck, James Meiss, Eugene M Izhikevich (2011) Metaheuristic Optimization. Scholarpedia 6(8)11472
Zhang C, Wu W, Chen XH, Xiong Y (2008) Convergence of BP algorithm for product unit neural networks with exponential weights. Neurocomputing 72(1—-3):513–520
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ahmed, S. Performance of derivative free search ANN training algorithm with time series and classification problems. Comput Stat 28, 1881–1914 (2013). https://doi.org/10.1007/s00180-012-0386-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-012-0386-1