Skip to main content
SpringerLink
Log in

Performance of derivative free search ANN training algorithm with time series and classification problems

  • Original Paper
  • Published:
Computational Statistics Aims and scope Submit manuscript

Abstract

A Manhattan search algorithm to minimize artificial neural network error function is outlined in this paper. From an existing position in Cartesian coordinate, a search vector moves in orthogonal directions to locate minimum function value. The search algorithm computes optimized step length for rapid convergence. This step is performed when consecutive search is successful in minimizing function value. The optimized step length identifies favorable descent direction to minimize function value. The search method is suitable for complex error surface where derivative information is difficult to obtain or when the error surface is nearly flat. The rate of change in function value is almost negligible near the flat surface. Most of the derivative based training algorithm faces difficulty in such scenarios. This algorithm avoids derivative information of an error function. Therefore, it is an attractive search method when derivative based algorithm faces difficulty due to complex ridges and flat valleys. In case the algorithm gets into trapped minimum, the search vector takes steps to move out of a local minimum by exploring neighborhood descent search directions. The algorithm differs from the first and second order derivative based training methods. To measure the performance of the algorithm, estimation of electric energy generation model from Fiji Islands and “L-T” letter recognition problems are solved. Bootstrap analysis shows that the algorithm’s predictive and classification abilities are high. The algorithm is reliable when solution to a problem is unknown. Therefore, the algorithm identifies benchmark solution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Ahmed S (2010) Multi-directional search to optimize neural network error function. Kybernetes 39(7):1145–1166

    Article  MATH  Google Scholar 

  • Alrefaei MH, Andradóttir S (1999) A simulated annealing algorithm with constant temperature for discrete stochastic optimization. Manag Sci 45:748–764

    Article  MATH  Google Scholar 

  • Audet C, Orban D (2006) Finding optimal algorithmic parameters using derivative free optimization. SIAM J Optim 17(3):642–664

    Article  MathSciNet  MATH  Google Scholar 

  • Carcangiu Sara, Carcangiu Alessandra Fanni, Augusto Montisci (2009) A constructive algorithm of neural approximation models for optimization problems. Int J Comput Math Electr Electron Eng 28(5):1276–1289

    Article  MATH  Google Scholar 

  • Conn AR, Scheinberg K, Vicente LN (2008) Geometry of interpolation sets in derivative free optimization. Math Program Ser B 111:141–172

    Article  MathSciNet  MATH  Google Scholar 

  • Efron B, Tibshirani R (1993) An introduction to bootstrap. Chapman and Hall, New York

    Book  MATH  Google Scholar 

  • Erkmen Burcu, Yıldırım Tulay (2008) Improving classification performance of sonar targets by applying general regression neural network with PCA. Expert Syst Appl 35(1–2):472–475

    Article  Google Scholar 

  • Gerencsér L, Hill SD, Vágó Z (1999) Optimization over discrete sets via SPSA. In: Proceedings of the 38th conference on decision and, control, pp 1791–1795

  • Ghosh R, Ghosh M, Yearwood J, Bagirov A (2005) Determining regularization parameters for derivative free neural learning. In: Proceeding MLDM’05 Proceedings of the 4th international conference on machine learning and data mining in pattern recognition

  • Tawfeig H, Vijanth S (2011) Predicting flow rate of V-shape custom tank using derivative free recursive algorithm. J Appl Sci 11:1279–1284

    Article  Google Scholar 

  • Hecht-Nielsen R (1990) NeuroComputing. Addison-Wesley Publishing Company, Reading

    Google Scholar 

  • Hesterberg T, Monaghan S, Moore DS, Clipson A, Epstein R (2003) Bootstrap methods and permutation tests. W.H. Freeman and Company, New York

    Google Scholar 

  • Hong LJ, Nelson BL (2006) Discrete optimization via simulation using COMPASS. Oper Res 54:115–129

    Article  MATH  Google Scholar 

  • Hong LJ, Nelson BL (2007) Selecting the best system when systems are revealed sequentially. IIE Trans 39:723–734

    Article  Google Scholar 

  • Hooke R, Jeeves TA (1961) Direct search solution of numerical and statistical problems. J Assoc Comput Mach 8:212–229

    Article  MATH  Google Scholar 

  • Hush DR, Horne B, Salas JM (1992) Error surfaces for multilayer. IEEE Trans Syst Man Cybern 22:1152–1161

    Article  Google Scholar 

  • Moore JJ, Wild SM (2009) Benchmarking derivative free optimization algorithms. SIAM J Optim 20(1):172–191

    Article  MathSciNet  Google Scholar 

  • Jacobs RA (1988) Increased rate of convergence through learning rate adaptation. Neural Netw 1:295–307

    Article  Google Scholar 

  • Kamarthi SV, Pittner S (1999) Accelerating neural network training using weight extrapolations. Neural Netw 12:1285–1299

    Article  Google Scholar 

  • Price RK, Spitznagel EL, Downey TJ, Meyer DJ, Risk NK, El-Ghazzawy OG (2000) Applying artificial neural network models to clinical decision making. Psychol Assess 12(1):40–51

    Article  Google Scholar 

  • Kleywegt A, Shapiro A, Homem-de-Mello T (2001) The sample average approximation method for stochastic discrete optimization. SIAM J Optim 12:479–502

    Google Scholar 

  • Knoke JD, Anderson CM, Koch GG (2006) Analyzing repeated measures marginal models on sample surveys with resampling methods. J Stat Softw 15(8):1–13

    Google Scholar 

  • Kordos M, Duch Kordos W (2008) Variable step search algorithm for feedforward networks, Neurocomputing. Corrected Proof, Available online 29 April 2008 (in press)

  • Krzyzak A, Dai W, Suen CY (1990) Classification of Large set of Handwritten Characters Using Modified Back Propagation Model. In: Proceedings of the international joint conference on neural networks III:225–232. IEEE Press, Piscataway, NJ

  • Takashi Kuremoto, Obayashi Kuremoto Masanao, Kobayashi Kunikazu (2009) Adaptive swarm behavior acquisition by a neuro-fuzzy system and reinforcement learning algorithm. Int J Intell Comput Cybern 2(4):724–744

    Article  MathSciNet  MATH  Google Scholar 

  • Derong Liu, Zhang Huaguang (2008) Neural networks: algorithms and applications. Neurocomputing 71(4–6):471–473

    Google Scholar 

  • Mirikitani Derrick T, Nikolaev Nikolay (2010) Efficient online recurrent connectionist learning with the ensemble Kalman filter. Neurocomputing 73(4–6):1024–1030

    Article  Google Scholar 

  • Mosteller F, Tukey (1968) Data analysis including statistics. In: Lindzey G, Aronson E (eds) Handbook of social psychology 2. Addision-Wesley, Reading Mass

    Google Scholar 

  • Polak E, Ribiere G (1969) Note Sur la Convergence de Methods de Directions Conjures. Revue Francaise Information Recherche Operationnelle 16:35–43

    MathSciNet  Google Scholar 

  • Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representation by error propagation. In: Rumelhart DE, McClelland JL, PDP research group (eds) Parallel distributed processing: explorations in the microstructure of cognition, vol 1. foundations. MIT Press, Cambridge, MA, USA, pp 318–362

  • Saini Lalit Mohan (2008) Peak load forecasting using Bayesian regularization, resilient and adaptive back propagation learning based artificial neural networks. Electr Power Syst Res 78(7):1302–1310

    Article  Google Scholar 

  • Salmalian K, Soleimani M, Rouhi S (2012) Fatigue life modeling and prediction of GRP composites using multi-objective evolutionary optimized neural networks. Int J Math Models Methods Appl Sci 1(6):1–10

    Google Scholar 

  • Salomon R, Van Hemmen L (1996) Accelerating back propagation through dynamic self-adaptation. Neural Netw 9(4):589–601

    Article  Google Scholar 

  • Shi L, Ólafsson S (2000) Nested partitions method for stochastic optimization. Methodol Comput Appl Probab 2:271–291

    Article  MathSciNet  MATH  Google Scholar 

  • Snee RD (1977) Some aspects of nonorthogonal data analysis, Part I. Developing prediction equations. J Qual Technol 5:67–79 Springer, Berlin, Heidelberg

    Google Scholar 

  • Stone M (1974) Cross-validation choice and assessment of statistical predictions (with discussions). J R Stat Soc Ser B 36:111–147

    MATH  Google Scholar 

  • Torczon V (1997) On the convergence of pattern search algorithms. SIAM J Control Optim 7(1):1–25

    Article  MathSciNet  MATH  Google Scholar 

  • Van Ooyen A, Nienhuis B (1992) Improving the convergence of the back propagation algorithm. Neural Netw 5:465–471

    Article  Google Scholar 

  • Vogl TP, Mangis JK, Rigler AK, Zink WT, Alkon DL (1988) Accelerating the convergence of the back-propagation method. Biol Cybern 59:257–263

    Article  Google Scholar 

  • Tai-Yue Wang, Chien-Yu Huang Wang (2008) Optimizing back-propagation networks via a calibrated heuristic algorithm with an orthogonal array. Expert Syst Appl 34(3):1630–1641

    Article  Google Scholar 

  • Yan D, Mukai H (1992) Stochastic discrete optimization. SIAM J Control Optim 30:594–612

    Article  MathSciNet  MATH  Google Scholar 

  • Yang Xin-She, Benjamin Bronner, Leo Trottier, Nick Orbeck, James Meiss, Eugene M Izhikevich (2011) Metaheuristic Optimization. Scholarpedia 6(8)11472

  • Zhang C, Wu W, Chen XH, Xiong Y (2008) Convergence of BP algorithm for product unit neural networks with exponential weights. Neurocomputing 72(1—-3):513–520

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Business, The University of the South Pacific, Suva, Fiji

    Shamsuddin Ahmed

Authors
  1. Shamsuddin Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shamsuddin Ahmed.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ahmed, S. Performance of derivative free search ANN training algorithm with time series and classification problems. Comput Stat 28, 1881–1914 (2013). https://doi.org/10.1007/s00180-012-0386-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00180-012-0386-1

Keywords

Search

Navigation