# Time Series Forecasting Using Neural Networks: Are Recurrent Connections Necessary?

- 50 Downloads

## Abstract

Artificial neural networks (NNs) are widely used in modeling and forecasting time series. Since most practical time series are non-stationary, NN forecasters are often implemented using recurrent/delayed connections to handle the temporal component of the time varying sequence. These recurrent/delayed connections increase the number of weights required to be optimized during training of the NN. Particle swarm optimization (PSO) is now an established method for training NNs, and was shown in several studies to outperform the classical backpropagation training algorithm. The original PSO was, however, designed for static environments. In dealing with non-stationary data, modified versions of PSOs for optimization in dynamic environments are used. These dynamic PSOs have been successfully used to train NNs on classification problems under non-stationary environments. This paper formulates training of a NN forecaster as dynamic optimization problem to investigate if recurrent/delayed connections are necessary in a NN time series forecaster when a dynamic PSO is used as the training algorithm. Experiments were carried out on eight forecasting problems. For each problem, a feedforward NN (FNN) is trained with a dynamic PSO algorithm and the performance is compared to that obtained from four different types of recurrent NNs (RNN) each trained using gradient descent, a standard PSO for static environments and the dynamic PSO algorithm. The RNNs employed were an Elman NN, a Jordan NN, a multirecurrent NN and a time delay NN. The performance of these forecasting models were evaluated under three different dynamic environmental scenarios. The results show that the FNNs trained with the dynamic PSO significantly outperformed all the RNNs trained using any of the other algorithms considered. These findings highlight that recurrent/delayed connections are not necessary in NNs used for time series forecasting (for the time series considered in this study) as long as a dynamic PSO algorithm is used as the training method.

## Keywords

Time series forecasting Neural networks Recurrent neural networks Resilient propagation Particle swarm optimization Cooperative quantum particle swarm optimization## Notes

## References

- 1.Abdulkarim SA (2016) Time series prediction with simple recurrent neural networks. Bayero J Pure Appl Sci 9(1):19–24CrossRefGoogle Scholar
- 2.Abdulkarim SA (2018) Time series forecasting using dynamic particle swarm optimizer trained neural networks. Ph.d. thesis, University of PretoriaGoogle Scholar
- 3.Adhikari R, Agrawal R (2011) Effectiveness of PSO based neural network for seasonal time series forecasting. In: Proceedings of the fifth Indian international conference on artificial intelligence, pp 231–244Google Scholar
- 4.Blackwell T, Bentley P (2002) Dynamic search with charged swarms. In: Proceedings of the genetic and evolutionary computation conference, pp 9–16Google Scholar
- 5.Blackwell T, Branke J (2006) Multiswarms, exclusion, and anti-convergence in dynamic environments. IEEE Trans Evol Comput 10(4):459–472CrossRefGoogle Scholar
- 6.Blackwell T, Branke J, Li X (2008) Particle swarms for dynamic optimization problems. Swarm intelligence. Springer, Berlin, pp 193–217Google Scholar
- 7.Chatterjee S, Hore S, Dey N, Chakraborty S, Ashour AS (2017) Dengue fever classification using gene expression data: a PSO based artificial neural network approach. In: Proceedings of the 5th international conference on frontiers in intelligent computing: theory and applications. Springer, Berlin, pp 331–341Google Scholar
- 8.Clerc M, Kennedy J (2002) The particle swarm-explosion, stability, and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73CrossRefGoogle Scholar
- 9.Deb K, Joshi D, Anand A (2002) Real-coded evolutionary algorithms with parent-centric recombination. IEEE Proc Congr Evol Comput 1:61–66Google Scholar
- 10.Dorffner G (1996) Neural networks for time series processing. Neural Netw World 6:447–468Google Scholar
- 11.Duhain J (2011) Particle swarm optimization in dynamically changing environment an empirical study. M.Sc. thesis, University of PretoriaGoogle Scholar
- 12.Eberhart R, Kennedy J (1995) A new optimizer using particle swarm theory. In: Proceedings of the sixth international symposium on micro machine and human science, vol 1. pp 39–43Google Scholar
- 13.Eberhart R, Shi Y (2000) Comparing inertia weights and constriction factors in particle swarm optimization. Proc IEEE Congr Evol Comput 1:84–88Google Scholar
- 14.Elman J (1990) Finding structure in time. Cogn Sci 14:179–211CrossRefGoogle Scholar
- 15.Engelbrecht A (2007) Computational intelligence: an introduction. Wiley, New YorkCrossRefGoogle Scholar
- 16.Hamzacebi C (2008) Improving artificial neural networks performance in seasonal time series forecasting. Inf Sci 178(23):4550–4559CrossRefGoogle Scholar
- 17.Han HG, Lu W, Hou Y, Qiao JF (2018) An adaptive-PSO-based self-organizing RBF neural network. IEEE Trans Neural Netw Learn Syst 29(1):104–117MathSciNetCrossRefGoogle Scholar
- 18.Harrison K, Ombuki-berman B, Engelbrecht A (2016) A radius-free quantum particle swarm optimization technique for dynamic optimization problems. In: Proceedings of IEEE congress on evolutionary computation, pp 578–585Google Scholar
- 19.Hore S, Chatterjee S, Santhi V, Dey N, Ashour AS, Balas VE, Shi F (2017) Indian sign language recognition using optimized neural networks. In: Information technology and intelligent transportation systems, Springer, pp 553–563Google Scholar
- 20.Hyndman R (2013) Time series data library. http://robjhyndman.com/TSDL. Accessed 2015 Dec 05
- 21.Jha G, Thulasiraman P, Thulasiram R (2009) PSO based neural network for time series forecasting. In: Proceedings of IEEE international joint conference on neural networks, pp 1422–1427Google Scholar
- 22.Jordan M (1986) Attractor dynamics and parallellism in a connectionist sequential machine. In: Eighth annual conference of the cognitive science society. pp 513–546Google Scholar
- 23.Kennedy J, Mendes R (2002) Population structure and particle swarm performance. Proc IEEE Congr Evol Comput 2:1671–1676Google Scholar
- 24.Lawal I, Abdulkarim S, Hassan M, Sadiq J (2016) Improving HSDPA traffic forecasting using ensemble of neural networks. In: Proceedings of 15th IEEE international conference machine learning and applications. IEEE, pp 308–313Google Scholar
- 25.LeCun Y, Bottou L, Orr G, Müller K (2012) Efficient Backprop. Neural networks: tricks of the trade. Springer, Berlin, pp 9–48Google Scholar
- 26.Li X, Dam K (2003) Comparing particle swarms for tracking extrema in dynamic environments. IEEE Proc Evol Comput 3:1772–1779Google Scholar
- 27.Mann H, Whitney D (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 18(1):50–60MathSciNetCrossRefzbMATHGoogle Scholar
- 28.Mao C, Lin R, Xu C, He Q (2017) Towards a trust prediction framework for cloud services based on PSO-driven neural network. IEEE Access 5:2187–2199CrossRefGoogle Scholar
- 29.Mendes R, Cortez P, Rocha M, Neves J (2002) Particle swarms for feedforward neural network training. In: Proceedings of IEEE international joint conference on neural networks, pp 1895–1899Google Scholar
- 30.Morrison R (2003) Performance measurement in dynamic environments. In: Proceedings of the GECCO workshop on evolutionary algorithms for dynamic optimization problems, pp 5–8Google Scholar
- 31.Muralitharan K, Sakthivel R, Vishnuvarthan R (2018) Neural network based optimization approach for energy demand prediction in smart grid. Neurocomputing 273:199–208CrossRefGoogle Scholar
- 32.Parsaie A, Azamathulla HM, Haghiabi AH (2018) Prediction of discharge coefficient of cylindrical weir-gate using GMDH-PSO. ISH J Hydraul Eng 24(2):116–123CrossRefGoogle Scholar
- 33.Rakitianskaia A, Engelbrecht A (2008) Cooperative charged particle swarm optimiser. In: IEEE congress on evolutionary computation, pp 933–939Google Scholar
- 34.Rakitianskaia A, Engelbrecht A (2009) Training neural networks with PSO in dynamic environments. In: Proceedings of IEEE congress on evolutionary computation, pp 667–673Google Scholar
- 35.Rakitianskaia A, Engelbrecht A (2012) Training feedforward neural networks with dynamic particle swarm optimisation. Swarm Intell 6(3):233–270CrossRefGoogle Scholar
- 36.Rakitianskaia A, Engelbrecht A (2015) Saturation in PSO neural network training: good or evil ? IEEE Congr Evolut Comput 2:125–132Google Scholar
- 37.Rezaee J (2014) Particle swarm optimisation for dynamic optimisation problems: a review. Neural Compu Appl 25:1507–1516CrossRefGoogle Scholar
- 38.Riedmiller M (1994) Rprop—description and implementation details. Technical ReportGoogle Scholar
- 39.Riedmiller M, Braun H (1993) A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: IEEE international conference on neural networks, pp 586–591Google Scholar
- 40.Röbel A (1994) The dynamic pattern selection algorithm: effective training and controlled generalization of backpropagation neural networks. In: Technical report. Tech-nische Universität BerlinGoogle Scholar
- 41.Tang Z, Fishwick P (1993) Feedforward neural nets as models for time series forecasting. ORSA J Comput 5(4):374–385CrossRefzbMATHGoogle Scholar
- 42.Unger N, Ombuki-Berman B, Engelbrecht A (2013) Cooperative particle swarm optimization in dynamic environments. In: Proceedings of IEEE symposium on swarm intelligence, pp 172–179Google Scholar
- 43.Van Den Bergh F (2001) An analysis of particle swarm optimizers. Ph.d. thesis, University of PretoriaGoogle Scholar
- 44.Van den Bergh F, Engelbrecht A (2001) Effects of swarm size on cooperative particle swarm optimisers. In: Proceedings of the 3rd annual conference on genetic and evolutionary computation, pp 892–899Google Scholar
- 45.Van den Bergh F, Engelbrecht A (2004) A cooperative approach to particle swarm optimization. IEEE Trans Evol Comput 8(3):225–239CrossRefGoogle Scholar
- 46.Waibel A, Hanazawa T, Hinton G, Shikano K, Lang KJ (1990) Phoneme recognition using time-delay neural networks. In: Readings in speech recognition. Elsevier, pp 393–404Google Scholar
- 47.Wessels L, Barnard E (1992) Avoiding false local minima by proper initialization of connections. IEEE Trans Neural Netw 3(6):899–905CrossRefGoogle Scholar
- 48.Wyk A, Engelbrecht A (2010) Overfitting by PSO trained feedforward neural networks. In: IEEE congress on evolutionary computation, pp 1–8Google Scholar