Distributed Recurrent Neural Network Learning via Metropolis-Weights Consensus
When data are shared among arbitrarily connected machines, the training process became an interesting challenge where each node is initialized with a specific scalar value, so it present a problem of computing their average taking into account interconnectivity between agents in order to ensure that the objective process converges as the centralized counterpart, the decentralized average consensus (DAC) is the most popular strategy due to its low-complexity. In this paper a random topology is choosing to validate a network of agents with a given probability of interconnectivity between every pair of neighbors nodes, the global regularized least-square problem requires an optimization procedure to solve it with decentralized fashion then, the question is what is the optimal output weight vector that we have to choose for the test task, here the DAC intervenes to encourage all agents having the same vectors or we will be on the case of local training, so we must choose appropriately the DAC strategy in order that all agents converge to the same state. The contribution key is to apply the Metropolis-Weights as a strategy of average consensus to compute the mean of the updates of nodes at each step with several tests, this protocol demonstrate convergence of the consensus algorithm for network without packet losses. Experimental results on prediction and identification tasks show a favorable performance in terms of accuracy and efficiency.
KeywordsDistributed learning Metropolis-Weights Recurrent neural network Alternating direction method of multipliers
The research leading to these results has received funding from the Ministry of Higher Education and Scientific Research of Tunisia under the grant agreement number LR11ES48.
- 3.Fischione, C.: Distributed Estimation, Lecture 9 Principles of Wireless Sensor Networks (2009)Google Scholar
- 4.Zinkevich, M., Langford, J., Smola, A.J.: Slow learners are fast. In: Advances in Neural Information Processing Systems, pp. 2331–2339 (2009)Google Scholar
- 7.Bezine, H., Alimi, M.A., Derbel, N: Handwriting trajectory movements controlled by a bêta-elliptic model. In: Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, p. 1228 (2003)Google Scholar
- 11.Flouri, K., Beferull-Lozano, B., Tsakalides, P: Training a SVM-based classifier in distributed sensor networks. In: 14th European Signal Processing Conference, pp. 1–5 (2006)Google Scholar
- 14.Boubaker, H., Kherallah, M., Alimi, M.A.: New algorithm of straight or curved baseline detection for short arabic handwritten writing. In: Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, p. 778 (2009)Google Scholar
- 18.Elloumi, W., Alimi, M.A.: A more efficient MOPSO for optimization. In: ACS/IEEE International Conference on Computer System and Applications (AICCSA) (2010)Google Scholar
- 19.Guijarro-Berdiñas, B., Martínez-Rego, D., Fernández-Lorenzo, S.: Privacy-preserving distributed learning based on genetic algorithms and artificial neural networks. In: Omatu, S., Rocha, M.P., Bravo, J., Fernández, F., Corchado, E., Bustillo, A., Corchado, J.M. (eds.) IWANN 2009. LNCS, vol. 5518, pp. 195–202. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-02481-8_27 CrossRefGoogle Scholar
- 20.Jaeger, H.: Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the “echo state network” approach, vol. 5. GMD-Forschungszentrum Informationstechnik (2002)Google Scholar
- 22.Xiao, L., Boyd, S., Lall, S.: Distributed average consensus with time-varying metropolis weights. Automatica (2006)Google Scholar
- 23.Dhahri, H., Alimi, M.A.: The modified differential evolution and the RBF (MDE-RBF) neural network for time series prediction. In: IEEE International Conference on Neural Networks - Conference Proceedings, p. 2938 (2006)Google Scholar
- 25.Elbaati, A., Boubaker, H., Kherallah, M., Alimi, M.A., Ennaji, A., Abed, H.E.: Arabic handwriting recognition using restored stroke chronology. In: Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, p. 411 (2009)Google Scholar