Optimal speedup conditions for a parallel back-propagation algorithm
One way for implementing a parallel back-propagation algorithm is based on distributing the examples to be learned among different processors. This method provides with spectacular speedups for each epoch of back-propagation learning, but it shows a major drawback: parallelization implies alterations of the gradient descent algorithm. This paper presents an implementation of this parallel algorithm on a transputer network. It mentions experimental laws about the back-propagation convergence speed, and claims that optimal conditions still exist for performing an actual speedup by implementing such a parallel algorithm. It points out theoretical and experimental optimal conditions, in terms of the number of processors and the size of the example packets.
Unable to display preview. Download preview PDF.
- F.Desprez, B.Tourancheau, Modélisation des performances de communication sur le Tnode avec le Logical system transputer toolset, La lettre du transputer et des calculateurs distribués 7 (Lab. d'Info, de Besançon, 1990) 65–72Google Scholar
- P.Fraigniaud, S.Miguet, Y.Robert, Scattering on a ring of processors, Parallel Computing 13 (North Holland, 1990) 377–383Google Scholar
- J.Ghosh, K.Hwang, Mapping neural networks onto message-passing multicomputers, Journal of Parallel and Distributed Computing 6 (Academic Press, 1989) 291–330Google Scholar
- H.Paugam-Moisy, On the convergence of a block-gradient algorithm for backpropagaton learning, Proc. of IJCNN'92-Baltimore (1992) III-919-924Google Scholar
- A.Singer, Implementations of artificial neural networks on the Connection Machine, Parallel Computing 14 (North-Holland, 1990) 305–316Google Scholar
- [6 ]A.Singer, Exploiting the inherent parallelism of artificial neural networks to achieve 1300 M interconnects per second, Proc. of INNC-90-Paris (1990) 656–660Google Scholar
- S.Wang, Communication problems in simulating neural networks on a hypercube machine, 1st European Workshop on Hypercube and Distributed Computers (1988)Google Scholar