Computational Mathematical Programming pp 357-381 | Cite as

# Parallel Computing in Optimization

## Abstract

One of the major developments in computing in recent years has been the introduction of a variety of parallel computers, and the development of algorithms that effectively utilize their capabilities. Very little of this parallel algorithm development, however, has been in numerical optimization. Nevertheless, significant opportunities exist for the utilization of parallelism in optimization, especially on computers that support independent concurrent processes. This paper first gives a very brief survey of parallel architectures and general characteristics of parallel algorithms. Next we indicate what we see as the leading opportunities for the utilization of parallelism in optimization. Then we survey the small amount of existing research in parallel optimization; most of this has been conducted at The Hatfield Polytechnic. Finally we discuss some recently initiated research at the University of Colorado concerned with solving optimization problems by parallel algorithms suitable for implementation on a local area network of computers; we focus on a new parallel algorithm for global optimization.

## Keywords

Parallel Algorithm Conjugate Gradient Method Local Area Network Parallel Architecture Sequential Algorithm## Preview

Unable to display preview. Download preview PDF.

## References

- L. Adams [1983], “An M-step preconditioned conjugate gradient method for parallel computation”,
*Proceedings of the 1983 International Conference on Parallel Processing*, pp. 36–43.Google Scholar - L. Adams and J. Ortega [1982], “A multi-color SOR method for parallel computation”,
*Proceedings of the 1982 International Conference on Parallel Processing*, pp. 53–56.Google Scholar - G. Baudet [1978], “Asynchronous iterative methods for multiprocessors”,
*Journal of the Association for Computing Machinery*25, pp. 226–244.Google Scholar - C. G. E. Boender, A. H. G. Rinnooy Kan, L. Stougie, and G. T. Timmer [1982], “A stochastic method for global optimization”,
*Mathematical Programming*22, pp. 125–140.Google Scholar - J. E. Dennis Jr. and R. B. Schnabel [1983],
*Numerical Methods for Nonlinear Equations and Unconstrained Optimization*, Prentice-Hall, Englewood Cliffs, New Jersey.Google Scholar - L. C. W. Dixon [1981], “The place of parallel computation in numerical optimization I, the local problem”, Technical Report No. 118, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- L. C. W. Dixon and K. D. Patel [1981], “The place of parallel computation in numerical optimization II, the multiextremal global optimisation problem”, Technical Report No. 119, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- L. C. W. Dixon and K. D. Patel [1982], “The place of parallel computation in numerical optimization IV, parallel algorithms for nonlinear optimisation”, Technical Report No. 125, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- L. C. W. Dixon, K. D. Patel, and P. G. Ducksbury [1983], “Experience running optimisation algorithms on parallel processing systems”, Technical Report No. 138, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- L. C. W. Dixon, P. G. Ducksbury, and P. Singh [1982], “A parallel version of the conjugate gradient algorithm for finite element problems”, Technical Report No. 132, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- J. J. Dongarra and R. E. Hiromoto [1983], “A collection of parallel linear equations routines for the Denelcor HEP”, Technical Report ANL/MCS-TM-15, Mathematics and Computer Science Divsion, Argonne National Laboratory.Google Scholar
- P. G. Ducksbury [1982], “The implementation of a parallel version of Price’s (CRS) algorithm on an ICL DAP”, Technical Report No. 127, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- I. S. Duff [1983], “The solution of sparse linear systems on the Cray-1”, Technical Report CSS 125 (revised), Computer Science and Systems Divsion, AERE Harwell.Google Scholar
- B. Feijoo and R. R. Meyer [1984], “Piecewise-linear approximation methods for non-separable convex optimization”, Technical Report No. 521, Computer Sciences Department, University of Wisconsin — Madison (to appear).Google Scholar
- M. J. Flynn [1966], “Very high-speed computing systems”,
*Proceedings of the IEEE*54, pp. 1901–1909.Google Scholar - K. W. Fong and T. L. Jordan [1977], “Some linear algebraic algorithms and their performance on the Cray-1”, Report LA-6774, Los Alamos National Laboratory.Google Scholar
- P. E. Gill, W. Murray, and M. H. Wright [1981],
*Practical Optimization*, Academic Press, London.Google Scholar - A. O. Griewank and Ph. L. Toint [1982], “On the unconstrained optimization of partially separable functions”, in
*Nonlinear Optimization 1981*, M. J. D. Powell ed., Academic Press, London, pp. 301–312.Google Scholar - D. Heller [1978], “A survey of parallel algorithms in numerical linear algebra”,
*SIAM Review*20, pp. 740-777. pp. 409–436.Google Scholar - R. W. Hockney and C. R. Jesshope [1981],
*Parallel Computers*, Adam-Hilger Ltd.,. Bristol, England.Google Scholar - E. C. Housos and O. Wing [1980], “Parallel nonlinear minimization by conjugate directions”,
*Proceedings of the 1980 International Conference on Parallel Processing*, pp. 157–158.Google Scholar - K. Hwang and F. A. Briggs [1984],
*Computer Architecture and Parallel Processing*, McGraw-Hill, New York.Google Scholar - T. L. Jordan [1979], “A performance evaluation of linear algebra software in parallel architectures”, in
*Performance Evaluation of Numerical Software*, L. D. Fosdick, ed., North-Holland, Amsterdam, pp. 59–76.Google Scholar - R. Kapur and J. Browne [1981], “Block tridiagonal system solution on reconfigurable array computers”,
*Proceedings of the 1981 International Conference on Parallel Processing*, pp. 92–99.Google Scholar - J. Kowalik and S. P. Kumar [1982], “An efficient parallel block conjugate gradient method for linear equations”,
*Proceedings of the 1982 International Conference on Parallel Processing*, pp. 47–52.Google Scholar - H. T. Kung [976], “Synchronized and asynchronous parallel algorithms for multiprocessors”, in
*Algorithms and Complexity: Recent Results and New Directions*, J. E. Traub, ed., Addison-Wesley, pp. 153–200.Google Scholar - B. W. Lampson, M. Paul, and H. J. Siegert [1981], eds.,
*Distributed Systems — Architecture and Implementation*, Springer-Verlag, Berlin.Google Scholar - A. V. Levy, A. Montalvo, S. Gomez, and A. Calderon [1981], “Topics in global optimization”, in
*Proceedings of the Third IIMAS Workshop*, Cocoyoc, Mexico, January 1981, J. P. Hennart, ed.Google Scholar - R. Lord, J. Kowalik, and S. Kumar [1980], “Solving linear algebraic equations on a MIMD computer”,
*Proceedings of the 1980 International Conference on Parallel Processing*, pp. 205–210.Google Scholar - J. J. McKeown [1979], “Experiments in implementing a nonlinear least-squares algorithm on a dual-processor computer”, Technical Report No. 102, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- J. Mohan [1982], “A study in parallel computation — the traveling salesman problem”, Technical Report CMU-CS-82-136, Department of Computer Science, Carnegie-Mellon University.Google Scholar
- K. D. Patel [1982b], “Implementation of a parallel (SIMD) modified Newton method on the ICL DAP”, Technical Report No. 131, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- K. D. Patel [1982a], “Parallel Computation and Numerical Optimisation”, Technical Report No. 129, Numerical Optimisation Centre, The Hatfield Polytechnic.Google Scholar
- W. L. Price [1981], “A new version of the controlled random search procedure for global optimisation”, Technical Report, Engineering Department, University of Leicester, England.Google Scholar
- A. H. G. Rinnooy Kan and G. T. Timmer [1983], “Stochastic methods for global optimization”, Report 8317/0, Econometric Institute, Erasmus University, Rotterdam.Google Scholar
- G. Rodrigue [1982], ed.,
*Parallel Computations*, Academic Press, New York.Google Scholar - J. B. Rosen [1983], “Global minimization of a linearly constrained concave function by partition of feasible domain”,
*Mathematics of Operations Research*8, pp.Google Scholar - A. Samen [1977], “Numerical parallel algorithms — a survey”, in
*High Speed Computer and Algorithm Organization*, D. Kuck, D. Lawrie, and A. Sameh, eds., Academic Press, pp. 207–228.Google Scholar - L. J. Siegel [1983], “Characteristics of Parallel Algorithms”, presented at the Taxonomy of Parallel Algorithms Workshop, Santa Fe, New Mexico, December 1983.Google Scholar
- T. A. Straeter [1973], “A parallel variable metric optimization algorithm,” NASA Technical Note D-7329, Langley Reseach Center, Hampton, Virginia.Google Scholar
- T. A. Straeter and A. T. Markos [1975], “A parallel Jacobson-Oksman optimization algorithm,” NASA Technical Note D-8020, Langley Reseach Center, Hampton, Virginia.Google Scholar
- H. van der Vorst [1982], “A vectorizable variant of some ICCG methods”,
*SIAM Journal on Scientific and Statistical Computing*3, pp. 350–356.Google Scholar - P. van Laarhoven [1984], “Parallel algorithms for unconstrained optimization,”
*Mathematical Programming*, to appear.Google Scholar