Algorithms for Continuous Optimization pp 531-559 | Cite as

# Parallel Nonlinear Optimization: Limitations, Challenges, and Opportunities

## Abstract

The availability and power of parallel computers is having a significant impact on how large-scale problems are solved in all areas of numerical computation, and is likely to have an even larger impact in the future. This paper attempts to give some indication of how the consideration of parallel computation is affecting, and is likely to affect, the field of nonlinear optimization. It does not attempt to survey the research that has been done in parallel nonlinear optimization. Rather it presents a set of examples, mainly from our own research, that is intended to illustrate many of the limitations, opportunities, and challenges inherent in incorporating parallelism into the field of nonlinear optimization. These examples include parallel methods for small to medium size unconstrained optimization problems, parallel methods for large block bordered systems of nonlinear equations, and parallel methods for both small and large-scale global optimization problems. Our overall conclusions are mixed. For generic, small to medium size problems, the consideration of parallelism does not appear to be leading to major algorithmic innovations. For many classes of large-scale problems, however, the consideration of parallelism appears to be creating opportunities for the development of interesting new methods that may be advantageous for parallel and possibly even sequential computation. In addition, a number of large-scale parallel optimization algorithms exhibit irregular, coarse-grain structure, which leads to interesting computer science challenges in their implementation.

## Keywords

Global Optimization Parallel Algorithm Shared Memory Line Search Implicit Method## Preview

Unable to display preview. Download preview PDF.

## References

- [1]G.S. Almasi and A. Gottlieb,
*Highly Parallel Computing*, Benjamin Cummings, Redwood City, CA, 1989.MATHGoogle Scholar - [2]R.H. Byrd, T. Derby, E. Eskow, K.P.B. Oldenkamp, and R.B. Schnabel, “A new stochastic/perturbation method for large-scale global optimization and its application to water cluster problems,”
*Large-Scale Optimization: State of the Art*, W. Hager, D. Hearn, and P. Pardalos, eds., Kluwer Academic Publishers, 1994, pp. 71–84Google Scholar - [3]R. Byrd, T. Derby, E. Eskow, K. Oldenkamp, R.B. Schnabel, and C. Triantafillou, “Parallel global optimization methods for molecular configuration problems”,
*Proceedings of Sixth SIAM Conference of Parallel Processing for Scientific Computation*, SIAM, Philadelphia, 1993, pp. 165–169.Google Scholar - [4]R.H. Byrd, C.L. Dert, A.H.G. Rinnooy Kan, and R.B. Schnabel, “Concurrent stochastic methods for global optimization”,
*Mathematical Programming*46, 1990, pp. 1–30.MathSciNetMATHCrossRefGoogle Scholar - [5]R.H. Byrd, E. Eskow, and R.B. Schnabel, “A new large-scale global optimization method and its application to Lennard-Jones problems”, University of Colorado Technical Report CS-CS-630–92, 1992.Google Scholar
- [6]R. H. Byrd, E. Eskow, R. B. Schnabel, and S. L. Smith, “Parallel global optimization: numerical methods, dynamic scheduling methods, and applications to molecular configuration”,
*Parallel Computation*, B. Ford and A. Fincham, eds., Oxford University Press, 1993, pp. 187–207.Google Scholar - [7]R.H. Byrd, R.B. Schnabel, and G.A. Shultz, “Parallel quasi-Newton methods for unconstrained optimization,”
*Mathematical Programming*42, 1988, pp. 273–306.MathSciNetMATHCrossRefGoogle Scholar - [8]R.H. Byrd, R.B. Schnabel, and G.A. Shultz, “Using parallel function evaluations to improve Hessian approximations for unconstrained optimization,”
*Annals of Operations Research*14, 1988, pp. 167–193.MathSciNetCrossRefGoogle Scholar - [9]D.F. Coker and R.O. Watts, “Structure and vibrational spectroscopy of the water dimer using quantum simulation”,
*J. Phys. Chem.*91, 1987, pp. 2513–2518.CrossRefGoogle Scholar - [10]T. Coleman, D. Shalloway, and Z. Wu, “A parallel build-up algorithm for global energy minimizations of molecular clusters using effective energy simulated annealing.” Technical Report CTC93TR130, Advanced Computing Research Institute, Cornell University, 1993.Google Scholar
- [11]J. E. Dennis and R. B. Schnabel,
*Numerical Methods for Unconstrained Optimization and Nonlinear Equations*, Prentice-Hall, Englewood Cliffs, NJ, 1983.MATHGoogle Scholar - [12]J.E. Dennis Jr. and V. Torczon, “Direct search methods on parallel computers,
*SIAM Journal on Optimization*1, 1991, pp. 448–474.MathSciNetMATHCrossRefGoogle Scholar - [13]D. Feng and R.B. Schnabel, “Globally convergent parallel algorithms for solving block bordered systems of nonlinear equations”,
*Optimization Methods and Software*2, 1993, pp. 269–295.CrossRefGoogle Scholar - [14]R. Fletcher, Practical Methods of Optimization, Second Edition, John Wiley & Sons, New York, 1987.MATHGoogle Scholar
- [15]G. Fox, S. Hiranandani, K. Kennedy, C. Koelbel, U. Kremer, C. Tseng, and M. Wu, “Fortran D language specification”, Center for Research on Parallel Computation Technical Report CRPC-TR90079, 1990.Google Scholar
- [16]P.D. Frank and G.R. Shubin, “A comparison of optimization-based approaches for a model computational aerodynamics design problem”,
*Journal of Computational Physics*98, 1992, pp. 74–89.MATHCrossRefGoogle Scholar - [17]P. E. Gill, W. Murray, and M. H. Wright,
*Practical Optimization*, Academic Press, London, 1981.MATHGoogle Scholar - [18]G. Golub and J.M. Ortega,
*Scientific Computing — An Introduction with Parallel Computing*, Academic Press, Boston, 1993.MATHGoogle Scholar - [19]L. Grandinetti, “Factorization versus nonfactorization in quasi-Newtonian methods for differentiable optimization,” Report N5, Dipartimento di Sistemi, Universitá della Calabria, 1978.Google Scholar
- [20]S. P. Han, “Optimization by updated conjugate subspaces”,
*Numerical Analysis: Pitman Research Notes in Mathematics Series 140*, D. F. Griffiths and G. A. Watson, eds., Longman Scientific and Technical, Burnt Mill, England, 1986, pp. 82–97.Google Scholar - [21]A. Griewank, “On automatic differentiation,”
*Mathematical Programming: Recent Developments and Applications*, M. Iri and K. Tanabe, eds., Kluwer Academic Publishers, Tokyo, 1989, pp. 83–108.Google Scholar - [22]R. Horst and H. Tuy,
*Global Optimization, Deterministic Approaches*, Springer Verlag, Berlin, 1992.Google Scholar - [23]X. Long, “Molecular dynamics simulations of clusters -- impure van der Waals and
*e*^{−}− (*H*_{2}*O*)_{n}systems”, Ph.D. Dissertation, Department of Chemistry, University of California, San Diego, 1992.Google Scholar - [24]R.S. Maier, J.B. Rosen, and G.L. Xue, “A discrete-continuous algorithm for molecular energy minimization”, AHPCRC Preprint 92–031, University of Minnesota, 1992.Google Scholar
- [25]S.G. Nash and A. Sofer, “Block truncated-Newton methods for parallel optimization,”
*Mathematical Programming*45, 1989, pp. 529–546.MathSciNetMATHCrossRefGoogle Scholar - [26]S.G. Nash and A. Sofer, “A general-purpose parallel algorithm for unconstrained optimization,”
*SIAM Journal on Optimization*1, 1991, pp. 530–547.MathSciNetMATHCrossRefGoogle Scholar - [27]J. Nocedal, private communication.Google Scholar
- [28]J.A. Northby, “Structure and binding of Lennard-Jones clusters: 13 ≤
*N*≤ 147”,*J. Chem. Phys.*87, 1987, pp. 6166–6177.CrossRefGoogle Scholar - [29]M.J. Quinn,
*Parallel Computing -- Theory and Practice*, McGraw Hill, New York, 1994.Google Scholar - [30]A.H.G. Rinnooy Kan and G.T. Timmer, “Stochastic methods for global optimization”,
*American Journal of Mathematical and Management Sciences*4, 1984, pp. 7–40.MathSciNetMATHGoogle Scholar - [31]A.H.G. Rinnooy Kan and G.T. Timmer, “Global Optimization”,
*Handbooks in Operations Research and Management Science, Volume I: Optimization*, G.L Nemhauser, A.H.J. Rinnooy Kan, and M.J. Todd, eds., North-Holland, 1989, pp. 631–662.Google Scholar - [32]S. Smith, “Adaptive Asynchronous Parallel Algorithms in Distributed Computation,” Ph.D. Thesis, University of Colorado at Boulder, 1991.Google Scholar
- [33]S. Smith, E. Eskow, and R.B. Schnabel, “Adaptive, asynchronous stochastic global optimization for sequential and parallel computation”,
*Large Scale Numerical Optimization*, T. Coleman and Y. Li, eds., SIAM, Philadelphia, 1990, pp. 207–227.Google Scholar - [34]S. Smith and R.B. Schnabel, “Centralized and distributed dynamic scheduling for adaptive parallel algorithms”,
*Unstructured Scientific Computation on Scalable Multiprocessors*, P. Mehrotra, J. Saltz, and R. Voigt, eds., MIT Press, Cambridge, Mass., 1992, pp. 301–321.Google Scholar - [35]S. Smith and R.B. Schnabel, “Dynamic scheduling strategies for an adaptive asynchronous parallel global optimization algorithm,” University of Colorado Technical Report CU-CS-625–92, 1992.Google Scholar
- [36]G.L. Steele, Jr., “High performance Fortran: status report”, in
*Proceedings of Workshop on Languages, Compilers, and Run-Time Environments for Distributed Memory Multiprocessors, SIGPLAN Notices*, Jan. 1993, pp. 1–4.Google Scholar - [37]H.W.J.M. Trienekens, “Parallel Branch and Bound Algorithms,” Ph.D. Thesis, Erasmus University, Rotterdam, The Netherlands, 1990.Google Scholar
- [38]P.J. van Laarhoven, “Parallel variable metric methods for unconstrained optimization,”
*Mathematical Programming*33, 1985, pp. 68–81.MathSciNetMATHCrossRefGoogle Scholar - [39]X. Zhang, R.H. Byrd, and R.B. Schnabel, “Parallel methods for solving nonlinear block bordered systems of equations,”
*SIAM Journal on Scientific and Statistical Computing*, 13, 1992, pp. 841–859.MathSciNetMATHCrossRefGoogle Scholar