Parallel Nonlinear Optimization: Limitations, Challenges, and Opportunities

  • Robert B. Schnabel
Part of the NATO ASI Series book series (ASIC, volume 434)


The availability and power of parallel computers is having a significant impact on how large-scale problems are solved in all areas of numerical computation, and is likely to have an even larger impact in the future. This paper attempts to give some indication of how the consideration of parallel computation is affecting, and is likely to affect, the field of nonlinear optimization. It does not attempt to survey the research that has been done in parallel nonlinear optimization. Rather it presents a set of examples, mainly from our own research, that is intended to illustrate many of the limitations, opportunities, and challenges inherent in incorporating parallelism into the field of nonlinear optimization. These examples include parallel methods for small to medium size unconstrained optimization problems, parallel methods for large block bordered systems of nonlinear equations, and parallel methods for both small and large-scale global optimization problems. Our overall conclusions are mixed. For generic, small to medium size problems, the consideration of parallelism does not appear to be leading to major algorithmic innovations. For many classes of large-scale problems, however, the consideration of parallelism appears to be creating opportunities for the development of interesting new methods that may be advantageous for parallel and possibly even sequential computation. In addition, a number of large-scale parallel optimization algorithms exhibit irregular, coarse-grain structure, which leads to interesting computer science challenges in their implementation.


Global Optimization Parallel Algorithm Shared Memory Line Search Implicit Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    G.S. Almasi and A. Gottlieb, Highly Parallel Computing, Benjamin Cummings, Redwood City, CA, 1989.zbMATHGoogle Scholar
  2. [2]
    R.H. Byrd, T. Derby, E. Eskow, K.P.B. Oldenkamp, and R.B. Schnabel, “A new stochastic/perturbation method for large-scale global optimization and its application to water cluster problems,” Large-Scale Optimization: State of the Art, W. Hager, D. Hearn, and P. Pardalos, eds., Kluwer Academic Publishers, 1994, pp. 71–84Google Scholar
  3. [3]
    R. Byrd, T. Derby, E. Eskow, K. Oldenkamp, R.B. Schnabel, and C. Triantafillou, “Parallel global optimization methods for molecular configuration problems”, Proceedings of Sixth SIAM Conference of Parallel Processing for Scientific Computation, SIAM, Philadelphia, 1993, pp. 165–169.Google Scholar
  4. [4]
    R.H. Byrd, C.L. Dert, A.H.G. Rinnooy Kan, and R.B. Schnabel, “Concurrent stochastic methods for global optimization”, Mathematical Programming 46, 1990, pp. 1–30.MathSciNetzbMATHCrossRefGoogle Scholar
  5. [5]
    R.H. Byrd, E. Eskow, and R.B. Schnabel, “A new large-scale global optimization method and its application to Lennard-Jones problems”, University of Colorado Technical Report CS-CS-630–92, 1992.Google Scholar
  6. [6]
    R. H. Byrd, E. Eskow, R. B. Schnabel, and S. L. Smith, “Parallel global optimization: numerical methods, dynamic scheduling methods, and applications to molecular configuration”, Parallel Computation, B. Ford and A. Fincham, eds., Oxford University Press, 1993, pp. 187–207.Google Scholar
  7. [7]
    R.H. Byrd, R.B. Schnabel, and G.A. Shultz, “Parallel quasi-Newton methods for unconstrained optimization,” Mathematical Programming 42, 1988, pp. 273–306.MathSciNetzbMATHCrossRefGoogle Scholar
  8. [8]
    R.H. Byrd, R.B. Schnabel, and G.A. Shultz, “Using parallel function evaluations to improve Hessian approximations for unconstrained optimization,” Annals of Operations Research 14, 1988, pp. 167–193.MathSciNetCrossRefGoogle Scholar
  9. [9]
    D.F. Coker and R.O. Watts, “Structure and vibrational spectroscopy of the water dimer using quantum simulation”, J. Phys. Chem. 91, 1987, pp. 2513–2518.CrossRefGoogle Scholar
  10. [10]
    T. Coleman, D. Shalloway, and Z. Wu, “A parallel build-up algorithm for global energy minimizations of molecular clusters using effective energy simulated annealing.” Technical Report CTC93TR130, Advanced Computing Research Institute, Cornell University, 1993.Google Scholar
  11. [11]
    J. E. Dennis and R. B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall, Englewood Cliffs, NJ, 1983.zbMATHGoogle Scholar
  12. [12]
    J.E. Dennis Jr. and V. Torczon, “Direct search methods on parallel computers, SIAM Journal on Optimization 1, 1991, pp. 448–474.MathSciNetzbMATHCrossRefGoogle Scholar
  13. [13]
    D. Feng and R.B. Schnabel, “Globally convergent parallel algorithms for solving block bordered systems of nonlinear equations”, Optimization Methods and Software 2, 1993, pp. 269–295.CrossRefGoogle Scholar
  14. [14]
    R. Fletcher, Practical Methods of Optimization, Second Edition, John Wiley & Sons, New York, 1987.zbMATHGoogle Scholar
  15. [15]
    G. Fox, S. Hiranandani, K. Kennedy, C. Koelbel, U. Kremer, C. Tseng, and M. Wu, “Fortran D language specification”, Center for Research on Parallel Computation Technical Report CRPC-TR90079, 1990.Google Scholar
  16. [16]
    P.D. Frank and G.R. Shubin, “A comparison of optimization-based approaches for a model computational aerodynamics design problem”, Journal of Computational Physics 98, 1992, pp. 74–89.zbMATHCrossRefGoogle Scholar
  17. [17]
    P. E. Gill, W. Murray, and M. H. Wright, Practical Optimization, Academic Press, London, 1981.zbMATHGoogle Scholar
  18. [18]
    G. Golub and J.M. Ortega, Scientific Computing — An Introduction with Parallel Computing, Academic Press, Boston, 1993.zbMATHGoogle Scholar
  19. [19]
    L. Grandinetti, “Factorization versus nonfactorization in quasi-Newtonian methods for differentiable optimization,” Report N5, Dipartimento di Sistemi, Universitá della Calabria, 1978.Google Scholar
  20. [20]
    S. P. Han, “Optimization by updated conjugate subspaces”, Numerical Analysis: Pitman Research Notes in Mathematics Series 140, D. F. Griffiths and G. A. Watson, eds., Longman Scientific and Technical, Burnt Mill, England, 1986, pp. 82–97.Google Scholar
  21. [21]
    A. Griewank, “On automatic differentiation,” Mathematical Programming: Recent Developments and Applications, M. Iri and K. Tanabe, eds., Kluwer Academic Publishers, Tokyo, 1989, pp. 83–108.Google Scholar
  22. [22]
    R. Horst and H. Tuy, Global Optimization, Deterministic Approaches, Springer Verlag, Berlin, 1992.Google Scholar
  23. [23]
    X. Long, “Molecular dynamics simulations of clusters -- impure van der Waals and e − (H 2 O)n systems”, Ph.D. Dissertation, Department of Chemistry, University of California, San Diego, 1992.Google Scholar
  24. [24]
    R.S. Maier, J.B. Rosen, and G.L. Xue, “A discrete-continuous algorithm for molecular energy minimization”, AHPCRC Preprint 92–031, University of Minnesota, 1992.Google Scholar
  25. [25]
    S.G. Nash and A. Sofer, “Block truncated-Newton methods for parallel optimization,” Mathematical Programming 45, 1989, pp. 529–546.MathSciNetzbMATHCrossRefGoogle Scholar
  26. [26]
    S.G. Nash and A. Sofer, “A general-purpose parallel algorithm for unconstrained optimization,” SIAM Journal on Optimization 1, 1991, pp. 530–547.MathSciNetzbMATHCrossRefGoogle Scholar
  27. [27]
    J. Nocedal, private communication.Google Scholar
  28. [28]
    J.A. Northby, “Structure and binding of Lennard-Jones clusters: 13 ≤N≤ 147”, J. Chem. Phys. 87, 1987, pp. 6166–6177.CrossRefGoogle Scholar
  29. [29]
    M.J. Quinn, Parallel Computing -- Theory and Practice, McGraw Hill, New York, 1994.Google Scholar
  30. [30]
    A.H.G. Rinnooy Kan and G.T. Timmer, “Stochastic methods for global optimization”, American Journal of Mathematical and Management Sciences 4, 1984, pp. 7–40.MathSciNetzbMATHGoogle Scholar
  31. [31]
    A.H.G. Rinnooy Kan and G.T. Timmer, “Global Optimization”, Handbooks in Operations Research and Management Science, Volume I: Optimization, G.L Nemhauser, A.H.J. Rinnooy Kan, and M.J. Todd, eds., North-Holland, 1989, pp. 631–662.Google Scholar
  32. [32]
    S. Smith, “Adaptive Asynchronous Parallel Algorithms in Distributed Computation,” Ph.D. Thesis, University of Colorado at Boulder, 1991.Google Scholar
  33. [33]
    S. Smith, E. Eskow, and R.B. Schnabel, “Adaptive, asynchronous stochastic global optimization for sequential and parallel computation”, Large Scale Numerical Optimization, T. Coleman and Y. Li, eds., SIAM, Philadelphia, 1990, pp. 207–227.Google Scholar
  34. [34]
    S. Smith and R.B. Schnabel, “Centralized and distributed dynamic scheduling for adaptive parallel algorithms”, Unstructured Scientific Computation on Scalable Multiprocessors, P. Mehrotra, J. Saltz, and R. Voigt, eds., MIT Press, Cambridge, Mass., 1992, pp. 301–321.Google Scholar
  35. [35]
    S. Smith and R.B. Schnabel, “Dynamic scheduling strategies for an adaptive asynchronous parallel global optimization algorithm,” University of Colorado Technical Report CU-CS-625–92, 1992.Google Scholar
  36. [36]
    G.L. Steele, Jr., “High performance Fortran: status report”, in Proceedings of Workshop on Languages, Compilers, and Run-Time Environments for Distributed Memory Multiprocessors, SIGPLAN Notices, Jan. 1993, pp. 1–4.Google Scholar
  37. [37]
    H.W.J.M. Trienekens, “Parallel Branch and Bound Algorithms,” Ph.D. Thesis, Erasmus University, Rotterdam, The Netherlands, 1990.Google Scholar
  38. [38]
    P.J. van Laarhoven, “Parallel variable metric methods for unconstrained optimization,” Mathematical Programming 33, 1985, pp. 68–81.MathSciNetzbMATHCrossRefGoogle Scholar
  39. [39]
    X. Zhang, R.H. Byrd, and R.B. Schnabel, “Parallel methods for solving nonlinear block bordered systems of equations,” SIAM Journal on Scientific and Statistical Computing, 13, 1992, pp. 841–859.MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Kluwer Academic Publishers 1994

Authors and Affiliations

  • Robert B. Schnabel
    • 1
  1. 1.Department of Computer ScienceUniversity of ColoradoBoulderUSA

Personalised recommendations