How difficult is nonlinear optimization? A practical solver tuning approach, with illustrative results

Abstract

Nonlinear optimization (NLO) encompasses a vast range of problems, from very simple to theoretically intractable instances. For this reason, it is impossible to offer guaranteed—while practically meaningful—advice to users of NLO software. This issue becomes apparent, when facing exceptionally hard and/or previously unexplored NLO challenges. We propose a heuristic quadratic meta-model based approach, and suggest corresponding key option settings to use with the Lipschitz global optimizer (LGO) solver suite. These LGO option settings are directly related to estimating the sufficient computational effort to handle a broad range of NLO problems. The proposed option settings are evaluated experimentally, by solving (numerically) a representative set of NLO test problems which are based on real-world optimization applications and non-trivial academic challenges. Our tests include also a set of scalable optimization problems which are increasingly difficult to handle as the size of the model-instances increases. Based on our computational results, it is possible to offer generally valid, practical advice to LGO users. Arguably (and mutatis mutandis), comparable advice can be given to users of other NLO software products with a similarly broad mandate to LGO’s. An additional benefit of such aggregated tests is that their results can effectively assist the rapid evaluation and verification of NLO solver performance during software development phases.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

References

  1. Ali, M. M., Khompatraporn, C., & Zabinsky, Z. B. (2005). A numerical evaluation of several stochastic algorithms on selected continuous global optimization test problems. Journal of Global Optimization, 31, 635–672.

    Article  Google Scholar 

  2. Antoniou, A., & Lu, W.-S. (2007). Practical optimization–Algorithms and engineering applications. New York: Springer.

    Google Scholar 

  3. Arora, S., & Barak, B. (2017). Computational complexity: A modern approach. New York: Cambridge University Press.

    Google Scholar 

  4. Averick, B. M., & Moré, J. J. (1991). The minpack-2 test problem collection. Technical report ANL/MCS-TM-157. Argonne, IL: Argonne National Laboratory.

  5. Bartholomew-Biggs, M. (2010). Nonlinear optimization with engineering applications. New York: Springer.

    Google Scholar 

  6. Boyd, S. P., & Vandenberghe, L. (2009). Convex optimization (7th ed.). New York: Cambridge University Press.

    Google Scholar 

  7. Brent, R. P. (1973). Algorithms for minimization without derivatives. Englewood Cliffs, NJ: Prentice-Hall.

    Google Scholar 

  8. Çaglayan, M. O., & Pintér, J. D. (2013). Development and calibration of currency market strategies by global optimization. Journal of Global Optimization, 56, 353–371.

    Article  Google Scholar 

  9. Castillo, I., Kampas, F. J., & Pintér, J. D. (2008). Solving circle packing problems by global optimization: Numerical results and industrial applications. European Journal of Operational Research, 191, 786–802.

    Article  Google Scholar 

  10. de Klerk, E., Roos, C., & Terlaky, T. (2001). Nonlinear optimization. Lecture notes: Delft University of Technology; and University of Waterloo.

    Google Scholar 

  11. Deschaine, L. M., Lillys, T. P., & Pintér, J. D. (2013). Groundwater remediation design using physics-based flow, transport, and optimization technologies. Environmental Systems Research 2, 2:6. Available for download at http://www.environmentalsystemsresearch.com/content/2/1/6.

  12. Dolan, E. D., Moré, J. J., & Munson, T. S. (2004). Benchmarking optimization software with COPS 3. Technical report ANL/MCS-TM-273, Mathematics and Computer Science Division. Argonne, IL: Argonne National Laboratory.

  13. Edgar, T. F., Himmelblau, D. M., & Lasdon, L. S. (2001). Optimization of chemical processes (2nd ed.). New York: McGraw-Hill.

    Google Scholar 

  14. Fasano, G. (2014). Solving non-standard packing problems by global optimization and heuristics. New York: Springer.

    Book  Google Scholar 

  15. Fasano, G., & Pintér, J. D. (Eds.). (2013a). Modeling and optimization in space engineering. New York: Springer.

    Google Scholar 

  16. Fasano, G., & Pintér, J. D. (2013b). Global optimization approaches to sensor placement: Model versions and illustrative results. In G. Fasano & J. D. Pintér (Eds.), Modeling and optimization in space engineering (pp. 235–247). New York: Springer.

    Chapter  Google Scholar 

  17. Fasano, G., & Pintér, J. D. (Eds.). (2015). Optimized packings and their applications. New York: Springer.

    Google Scholar 

  18. Fasano, G., & Pintér, J. D. (Eds.). (2016). Space engineering: Modeling and optimization with case studies. New York: Springer.

    Google Scholar 

  19. Finley, J. R., Pintér, J. D., & Satish, M. G. (1998). Automatic model calibration applying global optimization techniques. Environmental Modeling and Assessment, 3(1), 117–126.

    Article  Google Scholar 

  20. Floudas, C. A., & Pardalos, P. M. (1990). A collection of test problems for constrained global optimization algorithms. Lecture notes in computer science (Vol. 455). New York: Springer.

  21. Floudas, C. A., Pardalos, P. M., Adjiman, C. S., Esposito, W. R., Gümüş, Z. H., Harding, S. T., et al. (1999). Handbook of test problems in local and global optimization. Dordrecht: Kluwer.

    Book  Google Scholar 

  22. Forst, W., & Hoffmann, D. (2010). Optimization-Theory and practice. Seelisberg: Springer.

    Book  Google Scholar 

  23. Gammon, M. A., Pintér, J. D., & Schwartz, B. (2007). Enhancement of optimization capability in TacTool using the Lipschitz global optimizer (LGO) program. DRDC Atlantic TM 2006-167. Dartmouth, NS.

  24. Goossens, P., McPhee, J., Pintér, J. D., & Schmitke, C. (2007). Driving innovation: How mathematical modeling and optimization increase efficiency and productivity in vehicle design. Maplesoft, Waterloo, ON: Technical Memorandum.

    Google Scholar 

  25. Grossmann, I. E. (Ed.). (1996). Global optimization in engineering design. Dordrecht: Kluwer Academic Publishers.

    Google Scholar 

  26. Heineman, G. T., Pollice, G., & Selkow, S. (2016). Algorithms in a nutshell (2nd ed.). Sebastopol, CA: O’Reilly Media.

    Google Scholar 

  27. Hillier, F., & Lieberman, G. J. (2010). Introduction to operations research (9th ed.). New York: McGraw-Hill.

    Google Scholar 

  28. Hock, W., & Schittkowski, K. (1981). Test examples for nonlinear programming codes. Lecture notes in economics and mathematical systems (Vol. 187). Heidelberg: Springer.

  29. Horst, R., & Pardalos, P. M. (Eds.). (1995). Handbook of global optimization (Vol. 1). Dordrecht: Kluwer Academic Publishers.

    Google Scholar 

  30. Isenor, G., Pintér, J. D., & Cada, M. (2003). A global optimization approach to laser design. Optimization and Engineering, 4, 177–196.

    Article  Google Scholar 

  31. Kampas, F. J., Castillo, I., & Pintér, J. D. (2016). Optimized ellipse packings in regular polygons using embedded Lagrange multipliers (Submitted for publication).

  32. Kampas, F. J., & Pintér, J. D. (2006). Configuration analysis and design by using optimization tools in Mathematica. The Mathematica Journal, 10(1), 128–154.

    Google Scholar 

  33. Khompatraporn, Ch., Pintér, J. D., & Zabinsky, Z. B. (2005). Comparative assessment of algorithms and software for global optimization. Journal of Global Optimization, 31, 613–633.

    Article  Google Scholar 

  34. Kleijnen, J. P. C. (2015). Design and analysis of simulation experiments (2nd ed.). Basel: Springer.

    Book  Google Scholar 

  35. Lahey Computer Systems. (2015). Lahey/Fujitsu Fortran 95 (Release 7.5). Incline Village, NV.

  36. Locatelli, M., & Schoen, F. (2013). Global optimization: Theory, algorithms, and applications. MOS-SIAM series on optimization. Philadelphia, PA: Society for Industrial and Applied Mathematics.

  37. Mason, T. L., Emelle, C., van Berkel, J., Bagirov, A. M., Kampas, F. J., & Pintér, J. D. (2007). Integrated production system optimization using the Lipschitz Global Optimizer and the Discrete Gradient Method. Journal of Industrial and Management Optimization, 3(2), 257–277.

    Article  Google Scholar 

  38. Moré, J. J., Garbow, B. S., & Hillström, K. E. (1981). Testing unconstrained optimization software. ACM Transaction on Mathematical Software, 7, 17–41.

    Article  Google Scholar 

  39. Moré, J. J., & Wild, S. M. (2009). Benchmarking derivative-free optimization algorithms. SIAM Journal on Optimization, 20(1), 172–191.

    Article  Google Scholar 

  40. Nocedal, J., & Wright, S. J. (2006). Numerical optimization (2nd ed.). New York: Springer.

    Google Scholar 

  41. Neumaier, A., Shcherbina, O., Huyer, W., & Vinkó, T. (2005). A comparison of complete global optimization solvers. Mathematical Programming, Series B, 103, 335–356.

    Article  Google Scholar 

  42. Pantoleontos, G., Basinas, P., Skodras, G., Grammelis, P., Pintér, J. D., Topis, S., et al. (2009). A global optimization study on the devolatilisation kinetics of coal, biomass and waste fuels. Fuel Processing Technology, 90, 762–769.

    Article  Google Scholar 

  43. Pardalos, P. M., & Romeijn, H. E. (Eds.). (2002). Handbook of global optimization (Vol. 2). Dordrecht: Kluwer Academic Publishers.

    Google Scholar 

  44. Pintér, J. D. (1990). Globally optimized calibration of environmental models. Annals of Operations Research, 25, 211–222.

    Article  Google Scholar 

  45. Pintér, J. D. (1991). Stochastic modelling and optimization for environmental management. Annals of Operations Research, 31(1991), 527–544.

    Article  Google Scholar 

  46. Pintér, J. D. (1996). Global optimization in action. Dordrecht: Kluwer Academic Publishers.

    Book  Google Scholar 

  47. Pintér, J. D. (1997). LGO–A program system for continuous and Lipschitz optimization. In I. M. Bomze, T. Csendes, R. Horst, & P. M. Pardalos (Eds.), Developments in global optimization (pp. 183–197). Dordrecht: Kluwer Academic Publishers.

    Chapter  Google Scholar 

  48. Pintér, J. D. (2001). Globally optimized spherical point arrangements: Model variants and illustrative results. Annals of Operations Research, 104, 213–230.

    Article  Google Scholar 

  49. Pintér, J. D. (2002). Global optimization: Software, test problems, and applications. In P. M. Pardalos & H. E. Romeijn (Eds.), Handbook of Global Optimization (Vol. 2, pp. 515–569). Dordrecht: Kluwer Academic Publishers.

    Chapter  Google Scholar 

  50. Pintér, J. D. (2005). Nonlinear optimization in modeling environments: Software implementations for compilers, spreadsheets, modeling languages, and integrated computing systems. In V. Jeyakumar & A. M. Rubinov (Eds.), Continuous Optimization: Current Trends and Modern Applications (pp. 147–173). New York: Springer.

    Chapter  Google Scholar 

  51. Pintér, J. D. (Ed.). (2006). Global optimization: Scientific and engineering case studies. New York: Springer.

    Google Scholar 

  52. Pintér, J. D. (2009). Software development for global optimization. In P. M. Pardalos & T. F. Coleman (Eds.), Global optimization: Methods and applications. Fields institute communications (Vol. 55, pp. 183–204). Providence, RI: American Mathematical Society.

  53. Pintér, J. D. (2012). Calibrating artificial neural networks by global optimization. Expert Systems with Applications, 39, 25–32.

    Article  Google Scholar 

  54. Pintér, J. D. (2016). LGO—A model development and solver system for global-local nonlinear optimization. User’s guide. Distributed by PCS Inc., Canada, www.pinterconsulting.com.

  55. Pintér, J. D., Fels, M., Lycon, D. S., Meeuwig, D. J., & Meeuwig, J. W. (1995). An intelligent decision support system for assisting industrial wastewater management. Annals of Operations Research, 58, 455–477.

    Article  Google Scholar 

  56. Pintér, J. D., & Horváth, Z. (2013). Integrated experimental design and nonlinear optimization to handle computationally expensive models under resource constraints. Journal of Global Optimization, 57, 191–215.

    Article  Google Scholar 

  57. Pintér, J. D., & Kampas, F. J. (2006). MathOptimizer Professional: Key features and illustrative applications. In L. Liberti & N. Maculan (Eds.), Global optimization: From theory to implementation (pp. 263–279). New York: Springer.

    Chapter  Google Scholar 

  58. Pintér, J. D., & Kampas, F. J. (2013). Benchmarking nonlinear optimization software in technical computing environments: Global optimization in Mathematica with MathOptimizer Professional. TOP (An official journal of the Spanish Society of Statistics and Operations Research), 21, 133–162.

    Google Scholar 

  59. Pintér, J. D., Castellazzo, A., Vola, M., & Fasano, G. (2016a). Nonlinear regression analysis by global optimization: A case study in space engineering. In G. Fasano & J. D. Pintér (Eds.), Space engineering: Modeling and optimization with case studies (pp. 287–302). New York: Springer.

    Chapter  Google Scholar 

  60. Pintér, J. D., Kampas, F. J., & Castillo, I. (2016b). Globally optimized packings of non-uniform size spheres in \(R^{d}\): A computational study. (Submitted for publication).

  61. Press, W. H., Teukolsky, S. A., Wetterling, W. T., & Flannery, B. (1992). Numerical recipes in FORTRAN: The art of scientific computing. Cambridge: Cambridge University Press.

    Google Scholar 

  62. Rios, L. M., & Sahinidis, N. V. (2013). Derivative-free optimization: A review of algorithms and comparison of software implementations. Journal of Global Optimization, 56, 1247–1293.

    Article  Google Scholar 

  63. Schittkowski, K. (1987). More test examples for nonlinear programming. Lecture notes in economics and mathematical systems (Vol. 182). Berlin: Springer.

  64. Schittkowski, K. (2008). An updated set of 306 test problems for nonlinear programming with validated optimal solutions—User’s guide. Research Report, Department of Computer Science, University of Bayreuth.

  65. Skiena, S. S. (2008). The algorithm design manual (2nd ed.). New York: Springer.

    Book  Google Scholar 

  66. Stortelder, W. J. H., de Swart, J. J. B., & Pintér, J. D. (2001). Finding elliptic Fekete points sets: Two numerical solution approaches. Journal of Computational and Applied Mathematics, 130, 205–216.

    Article  Google Scholar 

  67. Strongin, R. G., & Sergeyev, Y. D. (2000). Global optimization with non-convex constraints: Sequential and parallel algorithms. Dordrecht: Kluwer Academic Publishers.

    Book  Google Scholar 

  68. Tawarmalani, M., & Sahinidis, N. (2002). Convexification and global optimization in continuous and mixed-integer nonlinear programming. Dordrecht: Kluwer Academic Publishers.

    Book  Google Scholar 

  69. Terlaky, T., (Ed). (1996). Interior point methods of mathematical programming. Kluwer Academic Publishers, Dordrecht. (New edition by Springer Science + Business Media, New York, 2013.)

  70. Tervo, J., Kolmonen, P., Lyyra-Laitinen, T., Pintér, J. D., & Lahtinen, T. (2003). An optimization-based approach to the multiple static delivery technique in radiation therapy. Annals of Operations Research, 119, 205–227.

    Article  Google Scholar 

  71. Van der Molen, D. T., & Pintér, J. D. (1993). Environmental model calibration under different problem specifications: An application to the model SED. Ecological Modelling, 68, 1–19.

    Article  Google Scholar 

  72. Weisstein, Eric W. (2016). “Binary.” From MathWorld—A Wolfram Web Resource. http://mathworld.wolfram.com/Binary.html.

  73. Zabinsky, Z. B. (2003). Stochastic adaptive search for global optimization. Dordrecht: Kluwer Academic Publishers.

    Book  Google Scholar 

  74. Zimmermann, A. (2006). Al Zimmermann’s programming contests. Circle Packing. http://recmath.com/ contest/CirclePacking/index.php.

Download references

Acknowledgements

This article is a contribution to the special issue of Annals of Operations Research devoted to my long-time friend (since 2016, also an esteemed colleague) Tamás Terlaky. Tamás is a remarkable individual of many talents: an exceptional researcher with a keen sense of unifying theory and practice; a hard-working and benevolent leader with integrity; and a very likable, generous, positive human being. Our frequent interactions over many years motivated my research, and contributed to forming my views (also) on the essence and practice of nonlinear optimization.

The work summarized here has been strongly motivated by and partially based on research collaboration with numerous colleagues as indicated by the applications cited in Sect. 1. In particular, I wish to acknowledge Ignacio Castillo, Giorgio Fasano and Frank Kampas for long-standing and continuing collaboration related to software development and benchmarking, object packings, and other real-world applications.

I also express my thanks to two anonymous referees of the present article for their constructive and meticulous comments.

The author’s research work has been partially supported by the following organizations over the years (adding that the author is solely responsible for the views expressed here):

AMPL Optimization, USA

Bilkent University, Turkey

Centre for Mathematics and Computer Science, The Netherlands

Dalhousie University, Canada

Defence Research and Development, Canada

Delft University of Technology, The Netherlands

Frontline Systems, USA

GAMS Development Corporation, USA

Hungarian Scientific Research Fund

HydroGeoLogic, USA

Kluwer Academic Publishers, The Netherlands

Lahey Computer Systems, USA

Lehigh University, USA

LINDO Systems, USA

Maplesoft, Canada

Maximal Software, USA

National Research Council, Canada

Özyeğin University, Turkey

Institute for Inland Water Management and Wastewater Treatment, The Netherlands

Paragon Decision Technology, The Netherlands

Saint Mary’s University, Canada

Shell International Exploration and Production, The Netherlands

Springer Science \(+\) Business Media, USA

Széchenyi István University, Hungary

The MathWorks, USA

TOMLAB Optimization, Sweden

University of Ballarat, Australia

University of Kuopio, Finland

Water Resources Research Centre, Hungary

Wolfram Research, USA.

Author information

Affiliations

Authors

Corresponding author

Correspondence to János D. Pintér.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pintér, J.D. How difficult is nonlinear optimization? A practical solver tuning approach, with illustrative results. Ann Oper Res 265, 119–141 (2018). https://doi.org/10.1007/s10479-017-2518-z

Download citation

Keywords

  • Nonlinear (global and local) optimization
  • Scalable heuristic meta-models of NLO problems
  • Software benchmarking
  • LGO solver suite
  • Test model collections and numerical results