Linear equalities in blackbox optimization

  • Charles Audet
  • Sébastien Le Digabel
  • Mathilde PeyregaEmail author


The mesh adaptive direct search (Mads) algorithm is designed for blackbox optimization problems subject to general inequality constraints. Currently, Mads does not support equalities, neither in theory nor in practice. The present work proposes extensions to treat problems with linear equalities whose expression is known. The main idea consists in reformulating the optimization problem into an equivalent problem without equalities and possibly fewer optimization variables. Several such reformulations are proposed, involving orthogonal projections, QR or SVD decompositions, as well as simplex decompositions into basic and nonbasic variables. All of these strategies are studied within a unified convergence analysis, guaranteeing Clarke stationarity under mild conditions provided by a new result on the hypertangent cone. Numerical results on a subset of the CUTEst collection are reported.


Derivative-free optimization Blackbox optimization Linear equality constraints Convergence analysis Mads 



The authors would like to thank two anonymous referees for their careful reading and helpful comments and suggestions. Work of the first author was supported by NSERC Grant 239436. The second author was supported by NSERC Grant 418250. The first and second authors are supported by AFOSR FA9550-12-1-0198.


  1. 1.
    Abramson, M.A., Audet, C., Couture, G., Dennis, Jr. J.E., Le Digabel, S., Tribes, C.: The NOMAD project. Software available at (2014)
  2. 2.
    Abramson, M.A., Audet, C., Dennis Jr, J.E., Le Digabel, S.: OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM J. Optim. 20(2), 948–966 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Abramson, M.A., Brezhneva, O.A., Dennis Jr, J.E., Pingel, R.L.: Pattern search in the presence of degenerate linear constraints. Optim. Methods Softw. 23(3), 297–319 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  4. 4.
    Audet, C.: A survey on direct search methods for blackbox optimization and their applications. In: Pardalos, P.M., Rassias, T.M. (eds.) Mathematics without Boundaries: Surveys in Interdisciplinary Research, Chapter 2, pp. 31–56. Springer, New York (2014)CrossRefGoogle Scholar
  5. 5.
    Audet, C., Dennis Jr, J.E.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Audet, C., Dennis Jr, J.E.: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20(1), 445–472 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  7. 7.
    Audet, C., Ianni, A., Le Digabel, S., Tribes, C.: Reducing the number of function evaluations in mesh adaptive direct search algorithms. SIAM J. Optim. 24(2), 621–642 (2014)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Bueno, L.F., Friedlander, A., Martínez, J.M., Sobral, F.N.C.: Inexact restoration method for derivative-free optimization with smooth constraints. SIAM J. Optim. 23(2), 1189–1213 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Çivril, A., Magdon-Ismail, M.: On selecting a maximum volume sub-matrix of a matrix and related problems. Theor. Comput. Sci. 410(47–49), 4801–4811 (2009)CrossRefzbMATHGoogle Scholar
  10. 10.
    Clarke, F.H.: Optimization and Nonsmooth Analysis. Wiley, New York (1983). Reissued in 1990 by SIAM Publications, Philadelphia, as, Vol. 5 in the series Classics in Applied Mathematics.Google Scholar
  11. 11.
    Conn, A.R., Gould, N.I.M., Toint, PhL: A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds. SIAM J. Numer. Anal. 28(2), 545–572 (1991)CrossRefzbMATHMathSciNetGoogle Scholar
  12. 12.
    Conn, A.R., Scheinberg, K., Toint, P.L.: A derivative free optimization algorithm in practice. In: Proceedings the of 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, St. Louis (1998)Google Scholar
  13. 13.
    Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MOS/SIAM Series on Optimization. SIAM, Philadelphia (2009)Google Scholar
  14. 14.
    Coope, I.D., Price, C.J.: On the convergence of grid-based methods for unconstrained optimization. SIAM J. Optim. 11(4), 859–869 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Dreisigmeyer, D.W.: Equality constraints, Riemannian manifolds and direct search methods. Technical Report LA-UR-06-7406, Los Alamos National Laboratory, Los Alamos (2006)Google Scholar
  16. 16.
    Gould, N.I.M., Orban, D., Toint, P.L.: CUTEst: a constrained and unconstrained testing environment with safe threads for mathematical optimization. Computational Optimization and Applications. Code available at (2014)
  17. 17.
    Gould, N.I.M., Toint, PhL: Nonlinear programming without a penalty function or a filter. Math. Program. 122(1), 155–196 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 18.
    Griffin, J.D., Kolda, T.G., Lewis, R.M.: Asynchronous parallel generating set search for linearly-constrained optimization. SIAM J. Sci. Comput. 30(4), 1892–1924 (2008)CrossRefMathSciNetGoogle Scholar
  19. 19.
    Hock, W., Schittkowski, K.: Test Examples for Nonlinear Programming Codes. Lecture Notes in Economics and Mathematical Systems, vol. 187. Springer, Berlin (1981)CrossRefGoogle Scholar
  20. 20.
    Ivanov, B.B., Galushko, A.A., Stateva, R.P.: Phase stability analysis with equations of state—a fresh look from a different perspective. Ind. Eng. Chem. Res. 52(32), 11208–11223 (2013)CrossRefGoogle Scholar
  21. 21.
    Jahn, J.: Introduction to the Theory of Nonlinear Optimization. Springer, Berlin (1994)CrossRefzbMATHGoogle Scholar
  22. 22.
    Kolda, T.G., Lewis, R.M., Torczon, V.: A generating set direct search augmented Lagrangian algorithm for optimization with a combination of general and linear constraints. Technical Report SAND2006-5315, Sandia National Laboratories (2006)Google Scholar
  23. 23.
    Kolda, T.G., Lewis, R.M., Torczon, V.: Stationarity results for generating set search for linearly constrained optimization. SIAM J. Optim. 17(4), 943–968 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Lawson, C.L., Hanson, R.J.: Solving Least Squares Problems. Prentice-Hall, Englewood Cliffs (1974)zbMATHGoogle Scholar
  25. 25.
    Le Digabel, S.: Algorithm 909: NOMAD: Nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37(4), 44:1–44:15 (2011)CrossRefGoogle Scholar
  26. 26.
    Lewis, R.M., Shepherd, A., Torczon, V.: Implementing generating set search methods for linearly constrained minimization. SIAM J. Sci. Comput. 29(6), 2507–2530 (2007)CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Lewis, R.M., Torczon, V.: Pattern search methods for linearly constrained minimization. SIAM J. Optim. 10(3), 917–941 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  28. 28.
    Lewis, R.M., Torczon, V.: Active set identification for linearly constrained minimization without explicit derivatives. SIAM J. Optim. 20(3), 1378–1405 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  29. 29.
    Lewis, R.M., Torczon, V.: A direct search approach to nonlinear programming problems using an augmented lagrangian method with explicit treatment of linear constraints. Technical report, College of William & Mary (2010)Google Scholar
  30. 30.
    Liu, L., Zhang, X.: Generalized pattern search methods for linearly equality constrained optimization problems. Appl. Math. Comput. 181(1), 527–535 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  31. 31.
    Martínez, J.M., Sobral, F.N.C.: Constrained derivative-free optimization on thin domains. J. Glob. Optim. 56(3), 1217–1232 (2013)CrossRefzbMATHGoogle Scholar
  32. 32.
    May, J.H.: Linearly constrained nonlinear programming: a solution method that does not require analytic derivatives. Ph.D thesis, Yale University (1974)Google Scholar
  33. 33.
    Mifflin, R.: A superlinearly convergent algorithm for minimization without evaluating derivatives. Math. Program. 9(1), 100–117 (1975)CrossRefzbMATHMathSciNetGoogle Scholar
  34. 34.
    Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  35. 35.
    Nocedal, J., Wright, S.J.: Numerical Optimization Springer Series in Operations Research. Springer, New York (1999)Google Scholar
  36. 36.
    Plantenga, T.D.: HOPSPACK 2.0 user manual. Technical Report SAND2009-6265, Sandia National Laboratories, Livermore (2009)Google Scholar
  37. 37.
    Powell, M.J.D.: Lincoa software. Software available at
  38. 38.
    Powell, M.J.D.: On fast trust region methods for quadratic models with linear constraints. Technical Report DAMTP 2014/NA02, Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Silver Street, Cambridge CB3 9EW (2014)Google Scholar
  39. 39.
    Price, C.J., Coope, I.D.: Frames and grids in unconstrained and linearly constrained optimization: a nonsmooth approach. SIAM J. Optim. 14(2), 415–438 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  40. 40.
    Rudin, W.: Functional Analysis International Series in Pure and Applied Mathematics, 2nd edn. McGraw-Hill Inc., New York (1991)Google Scholar
  41. 41.
    Sampaio, P.R., Toint, P.L.: A derivative-free trust-funnel method for equality-constrained nonlinear optimization. Technical report, NAXYS-Namur Center for Complex Systems, Belgium(2014)Google Scholar
  42. 42.
    Selvan, S.E., Borckmans, P.B., Chattopadhyay, A., Absil, P.-A.: Spherical mesh adaptive direct search for separating quasi-uncorrelated sources by range-based independent component analysis. Neural Comput. 25(9), 2486–2522 (2013)CrossRefMathSciNetGoogle Scholar
  43. 43.
    Torczon, V.: On the convergence of pattern search algorithms. SIAM J. Optim. 7(1), 1–25 (1997)CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Charles Audet
    • 1
  • Sébastien Le Digabel
    • 1
  • Mathilde Peyrega
    • 1
    Email author
  1. 1.GERAD and Département de mathématiques et génie industrielÉcole Polytechnique de MontréalMontrealCanada

Personalised recommendations