Optimization Letters

, Volume 7, Issue 3, pp 421–433 | Cite as

Efficient use of parallelism in algorithmic parameter optimization applications

Original Paper


In the context of algorithmic parameter optimization, there is much room for efficient usage of computational resources. We consider the Opal framework in which a nonsmooth optimization problem models the parameter identification task, and is solved by a mesh adaptive direct search solver. Each evaluation of trial parameters requires the processing of a potentially large number of independent tasks. We describe and evaluate several strategies for using parallelism in this setting. Our test scenario consists in optimizing five parameters of a trust-region method for smooth unconstrained minimization.


Algorithmic parameter optimization Direct search Parallelism 


  1. 1.
    Abramson, M., Audet, C., Couture, G., Dennis, J. Jr., Le Digabel, S.: The NOMAD project. Software available at http://www.gerad.ca/nomad
  2. 2.
    Abramson M., Audet C., Dennis J. Jr., Le Digabel S.: OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM. J. Optim. 20(2), 948–966 (2009). doi: 10.1137/080716980 MathSciNetMATHCrossRefGoogle Scholar
  3. 3.
    Audet C., Béchard V., Le Digabel S.: Nonsmooth optimization through mesh adaptive direct search and variable neighborhood search. J. Glob. Optim. 41(2), 299–318 (2008). doi:10.1007/s10898-007-9234-1 MATHCrossRefGoogle Scholar
  4. 4.
    Audet, C., Dang, C.K., Orban, D.: Algorithmic parameter optimization of the DFO method with the OPAL framework. In: Naono, K., Teranishi, K, Cavazos, J, Suda, R. (eds.) Software Automatic Tuning: From Concepts to State-of-the-Art Results, chap. 15, pp. 255–274. Springer, New york (2010)Google Scholar
  5. 5.
    Audet C., Dennis J. Jr: Analysis of generalized pattern searches. SIAM J. Optim 13(3), 889–903 (2003). doi:10.1137/S1052623400378742 MathSciNetMATHCrossRefGoogle Scholar
  6. 6.
    Audet C., Dennis J. Jr: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2006)MathSciNetMATHCrossRefGoogle Scholar
  7. 7.
    Audet C., Dennis J. Jr: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20(4), 445–472 (2009). doi:10.1137/070692662 MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    Audet C., Dennis J. Jr, Le Digabel S.: Parallel space decomposition of the mesh adaptive direct search algorithm. SIAM J. Optim. 19(3), 1150–1170 (2008). doi:10.1137/070707518 MathSciNetMATHCrossRefGoogle Scholar
  9. 9.
    Audet C., Dennis J. Jr, Le Digabel S.: Globalization strategies for mesh adaptive direct search. Comput. Optim. Appl. 46(2), 193–215 (2010). doi:10.1007/s10589-009-9266-1 MathSciNetMATHCrossRefGoogle Scholar
  10. 10.
    Audet C., Orban D.: Finding optimal algorithmic parameters using derivative-free optimization. SIAM J. Optim. 17(3), 642–664 (2006). doi:10.1137/040620886 MathSciNetMATHCrossRefGoogle Scholar
  11. 11.
    Choi T., Kelley C.: Superlinear convergence and implicit filtering. SIAM J. Optim. 10(4), 1149–1162 (2000)MathSciNetMATHCrossRefGoogle Scholar
  12. 12.
    Clarke, F.: Optimization and Nonsmooth Analysis. Wiley, New York (1983) (reissued in 1990 by SIAM Publications, Philadelphia, as Vol. 5 in the series Classics in Applied Mathematics)Google Scholar
  13. 13.
    Conn A., Gould N., Toint P.: Trust-Region Methods. MPS-SIAM Series on Optimization. SIAM, Philadelphia (2000)CrossRefGoogle Scholar
  14. 14.
    Conn, A., Scheinberg, K., Vicente, L.: Introduction to Derivative-Free Optimization. MOS/SIAM Series on Optimization. SIAM, Philadelphia (2009)Google Scholar
  15. 15.
    Gilmore, P., Choi, T., Eslinger, O., Kelley, C., Patrick, H., Gablonsky, J.: IFFCO (implicit filtering for constrained optimization). Software available at: http://www4.ncsu.edu/~ctk/iffco.html
  16. 16.
    Gould N., Lucidi S., Toint P.: Solving the trust-region subproblem using the Lanczos method. SIAM J. Optim. 9(2), 504–525 (1999)MathSciNetMATHCrossRefGoogle Scholar
  17. 17.
    Gould N., Orban D., Toint P.: CUTEr (and SifDec): a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. 29(4), 373–394 (2003). doi:10.1145/962437.962439 MathSciNetMATHCrossRefGoogle Scholar
  18. 18.
    Gould N., Orban D., Toint P.: GALAHAD, a library of thread-safe Fortran 90 packages for large-scale nonlinear optimization. ACM Trans. Math. Softw. 29(4), 353–372 (2003). doi:10.1145/962437.962438 MathSciNetMATHCrossRefGoogle Scholar
  19. 19.
    Gray G., Kolda T.: Algorithm 856: APPSPACK 4.0: Asynchronous parallel pattern search for derivative-free optimization. ACM Trans. Math. Softw. 32(3), 485–507 (2006). doi:10.1145/1163641.1163647 MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Griffin J., Kolda T., Lewis R.: Asynchronous parallel generating set search for linearly-constrained optimization. SIAM J. Sci. Comp. 30(4), 1892–1924 (2008). doi:10.1137/060664161 MathSciNetCrossRefGoogle Scholar
  21. 21.
    Gropp W., Lusk E., Skjellum A.: Using MPI: portable parallel programming with the message- passing interface, 2nd edn. Scientific and Engineering Computation Series. MIT Press, Cambridge (1994)Google Scholar
  22. 22.
    Le Digabel, S.: NOMAD user guide. Tech. Rep. G-2009-37, Les cahiers du GERAD (2009)Google Scholar
  23. 23.
    Le Digabel S.: Algorithm 909: NOMAD: Nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37(4), 44:1–44:15 (2011). doi:10.1145/1916461.1916468 MathSciNetCrossRefGoogle Scholar
  24. 24.
    Pardalos P.: Parallel Processing of Discrete Problems, IMA Volumes in Mathematics and its Applications, vol. 106. Springer, Berlin (1999)CrossRefGoogle Scholar
  25. 25.
    Shylo O., Middelkoop T., Pardalos P.: Restart strategies in optimization: parallel and serial cases. Parallel Comput. 37(1), 60–68 (2011)MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  1. 1.GERAD and Département de Mathématiques et de Génie IndustrielÉcole polytechnique de MontréalMontrealCanada

Personalised recommendations