Penalty variable sample size method for solving optimization problems with equality constraints in a form of mathematical expectation

  • Nataša Krklec Jerinkić
  • Andrea RožnjikEmail author
Original Paper


Equality-constrained optimization problems with deterministic objective function and constraints in the form of mathematical expectation are considered. The constraints are approximated by employing the sample average where the sample size varies throughout the iterations in an adaptive manner. The proposed method incorporates variable sample size scheme with cumulative and unbounded sample into the well- known quadratic penalty iterative procedure. Line search is used for globalization and the sample size is updated in a such way to preserve the balance between two types of errors—errors coming from the sample average approximation and the approximation of the optimal point. Moreover, the penalty parameter is also updated in an adaptive way. We prove that the proposed algorithm pushes the sample size and the penalty parameter to infinity which further allows us to prove the almost sure convergence towards a Karush-Kuhn-Tucker optimal point of the original problem under the rather standard assumptions. Numerical comparison on a set of relevant problems shows the advantage of the proposed adaptive scheme over the heuristic (predetermined) sample scheduling in terms of number of function evaluations as a measure of the optimization cost.


Stochastic optimization Equality constraints Sample average approximation Variable sample size Quadratic penalty method Line search 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



We are grateful to the anonymous referee whose comments and suggestions helped us to improve the quality of this paper.


Nataša Krklec Jerinkić is supported by the Serbian Ministry of Education, Science and Technological Development, grant no. 174030.


  1. 1.
    Bastin, F.: Trust-region algorithms for nonlinear stochastic programming and mixed logit models, PhD thesis, University of Namur, Belgium (2004)Google Scholar
  2. 2.
    Bastin, F., Cirillo, C., Toint, P.L.: An adaptive Monte Carlo algorithm for computing mixed logit estimators. Comput. Manag. Sci. 3(1), 55–79 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Bastin, F., Cirillo, C., Toint, P.L.: Convergence theory for nonconvex stochastic programming with an application to mixed logit. Math. Program. 108 (2–3), 207–234 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Birge, J.R., Louveaux, F.: Introduction to stochastic programming. Springer Series in Operations Research and Financial Engineering. Springer Science+Business Media, LLC, New York (2011)Google Scholar
  5. 5.
    Deng, G., Ferris, M.C.: Variable-number sample path optimization. Math. Program. 117(1–2), 81–109 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Friedlander, M.P., Schmidt, M.: Hybrid deterministic-stochastic methods for data fitting. SIAM J. Sci. Comput. 34(3), 1380–1405 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Hock, W., Schittkowski, K.: Test examples for nonlinear programming codes, Lecture Notes in Economics and Mathematical Systems 187. Springer, New York (1981)CrossRefzbMATHGoogle Scholar
  8. 8.
    Homem-de-Mello, T.: Variable-sample methods for stochastic optimization. ACM Trans. Model. Comput. Simul. 13(2), 108–133 (2003)CrossRefzbMATHGoogle Scholar
  9. 9.
    Homem-de-Mello, T.: On rates of convergence for stochastic optimization problems under nonindependent and identically distributed sampling. SIAM J. Optim. 19(2), 524–551 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Kleywegt, A., Shapiro, A., Homem-de-Mello, T.: The sample average approximation method for stochastic discrete optimization. SIAM J. Optim. 12(2), 479–502 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Krejić, N., Krklec, N.: Line search methods with variable sample size for unconstrained optimization. J. Comput. Appl. Math. 245, 213–231 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Krejić, N., Krklec Jerinkić, N.: Nonmonotone line search methods with variable sample size. Numer. Algorithms 68(4), 711–739 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Krejić, N., Krklec Jerinkić, N.: Spectral projected gradient method for stochastic optimization. J. Global Optim. 73(1), 59–81 (2019)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Krejić, N., Krklec Jerinkić, N., Rožnjik, A.: Variable sample size method for equality constrained optimization problems. Optim. Lett. 12(3), 485–497 (2018)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Linderoth, J., Shapiro, A., Wright, S.: The empirical behavior of sampling methods for stochastic programming. Ann. Oper. Res. 142, 215–241 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Nocedal, J., Wright, S.J.: Numerical Optimization, Springer Series in Operations Research. Springer, New York (2006)Google Scholar
  17. 17.
    Pasupathy, R.: On choosing parameters in retrospective-approximation algorithms for stochastic root finding and simulation optimization. Oper. Res. 58(4), 889–901Google Scholar
  18. 18.
    Polak, E., Royset, J.O.: Eficient sample sizes in stochastic nonlinear programming. J. Comput. Appl. Math. 217(2), 301–310 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Royset, J.O., Szechtman, R.: Optimal budget allocation for sample average approximation. Oper. Res. 61(3), 777–790 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Santoso, T., Ahmed, S., Goetschalckx, M., Shapiro, A.: A stochastic programming approach for supply chain network design under uncertainty. Eur. J. Oper. Res. 167, 96–115 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Shapiro, A.: Monte Carlo sampling methods. In: Stochastic Programming, Handbook in Operations Research and Management Science 10, pp 353–425. Elsevier (2003)Google Scholar
  22. 22.
    Shapiro, A., Dentcheva, D., Ruszczynski, A.: Lectures on stochastic programming: modeling and theory. MPS-SIAM Series on Optimization (2009)Google Scholar
  23. 23.
    Wallace, S.W., Ziemba, W.T. (eds.): Applications of stochastic programming. SIAM, Philadelphia (2005)zbMATHGoogle Scholar
  24. 24.
    Wang, W., Ahmed, S.: Sample average approximation of expected value constrained stochastic programs. Oper. Res. Lett. 36(5), 515–519 (2008)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Mathematics and InformaticsUniversity of Novi SadNovi SadSerbia
  2. 2.Faculty of Civil EngineeringUniversity of Novi SadSuboticaSerbia

Personalised recommendations