Quasi-Newton smoothed functional algorithms for unconstrained and constrained simulation optimization

Abstract

We propose a multi-time scale quasi-Newton based smoothed functional (QN-SF) algorithm for stochastic optimization both with and without inequality constraints. The algorithm combines the smoothed functional (SF) scheme for estimating the gradient with the quasi-Newton method to solve the optimization problem. Newton algorithms typically update the Hessian at each instant and subsequently (a) project them to the space of positive definite and symmetric matrices, and (b) invert the projected Hessian. The latter operation is computationally expensive. In order to save computational effort, we propose in this paper a quasi-Newton SF (QN-SF) algorithm based on the Broyden-Fletcher-Goldfarb-Shanno (BFGS) update rule. In Bhatnagar (ACM TModel Comput S. 18(1): 27–62, 2007), a Jacobi variant of Newton SF (JN-SF) was proposed and implemented to save computational effort. We compare our QN-SF algorithm with gradient SF (G-SF) and JN-SF algorithms on two different problems – first on a simple stochastic function minimization problem and the other on a problem of optimal routing in a queueing network. We observe from the experiments that the QN-SF algorithm performs significantly better than both G-SF and JN-SF algorithms on both the problem settings. Next we extend the QN-SF algorithm to the case of constrained optimization. In this case too, the QN-SF algorithm performs much better than the JN-SF algorithm. Finally we present the proof of convergence for the QN-SF algorithm in both unconstrained and constrained settings.

Appendix: Proofs of Sect. 4

Proof of Lemma 1

Note that $Q_l(n)$ is measurable $\mathcal {F}(n)$ for all $n\ge 0$. Further, it is easy to see that $E[Q_l(n+1)|\mathcal {F}(n)] = Q_l(n)$ a.s., $\forall n \ge 0$. We can show that for any real $a_n$ and $b_n$,

$$\begin{aligned} \bigg ( \sum _{n=1}^K (a_n - b_n) \bigg )^2&= \bigg ( \sum _{n=1}^K \frac{a_n-b_n}{\sqrt{a_n^2+b_n^2}} \sqrt{a_n^2+b_n^2} \bigg )^2\\&\le \sum _{n=1}^K \bigg (1-\frac{2 a_n b_n}{a_n^2 + b_n^2} \bigg ) \sum _{n=1}^K (a_n^2 + b_n^2) \le 2K \sum _{n=1}^K(a_n^2+ b_n^2), \end{aligned}$$

where first we have used the Cauchy-Schwarz inequality and the second inequality follows since $-\frac{2a_nb_n}{a_n^2+b_n^2} \le 1$. Hence we have

$$\begin{aligned} E[Q_l^2(n)]&\le \frac{2n}{\beta ^2} \sum _{m=1}^n b^2(m) E \bigg ( \eta _{l}^2(m) \Big ( h(X^{'}_{m}) - h(X_m) \Big )^2 \\&\quad + E^2 \Big ( \eta _l(m) \big ( h(X^{'}_{m}) - h(X_m) \big ) \big | \mathcal {F}(m-1) \Big ) \bigg ). \end{aligned}$$

Here, we let $E^a(X)$ denote $(E(X))^a$. By conditional Jensen’s inequality, we have

$$\begin{aligned}&E^2 \Big ( \eta _l(m) \big ( h(X^{'}_{m}) - h(X_m) \big ) \big | \mathcal {F}(m-1) \Big ) \\&\quad \le E \Big ( \eta _l^2(m) \big ( h(X^{'}_{m}) - h(X_m) \big )^2 \big | \mathcal {F}(m-1) \Big ). \end{aligned}$$

Hence by the Cauchy-Schwarz inequality

$$\begin{aligned} E[Q_l^2(n)]&\le \frac{4n}{\beta ^2} \sum _{m=1}^n b^2(m) E \bigg ( \eta _{l}^2(m) \Big ( h(X^{'}_{m}) - h(X_m) \Big )^2 \bigg ) \\&\le \frac{4n}{\beta ^2} \sum _{m=1}^n b^2(m) E^{1/2} \Big ( \eta _{l}^4(m) \Big ) E^{1/2} \Big ( \big ( h(X^{'}_{m}) - h(X_m) \big )^4 \Big ). \end{aligned}$$

Since h(.) is a Lipschitz continuous function, we have $|h(X^{'}_m) - h(X_m)|^4 \le K ||X^{'}_m - X_m ||^4$, for some constant $K > 0$. Hence, $E^{1/2}[(h(X^{'}_m) - h(X_m))^4] \le \sqrt{K} E^{1/2} || X^{'}_m - X_m ||^4$. As a consequence of Assumption 3, $\sup _m E[ || X_m - X^{'}_m ||^4] < \infty$ [4]. Thus, $E[Q^2_{l}(n)] < \infty$, for all $n \ge 1$, i.e., $Q_l(n)$ are square-integrable and hence also integrable random variables. Thus $(Q_l(n),\mathcal {F}(n)), n\ge 0$ is a square-integrable martingale sequence. We now show that its quadratic variation process is convergent. Thus, note that

$$\begin{aligned}&\sum _n E[(Q_l(n+1) - Q_l(n) )^2 | \mathcal {F}(n)] \\&= \sum _n E \Big [ \Big ( b(n+1) \Big ( \frac{\eta _l(n+1)}{\beta } \Big ( h(X^{'}_{n+1}) - h(X_{n+1}) \Big ) \\&\quad -E \Big ( \frac{\eta _l(n+1)}{\beta } \big ( h(X^{'}_{n+1}) - h(X_{n+1}) \big ) \big | \mathcal {F}(n) \Big ) \Big ) \Big )^2 \Big | \mathcal {F}(n) \Big ] \\&\le \sum _n b^2(n+1) \bigg ( E \Big [ \frac{\eta _l^2(n+1)}{\beta ^2} \big ( h(X^{'}_{n+1}) - h(X_{n+1}) \big )^2 \big | \mathcal {F}(n) \Big ]\\&\quad + E \Big [ E^2 \big [ \frac{\eta _l(n+1)}{\beta } \big ( h(X^{'}_{n+1}) - h(X_{n+1}) \big ) \big | \mathcal {F}(n) \big ] \big | \mathcal {F}(n) \Big ] \bigg ) \\&\le \sum _n 2b^2(n+1) \bigg ( E \Big [ \frac{\eta _l^2(n+1)}{\beta ^2} \big ( h(X^{'}_{n+1}) - h(X_{n+1}) \big )^2 \big | \mathcal {F}(n) \Big ] \bigg ) \end{aligned}$$

where the second inequality follows by another application of conditional Jensen’s inequality. It can now be seen as before using an application of the Cauchy-Schwarz inequality as well as Assumptions 2 and 3 that

$$\begin{aligned} \sup _n \frac{1}{\beta ^2} E \big [ \eta _l^2(n+1) (h(X^{'}_{n+1}) - h(X_{n+1}))^2 \big | \mathcal {F}(n) \big ] < \infty \text{ w.p. } \text{1. } \end{aligned}$$

Now from Assumption 4, $\sum _n E[(Q_l(n+1) - Q_l(n) )^2 | \mathcal {F}(n)] < \infty$ a.s. Thus, the quadratic variation process of $\{Z_l(n)\}$ is almost surely convergent. Hence, by the martingale convergence theorem for square integrable martingales, $\{Q_l(n)\}$ are a.s. convergent martingale sequences. $\square$

Proof of Lemma 3

Note that (9) can be rewritten as

$$\begin{aligned} Z_l(n+1)&= Z_l(n) + b(n) \bigg ( E \Big ( \frac{\eta _l(n)}{\beta } \Big ( h \big ( X'_n \big ) - h \big ( X_n \big ) \Big ) \big | \mathcal {F}(n-1) \Big ) - Z_l(n) \bigg ) \nonumber \\&\quad + b(n) \bigg ( \frac{\eta _l(n)}{\beta } \Big ( h \big ( X'_n \big ) - h \big ( X_n \big ) \Big ) \nonumber \\&\quad - E \Big ( \frac{\eta _l(n)}{\beta } \Big ( h \big ( X'_n \big ) - h \big ( X_n \big ) \Big ) \big | \mathcal {F}(n-1) \bigg ). \end{aligned}$$

(34)

Hence we have

$$\begin{aligned}&\sum _n (Z_l(n+1) - Z_l(n))\nonumber \\&\quad = \sum _n b(n) \Big ( E \Big ( \frac{\eta _l(n)}{\beta } \Big ( h \big ( X'_n \big ) - h \big ( X_n \big ) \Big ) \big | \mathcal {F}(n-1) \Big ) - Z_l(n) \Big )\nonumber \\&\qquad \quad + \sum _n b(n) \Big ( \frac{\eta _l(n)}{\beta } \Big ( h \big ( X'_n \big ) - h \big ( X_n \big ) \Big ) - E \Big ( \frac{\eta _l(n)}{\beta } \Big ( h \big ( X'_n \big ) - h \big ( X_n \big ) \Big ) \big | \mathcal {F}(n-1) \Big ) \Big ). \end{aligned}$$

(35)

Note that as a consequence of Lemma 1, $\exists Q_l(\infty )<\infty$ a.s. such that $Q_l(n) \rightarrow Q_l(\infty )$ a.s. as $n\rightarrow \infty$. Now from the definition of $Q_l(n)$, it is clear that

$$\begin{aligned} Q_l(\infty ) =&\sum _{m=1}^\infty b(m) \bigg ( \frac{\eta _l(m)}{\beta } \Big ( h(X^{'}_{m}) - h(X_m) \Big ) \\&- E \Big ( \frac{\eta _l(m)}{\beta } \big ( h(X^{'}_{m}) - h(X_m) \big ) \big | \mathcal {F}(m-1) \Big ) \bigg ) < \infty \text{ a.s. } \end{aligned}$$

Note that the second term in the RHS of (35) is precisely $Q_l(\infty )<\infty$. As a consequence of the above, it is sufficient to show the boundedness of the following recursion in place of (34):

$$\begin{aligned} {\bar{Z}}_l(n+1) = {\bar{Z}}_l(n) + b(n) \bigg ( E \Big ( \frac{\eta _l(n)}{\beta } \Big ( h \big ( X'_n \big ) - h \big ( X_n \big ) \Big ) \big | \mathcal {F}(n-1) \Big ) - {\bar{Z}}_l(n) \bigg ),\nonumber \\ \end{aligned}$$

(36)

with ${\bar{Z}}(0) = Z(0)$.

As in the proof of Lemma 1 it can be seen that

$$\begin{aligned} \sup _n E \left[ \frac{\eta _l(n)}{\beta } \left( h \left( X'_n \right) - h \left( X_n \right) \right) \big | \mathcal {F}(n-1) \right] < \infty \end{aligned}$$

with probability 1. Now since $b(n) \rightarrow 0$ as $n \rightarrow \infty$, there exists a $p_0$ such that for all $n \ge p_0, \, 0 \le b(n) \le 1$. Hence for all $n \ge p_0, \, {\bar{Z}}(n+1)$ is a convex combination of ${\bar{Z}}(n)$ and a quantity that is almost surely uniformly bounded. The claim follows¹.