Recursive Modified Pattern Search on High-Dimensional Simplex : A Blackbox Optimization Technique

Abstract

In this paper, a novel derivative-free pattern search based algorithm for Black-box optimization is proposed over a simplex constrained parameter space. At each iteration, starting from the current solution, new possible set of solutions are found by adding a set of derived step-size vectors to the initial starting point. While deriving these step-size vectors, precautions and adjustments are considered so that the set of new possible solution points still remain within the simplex constrained space. Thus, no extra time is spent in evaluating the (possibly expensive) objective function at infeasible points (points outside the unit-simplex space); which being the primary motivation of designing a customized optimization algorithm specifically when the parameters belong to a unit-simplex. While minimizing any objective function of m parameters, within each iteration, the objective function is evaluated at 2m new possible solution points. So, upto 2m parallel threads can be incorporated which makes the computation even faster while optimizing expensive objective functions over high-dimensional parameter space. Once a local minimum is discovered, in order to find a better solution, a novel ‘re-start’ strategy is considered to increase the likelihood of finding a better solution. Unlike existing pattern search based methods, a sparsity control parameter is introduced which can be used to induce sparsity in the solution in case the solution is expected to be sparse in prior. A comparative study of the performances of the proposed algorithm and other existing algorithms are shown for a few low, moderate and high-dimensional optimization problems. Upto 338 folds improvement in computation time is achieved using the proposed algorithm over Genetic algorithm along with better solution. The proposed algorithm is used to estimate the simultaneous quantiles of North Atlantic Hurricane velocities during 1981–2006 by maximizing a non-closed form likelihood function with (possibly) multiple maximums.

Appendices

Appendix A: Discussion on the number of operations and objective function evaluations required at each iteration of RMPSS

Here we find the order of the number of basic operations and the number of objective function evaluations required at each iteration in terms of the dimension of the parameter space. Therefore, we find the upper bound of number of operations required for the worst case scenario which would be sufficient to determine the order of the number of basic operations.

Suppose we want to minimize f(p) where p ∈S. At the beginning of each iteration, 4 arrays of length m, i.e., s⁺, s⁻, f⁺, f⁺ are initialized (see step (2) of STAGE 1 in Section 2). During each iteration, starting from the current value of the parameter 2m candidate solutions are generated in a such way that each of them belongs to the domain S. Search algorithm for first m of these movements have been described in step (3) of STAGE 1 of Section 2.

In step (3) of STAGE 1 of Section 2, note that it requires not more than m operations to find $S_{i}^{+}$. To find $K_{i}^{+}$, it takes at most m operations. As we are considering the worst case scenario in terms of maximizing the number or required operations, assume $K_{i}^{+} \geq 1$. In step (3.1) and (3.2) of STAGE 1, suppose the value of $S_{i}^{+}$ is updated atmost k times. So, we have $\frac {s_{initial}}{\rho ^{k-1}} \leq \phi$ but $\frac {s_{initial}}{\rho ^{k-2}} > \phi$. Hence $k = 1+\left [\frac {\log (\frac {s_{initial}}{\phi })}{\log (\rho )}\right ]$, where [x] returns the largest interger less than or equal to x. Corresponding to each update step of $S_{i}^{+}$, first it is checked whether $s_{i}^{+} \leq \phi$ or not. It involves a single operation. Then deriving $\mathbf {q}_{i}^{+}$ involves not more than 2m steps because the most complicated scenario occurs for updating the positions of $\mathbf {q}_{i}^{+}$ which belong to $S_{i}^{+}$. And in that case, it takes total two operations for each site, one operation to find $\frac {s_{i}^{+}}{K_{i}^{+}}$ and one more to evaluate to subtract that quantity from $p_{i}^{(l)}$ for $l \in S_{i}^{+}$. In order to check whether $\mathbf {q}_{i}^{+} \in \mathbf {S}$ or not, it requires m operations. For the worst case scenario, we also add one more step required for updating $s_{i}^{+} = \frac {s_{i}^{+}}{\rho }$. Hence the search procedure of any movement (i.e., for any i ∈ {1, … , m}) in step (3) of STAGE 1 requires m + m + k × (1 + 2m + m + 1) = m × (2 + 3k) + 2k operations. Hence, for m movements (mentioned in step (3) of STAGE 1 in Section 2) it requires not more than m² × (2 + 3k) + 2mk operations. In a similar way, it can be shown that for step (4) also the maximum number of required operations is not more than m² × (2 + 3k) + 2mk.

In step (5) of STAGE 1 in Section 2, to find k₁ or k₂, it takes (m − 1) operations. The required number of steps for this step will be maximized if $\min \limits (f_{k_{1}}^{+},f_{k_{2}}^{-}) < Y^{(j)}$. Under this scenario, two more operations (i.e., comparisons) are required to find p_temp. So this step requires not more than 2 × (m − 1) + 2 = 2m operations.

In step (6) of STAGE 1 in Section 2, it takes at most m operations to find S_updated. In case K_updated is not m, the required number of operations required in this step would be more than the case when K_updated = m. For finding out the number of operations required for the worst case scenario, assume K_updated < m. To find the value of garbage, maximum number of required steps is not more than m. Finally, it can be noted that in step (6.2), updating the value of the parameter of interest from p^(j) to p^(j+ 1) requires not more than 2m steps. So maximum number of operations required for step (6) of STAGE 1 is not more than m + m + 2m = 4m.

In step (7) of STAGE 1 in Section 2, to find (p^(j)(i) −p^(j− 1)(i))² for each i ∈ {1, … , m}, we need one operation for taking difference, and one operation for taking the square. Hence to find the sum of the squares, it needs (3m − 1) more operations. Comparing it’s value with tol_fun takes one more operation. In the worst case scenario, it would take two more operations till the end of the iteration, i.e., update of s^(j) at step (7) and it’s comparison with ϕ at step (8). Hence after step (6), the required number of operations would be at most (3m − 1) + 1 + 2 = 3m + 2.

Hence for each iteration, in the worst case scenario, the number of required basic operations is not more than m²(2 + 3k) + 2mk + 2m + 3m + (3m + 2) = m²(2 + 3k) + m(2k + 8) + 2. So, number of basic operations required for each iteration in our algorithm is of O(m²) where m is the number of parameters to estimate.

Note that the number of times the objective function is evaluated in each iteration is 2m + 1 (once at step (2), m times at step (3.3) and m times at step (4.3) of STAGE 1 in Section 2). Thus, we note that the order of the number of function evaluations at each iteration step is of O(m).

Appendix B: Model and Likelihood of Simultaneous Quantile Regression

Let $Q_{y}(\tau |x)=\inf \{q : P(Y\leq q| X=x)\geq \tau \}$ denote the τ-th conditional quantile of a response Y at X = x for 0 ≤ τ ≤ 1, where X is the predictor. A linear simultaneous quantile regression model for Q_y(τ|x) at a given τ is given by

$$Q_{y}(\tau|x)=\beta_{0}(\tau)+x\beta(\tau)$$

where β₀(τ) denotes the intercept and β(τ) denotes the slope which are smoothly varying function of τ. After transforming the predictor and the response variables to unit variable by some monotonic transformation, as shown in Das and Ghosal (2017a), the linear quantile function can be represented as

$$ Q_{y}(\tau|x)= x\xi_{1}(\tau)+(1-x)\xi_{2}(\tau) \text{for} \tau \in [0,1], x,y \in [0,1], $$

(B.1)

for some functions ξ₁(τ) and ξ₂(τ) which are monotonically increasing in τ for τ ∈ [0,1] satisfying ξ₁(0) = ξ₂(0) = 0, ξ₁(1) = ξ₂(1) = 1. Equation B.1 can be re-framed as

$$ Q_{y}(\tau|x)= \beta_{0}(\tau)+x\beta_{1}(\tau) \text{for} \tau \in [0,1], x,y \in [0,1], $$

where β₀(τ) = ξ₂(τ) and β₁(τ) = ξ₁(τ) − ξ₂(τ) denotes the slope and the intercept of the quantile regression. The conditional density for Y is given by

$$ f_{y}(y|x) = \left( \frac{\partial}{\partial \tau}Q_{y}(\tau|x)|_{\tau=\tau_{x}(y)}\right)^{-1}=\left( \frac{\partial}{\partial \tau}\beta_{0}(\tau)+x\frac{\partial}{\partial \tau}\beta(\tau)|_{\tau=\tau_{x}(y)}\right)^{-1}, $$

(B.2)

where τ_x(y) solves the equation

$$ x\xi_{1}(\tau)+(1-x)\xi_{2}(\tau) = y. $$

(B.3)

Therefore for any given dataset $\{(X_{i},Y_{i})\}_{i=1}^{n}$, the likelihood is given by ${\prod }_{i=1}^{n} f_{Y}(y_{i}|x_{i})$.

Let 0 = t₀ < t₁ < … < t_k = 1 be the equidistant knots on the interval [0,1] such that t₀ = 0, t_k = 1 and $t_{i}=\frac {1}{k}$ for all i = 0,1, … , k − 1. Suppose $\{B_{j,h}(t)\}_{j=1}^{k+h}$ denote the basis functions of h^th degree B-splines on [0,1] on the above mentioned set of equidistant knots. Now, the basis expansion of ξ₁(⋅) and ξ₂(⋅) are given by

$$ \begin{array}{@{}rcl@{}} && \xi_{1}(\tau)=\sum\limits_{j=1}^{k+h} \theta_{j} B_{j,h}(\tau) \text{ where } 0=\theta_{1}<\theta_{2}<\cdots<\theta_{k+h}=1, \\ && \xi_{2}(\tau)=\sum\limits_{j=1}^{k+h} \phi_{j} B_{j,h}(\tau) \text{ where } 0=\phi_{1}<\phi_{2}<\cdots<\phi_{k+h}=1. \end{array} $$

(B.4)

Note that estimating $\{\theta _{j},\phi _{j}\}_{j=1}^{k+h}$ is equivalent to estimating $G =\{\gamma _{j}\}_{j=1}^{k+h-1},$ $D = \{\delta _{j}\}_{j=1}^{k+h-1}$ where

$$ \gamma_{j},\delta_{j} \geq 0, j=1,\cdots,k+h-1, \text{and} \sum\limits_{j=1}^{k+h-1}\gamma_{j}= \sum\limits_{j=1}^{k+h-1}\delta_{j}=1. $$

Appendix C: Application of RMPSS to estimate the membership probability vector of mixture model

In this section we discuss how RMPSS can be used to estimate the proportion vector of a mixture model (e.g., Gaussian mixture). Suppose X is a random variable which is coming from mixture of C classes with density functions $\{f_{j}(\cdot |\theta _{j})\}_{j=1}^{C}$ with probabilities p = (p₁, … , p_C) such that $p_{j}\geq 0,{\sum }_{j=1}^{C}p_{j} = 1$. So in this case, the density of the univariate mixture model is given by

$$ L(f(x|\mathbf{p},\boldsymbol{\theta}) = \sum\limits_{j=1}^{C} p_{j}f_{j}(x|\theta_{j}), $$

(B.1)

where θ = (θ₁, … , θ_C) denotes the parameters of C classes. For a given sample $\{x_{i}\}_{i=1}^{n}$, the likelihood is given by

$$L(\mathbf{p},\boldsymbol{\theta}) = \prod\limits_{i=1}^{n}\sum\limits_{j=1}^{C} p_{j}f_{j}(x_{i}|\theta_{j}).$$

We consider the case where θ_js are univariate.

Case 1 ::

When θ is known :

In case, θ is known, the likelihood is a function of only the proportion vector p. Now, since θ_js are known, while estimating p, problem of identifiability would not occur as the order of θ_js cannot change, so changing the order of elements of p would not produce the same likelihood value for all $x \sim X$ assuming all θ_js are different. So the proportion vector can be estimated using RMPSS without further modification.

Case 2 ::

When θ is unknown :

In case, θ is also unknown, both p = (p₁, … , p_C) and θ = (θ₁, … , θ_C) are needed to be estimated. However, unlike previous scenario, in this case, the likelihood function is not identifiable. For example, suppose $\mathbf {p}^{*}=(q_{1},q_{2},q_{3}), \mathbf {p}^{**}=(q_{2},q_{1},q_{3}), \boldsymbol {\theta }^{*} = (\phi _{1},\phi _{2}, \phi _{3}), \boldsymbol {\theta }^{**} = (\phi _{2},\phi _{1}, \phi _{3})$ for C = 3. Then note that L(p^∗, θ^∗) = L(p^∗∗, θ^∗∗) holds for any given sample. In order to get rid of the identifiability problem, we set a natural ordering of the θ parameter space such that θ₁ > θ₂ > ⋯ > θ_C. Define, β_j = θ_j − θ_j− 1 for j = 2, ⋯ , C. So, in case θ ∈R^C, then the new set of parameters are given by $\boldsymbol {\theta }^{\prime } = \{\theta _{1},\beta _{2},\beta _{3},\ldots ,\beta _{C}\} \in \mathrm {R}\times \mathrm {R^{+}}^{C-1}$. Now, to estimate the solution $\hat {\mathbf {p}}$ and $\hat {\boldsymbol {\theta }^{\prime }}$ for which the likelihood function is maximized, RMPSS can be used along with any other Black-box algorithm (e.g., GA, SA, PSO etc), such that at any given iteration, RMPSS is used to maximize the likelihood in terms of p fixing the value of $\boldsymbol {\theta }^{\prime }$ at current value, then Genetic Algorithm (or any other Blackbox algorithm) is used to maximize the likelihood in terms of $\boldsymbol {\theta }^{\prime }$ fixing the value of p at the current updated value. Suppose p^(k) and $\boldsymbol {\theta ^{\prime }}^{(k)}$ denote the updated values of p and $\boldsymbol {\theta ^{\prime }}$ at the beginning of k-th iteration (k ≥ 2). Then following iteration steps should be performed iterartively until final solution is obtained.

• While $L_{1}(\mathbf {p}^{(k)},\boldsymbol {\theta ^{\prime }}^{(k)}) \neq L_{1}(\mathbf {p}^{(k-1)},\boldsymbol {\theta ^{\prime }}^{(k-1)})$

  • 1. Set k = k + 1.

  • 2. Set $\mathbf {p}^{(k)} = {\arg \max \limits }_{\mathbf {p}}, L_{1}(\mathbf {p}, \boldsymbol {\theta ^{\prime }} = \boldsymbol {\theta ^{\prime }}^{(k-1)}$) by solving with RMPSS where p ∈ Δ^C− 1,

  • 3. Set $\boldsymbol {\theta ^{\prime }}^{(k)} = {\arg \max \limits }_{\boldsymbol {\theta ^{\prime }}} L_{1}(\mathbf {p} = \mathbf {p}^{(k)}, \boldsymbol {\theta ^{\prime }})$ by solving with GA (or any other Black-box optimization technique) where $\boldsymbol {\theta ^{\prime }} \in \mathrm {R}\times \mathrm {R^{+}}^{C-1}$,

where $L_{1}(\mathbf {p}, \boldsymbol {\theta ^{\prime }}) = L(\mathbf {p}, \boldsymbol {\theta })$ and

$$ \begin{array}{@{}rcl@{}} {\Delta}^{C-1} = \left\{(p_{1}, \ldots, p_{C}) \in \mathbb{R}^{C} | p_{i} \geq 0, i=1,\ldots,C, \sum\limits_{i=1}^{C}p_{i}= 1\right\}. \end{array} $$