Black-box optimization on hyper-rectangle using Recursive Modified Pattern Search and application to ROC-based Classification Problem

Abstract

In statistics, it is common to encounter multi-modal and non-smooth likelihood (or objective function) maximization problems, where the parameters have known upper and lower bounds. This paper proposes a novel derivative-free global optimization technique that can be used to solve those problems even when the objective function is not known explicitly or its derivatives are difficult or expensive to obtain. The technique is based on the pattern search algorithm, which has been shown to be effective for black-box optimization problems. The proposed algorithm works by iteratively generating new solutions from the current solution. The new solutions are generated by making movements along the coordinate axes of the constrained sample space. Before making a jump from the current solution to a new solution, the objective function is evaluated at several neighborhood points around the current solution. The best solution point is then chosen based on the objective function values at those points. Parallel threading can be used to make the algorithm more scalable. The performance of the proposed method is evaluated by optimizing up to 5000-dimensional multi-modal benchmark functions. The proposed algorithm is shown to be up to 40 and 368 times faster than genetic algorithm (GA) and simulated annealing (SA), respectively. The proposed method is also used to estimate the optimal biomarker combination from Alzheimer’s disease data by maximizing the empirical estimates of the area under the receiver operating characteristic curve (AUC), outperforming the contextual popular alternative, known as step-down algorithm.

Appendices

Appendix A : RMPS and Generalized Pattern Search

Here we discuss some of the key features of RMPS along with the differences of RMPS with a few existing Pattern Search based optimization techniques (e.g., Torczon (1997)). Firstly, the restart strategy with smaller step-size decay rate is something which is possibly proposed for the first time in the context of Pattern search, to the best of our knowledge. Secondly, unlike algorithm 1 of Torczon (1997), instead of unconstrained minimization, the proposed algorithm minimizes the black-box function on a hyper-rectangle. In Fermi and Metropolis (1952) and Torczon (1997), the coordinate-wise jump sizes were kept equal inside an iteration while in the proposed algorithm, the domain of each coordinate being bounded, in every iteration, local-step sizes are modified separately for each coordinates in each direction (positive and negative), as required. In GPS, each coordinate-wise jump step-sizes are evaluated using ‘exploratory moves algorithm’ (see Torczon (1997)) while in the proposed algorithm it is straightforward and does not use ‘exploratory moves algorithm’. While optimizing a function on a hyper-rectangle, since the domain is transformed into an unit hyper-cube, the global step-size is kept same for each coordinate. So while determining the step-sizes of coordinate-wise movements, the proposed algorithm uses different strategy than the ‘exploratory moves algorithm’. The most unique feature of the proposed algorithm is the restart strategy as described in the main paper, in details.

Appendix B : Tuning parameters and their roles

In this section, a brief description of the tuning parameters and their roles are provided which are as follows.

• step decay rate ($\rho$) : $\rho$ determines the rate of change of global step size at the end of each iteration. . So it is understandable that the value of $\rho$ must be greater than 1. Taking smaller values of $\rho$ will make the decay of step sizes slower, which would allow finer search within the domain at the cost of more computation time.

• step size threshold ($\phi$) : $\phi$ controls the precision of the solution. This is the minimum possible value that the global step size and the local step sizes could take. Once the global step size reaches below $\phi$, the run is terminated. Taking $\phi$ to be smaller results in better precision in the cost of higher computation time.

• $\textit{tol\_fun}$ : $\textit{tol\_fun}$ denotes the minimum amount of improvement after an iteration in the solution which is required to keep the value of global step size unchanged. In other words, if the differences of solutions obtained in two consecutive iterations is less than tol_fun, the improvement is considered to be ‘not significant’ and in that scenario, the global step size is decreased for next iteration for employing a finer search.

• $\textit{tol\_fun\_2}$ : The second run onwards, whenever a run terminates, it is checked whether the solution returned by the current run is the same or different with the solution returned by the previous run. However, to check whether they are exactly equal, they need to be matched up to several decimal places depending on the type of storage variable used by the software. Thus it might result into performance of a lot of extra runs just to improve the solution at distant decimal places which might be not of our interest. Therefore, once the euclidean distance of solution points obtained from two consecutive runs become less than $\textit{tol\_fun\_2}$, the algorithm is terminated and the final result is returned.

Table 4 Domains of search regions of the benchmark functions considered in Section 4 in the main paper

Appendix C : Additional comparative performance study of RMPS

The domains of the optimization problems considered in Table 1 of the main paper is provided in Table 4. Figures 5 and 6 can be used as an aid for visualization corresponding to the theoretical properties of RMPS shown in Section 3 of the main paper.

Fig. 5

$\textbf{U} = \prod _{i=1}^n(u_i-r,u_i+r)$

Fig. 6

Neighborhood of $u_i$

1.1 Exploiting convexity

The prior knowledge of convexity can be used to save computation time using RMPS. In order to improve computation time for minimizing convex functions, we also consider RMPS taking $\rho =4$ which employs steeper decrease in global step size and local step sizes. In Table 5, a comparison study of performances of RMPS, and changed RMPS with the prior knowledge of convexity (RMPS(c), (i.e., max_runs=1, $\rho =4$ with default values of other parameters), GA and SA are provided for minimizing Sphere and Sum squares function for various dimensions starting from 10 randomly generated starting points (under 10 random number generating seeds in MATLAB) in each cases. It is noted that in each cases, RMPS(c) performs faster than RMPS. Also in terms of computation times, using RMPS(c) up to 40 folds improvement is observed compared to GA and up to 92 folds improvement is observed compared to SA.

In Table 6 we note down the comparative performance results of RMPS, GA and SA when the true solution lies at a boundary point of the domain.

Table 5 Comparative study of RMPS, RMPS(c)(i.e., max_runs=1, $\rho =4$ with default values of other parameters)

Table 6 Comparative study of RMPS, GA and SA for minimizing 100 and 1000 dimensional Ackley, Griewank, Rastrigin, Schwefel, Sphere and Sum squares functions where the true solution is a boundary point. Benchmark functions are minimized starting from 10 randomly generated starting points from the corresponding domains using RMPS, GA and SA. The minimum of the 10 obtained minimum values of the objective functions is noted down for each method. For RMPS, also the maximum of these 10 obtained minimum objective function values is noted down as well

Appendix D : Optimization on Hypervolume under manifolds

Upper and Lower Bound Approach (ULBA) :

Hsu and Chen (2016) showed the following equality holds true

$$\max \{0, (H-1)P_{A}(\varvec{\beta }) - (H-2)\} \le D(\varvec{\beta }) \le P_{H}(\varvec{\beta }),$$

where $P_{A}(\varvec{\beta })$ and $P_{H}(\varvec{\beta })$ are defined by

$$P_{A}(\varvec{\beta }) =\frac{1}{H-1} \displaystyle \sum _{j=1}^{H-1}P(\varvec{\beta }^{T}\textbf{X}_{j+1}>\varvec{\beta }^{T}\textbf{X}_{j}), P_{H}(\varvec{\beta }) = \min _{1 \le j \le H-1} P(\varvec{\beta }^{T}\textbf{X}_{j+1}>\varvec{\beta }^{T}\textbf{X}_{j}).$$

They proposed to maximize $P_{A}(\varvec{\beta })$ in order to obtain the optimal biomarker combination. Compared to EHUM, ULBA objective function is much less expensive to compute. However, rest of the challenges for maximizing EHUM (e.g., discontinuity, possible multi-modal nature) also applies here as well.

Step-down algorithm :

The step-down algorithm for maximizing any given objective function is given as follows:

• 1. Step 1. EHUM values of the individual biomarkers are computed and based on their individual EHUM values, they are arranged in decreasing order. Suppose $X_{(1)}$ and $X_{(d)}$ denote the biomarkers with the highest and the lowest individual EHUM values, respectively.

• 2. Step 2. The first two biomarkers with the highest EHUM values are taken and combined as $V_2=X_{(1)} + \lambda _2 X_{(2)}$, where $\lambda _2$ is a parameter that needs to be estimated.

• 3. Step 3. The objective function for the combined biomarker $V_2$ is maximized with respect to $\lambda _2$. Let $\widehat{V}_2 = X_{(1)} + \widehat{\lambda }_2 X_{(2)}$ denote the updated combination vector.

• 4. Step 4. For $i = 3,\ldots ,d$ define $V_i = \widehat{V}_{i-1}+\lambda _i X_{(i)}$ and maximize the objective function with respect to $\lambda _i$. The combination vector obtained at i-th step is given by $\widehat{\lambda }_i$.

The estimated optimal marker using step-down algorithm is given by $\widehat{V}_d = X_{(1)} + \widehat{\lambda }_2 X_{(2)} + \cdots + \widehat{\lambda }_d X_{(d)}$.

Fig. 7

Original picture containing missing pixels

Fig. 8

Completed picture using SCAD penalized matrix completion with RMPS

Appendix E : On parallel computation with RMPS : Matrix Completion Problem with SCAD penalty

RMPS is parallelizable and up to 2n parallel threads can be used while solving a n-dimensional black-box problem. However in the simulation study part, the time required for optimizing each function is noted down for single thread computing only. To perform parallel computing in MATLAB, in case parfor loop is used instead of for loop, depending on the operations performed within the loops (e.g., objective function evaluation), a scenario might arise where for loop works faster than parfor loop. Because at the beginning of the parfor loop, an amount of time is spent in distributing the parallelizable works to different workers and after termination of parallel jobs, some amount of time is spent for gathering the results. However no such additional time is spent when job is done in single thread (via for loop). So, in case the objective function is not computationally expensive enough compared to the amount of time required for distribution and collection of results in parallel computing (using parfor loop), it is possible for the multi-threaded job to be more time consuming compared to the single-thread computation. We note that for all the considered simulation experiments, time required using single thread computation is faster than using parallel computation. Therefore, all the results regarding computation times are noted down for single threaded computing only. However it might not be the case if the objective function is actually computationally very expensive. In order to show the benefit of using parallel computation with RMPS we consider the following case study.

The problem of recovering an unknown matrix from only a given fraction of its entries is known as matrix completion problem. Candes and Recht (2009) first proposed a method to recover a matrix from a few given entries solving a convex optimization problem. Later to solve this problem, Candes and Tao (2010) minimized nuclear norm of the matrix subject to the constraint that the given entries of the matrix should be the same. In other words, suppose we have a matrix $\textbf{Y}=(y_{ij})_{n \times n}$ with some missing values. In that case, as mentioned in Candes and Tao (2010), the complete matrix $\textbf{Y}$ can be obtained by solving the following problem,

$$\begin{aligned}&\text {minimize :} \; ||\textbf{X}||_* \\&\text {subject to :} \; x_{ij}=y_{ij} \;\text {for all observed} \; (i,j), \end{aligned}$$

where $||\textbf{M}||_*=\sum _i\sigma _i(\textbf{M})$ denotes the nuclear norm, $\sigma _i(\textbf{M})$ being the i-th singular value of matrix $\textbf{M}$. This problem can be solved using convex optimization technique. On a closer look, it can be noted that minimizing nuclear norm in this fashion in similar to the LASSO (Tibshirani (1996)) penalty term. Fan and Li (2001) proposed Smoothly Clipped Absolute Deviation (SCAD) penalty which was shown to have more desirable properties compared to LASSO for solving shrinkage based variable selection problems. But unlike LASSO, SCAD penalty is not a convex minimization (or concave maximization) problem. In this section, the matrix completion problem is solved using the SCAD penalty with RMPS. The matrix completion problem using SCAD penalty can be re-formulated as

$$\begin{aligned}&\text {minimize :} \; \sum _i f(\sigma _i(\textbf{X})) \nonumber \\&\text {subject to :} \; x_{ij}=y_{ij} \;\text {for all observed} \; (i,j), \end{aligned}$$

(1)

where $\sigma _i(\textbf{X})$ are singular values and $f_i$ is the SCAD penalty function dependent of tuning parameters $\lambda$ and $a\;(=3.7)$ (see Fan and Li (2001)).

We consider a picture (Fig. 7) with $61\times 61$ pixels where approximately half (1877 to be precise) of its pixels are missing. The problem given by Eq. 1 can been seen as a black-box function of dimension of 1877 (i.e., the number of missing pixels). It is also known that the numerical value of grey level of each pixel must be between 0 and 255. This problem is solved using RMPS method. We fit the model for 30 values of $\lambda$ which are $\{100,200, \ldots , 3000\}$ and only the best visual output ($\lambda =900$) is reported in Fig. 8.

Unlike the objective functions considered in the performance evaluation studies in the main paper, the evaluation of SCAD penalty based on the singular values of the matrix is very computationally intense. Thus, unlike previous cases, here using parallel computing is noted to be beneficial. It should be noted that this is a 1877 dimensional problem and therefore up to 3754 parallel threads can be used while solving it using RMPS algorithm. We use 4 parallel threads to derive the complete image given in Fig. 8. For comparison of computation time required by single threading and parallel threading with 4 threads, the required computation times for first 50, 100 and 200 iterations are provided for all cases in Table 7. We get more than 3 folds improvement in computation time using parallel threading (with 4 threads) instead of single threading.

Table 7 Computation times (in seconds) required for first 50, 100 and 200 iterations of RMPS while solving matrix completion problem with SCAD penalty using single thread and 4 parallel threads