Abstract
This paper investigates the integration of a surrogateassisted multiobjective evolutionary algorithm (MOEA) and a parallel computation scheme to reduce the computing time until obtaining the optimal solutions in evolutionary algorithms (EAs). A surrogateassisted MOEA solves multiobjective optimization problems while estimating the evaluation of solutions with a surrogate function. A surrogate function is produced by a machine learning model. This paper uses an extreme learning surrogateassisted MOEA/D (ELMOEA/D), which utilizes one of the wellknown MOEA algorithms, MOEA/D, and a machine learning technique, extreme learning machine (ELM). A parallelization of MOEA, on the other hand, evaluates solutions in parallel on multiple computing nodes to accelerate the optimization process. We consider a synchronous and an asynchronous parallel MOEA as a masterslave parallelization scheme for ELMOEA/D. We carry out an experiment with multiobjective optimization problems to compare the synchronous parallel ELMOEA/D with the asynchronous parallel ELMOEA/D. In the experiment, we simulate two settings of the evaluation time of solutions. One determines the evaluation time of solutions by the normal distribution with different variances. On the other hand, another evaluation time correlates to the objective function value. We compare the quality of solutions obtained by the parallel ELMOEA/D variants within a particular computing time. The experimental results show that the parallelization of ELMOEA/D significantly reduces the computational time. In addition, the integration of ELMOEA/D with the asynchronous parallelization scheme obtains higher quality of solutions quicker than the synchronous parallel ELMOEA/D.
Similar content being viewed by others
1 Introduction
Evolutionary algorithms (EAs) have been applied to many realworld applications like engineering (Obayashi et al. 2010; Oyama et al. 2017), data mining (Devos et al. 2014; Soufan et al. 2015), electronics (Roberge et al. 2014), and nanoscience (Shayeghi et al. 2015; Davis et al. 2015) because of their high search capability without any preliminary knowledge of a target problem. Since many realworld applications include two or more objectives with conflict, i.e., a tradeoff, multiobjective evolutionary algorithms (MOEAs) have attracted much attention for dealing with multiple objectives simultaneously. One problem when applying MOEAs to realworld applications is a necessity of enormous computing time to obtain optimal solutions. This is because MOEAs need to evaluate many tentative solutions through the optimization process, and the evaluation of tentative solutions is computationally expensive.
Previous works have proposed surrogateassisted EAs (SAEAs) (Jin 2011; Haftka et al. 2016) to reduce the computational time required for the EA process. SAEAs construct a surrogate model that estimates the evaluation values of solutions from already evaluated solutions using machine learning techniques. A proper optimizer, like an MOEA, is executed using the evaluation estimated by a constructed surrogate model. Finally, the surrogateassisted optimization obtains promising solutions that can exhibit superior actual evaluation value, and a timeconsuming process actually evaluates them. The computational time of the estimation with a surrogate model is extremely shorter than the actual evaluation, and the actual evaluation is applied merely to promising solutions. Thus, an SAEA can reduce the computational time to obtain optimal solutions. Previous works have proposed some surrogateassisted MOEAs, for example, MOEA/DRBF (Zapotecas Martínez and Coello Coello 2013) or ParEGO (Knowles 2006), which use a Gaussian process or a radial based function (RBF) as a surrogate model. ELMOEA/D (Pavelski et al. 2014, 2016), extreme learning assisted MOEA/D, is one of the stateoftheart as a novel SAEA. This method adopts the surrogate model based on an extreme learning machine (ELM) (Huang et al. 2004), and MOEA/D Zhang and Li (2007) generates promising solutions with the ELM surrogate evaluation. Their work demonstrated that ELMOEA/D outperforms ParEGO and MOEA/DRBF on multiobjective optimization benchmarks especially when the dimension of the search space is large.
To further reduce computational time, the parallelization of SAEAs can be considered as a potential attempt. Parallel EAs (PEAs) (Alba and Tomassini 2002; Alba et al. 2013) also have been studied in the last decades. In particular, a masterslave parallelization is a straightforward approach to implement PEAs. A masterslave PEA performs the main procedure of EAs, i.e., initialization, selection, variation, and replacement, on one master node. Many slave nodes, on the other hand, execute evaluations of newly generated solutions in parallel. Since the solution evaluation is the most timeconsuming part in EAs, PEAs can reduce the computational time compared to sequentially evaluating solutions. In recent years, several works have studied the integration of SAEA with parallelization for reducing the computational time of the EA process. For example, a parallel efficient global optimization has been proposed (Wang et al. 2016) and applied to the optimization of microwave antennas (Liu et al. 2019). The work of Akinsolu et al. (2019) proposed parallel surrogateassisted MOEA with the Gaussian process and applied it to electromagnetic design optimization.
Masterslave PEAs can be classified into two schemes, synchronous PEAs (SPEAs) and asynchronous PEAs (APEAs). SPEAs wait for all evaluations of solutions done by slave nodes and generate a new population by utilizing all newly evaluated solutions. APEAs, on the other hand, continuously generate a new solution without waiting for evaluations of other solutions. Since APEAs do not wait for the slowest solution evaluation, it is possible to efficiently utilize the computing resource of slave nodes in a situation where the evaluation of solutions differs. The previous works that integrate SAEA and the parallelization have considered only the synchronous PEA. However, it is generally assumed that the evaluation time of solutions is enormous and generally differs. For this fact, we consider that the integration of an SAEA with the asynchronous parallelization scheme is effective. The purpose of this study is to clarify which scheme of the synchronous and the asynchronous scheme is suitable for SAEA. Specifically, considering the parallelization of ELMOEA/D, we compare a synchronous parallel ELMOEA/D (SPELMOEA/D) and an asynchronous parallel ELMOEA/D (APELMOEA/D).
This paper conducts experiments that compare SPELMOEA/D with APELMOEA/D to investigate which parallelization scheme is appropriate for ELMOEA/D. We test these methods on wellknown multiobjective optimization benchmarks, ZDT series (Zitzler et al. 2000), WFG series (Huband et al. 2005), and DTLZ series (Deb et al. 2002). We also use a few benchmarks from realworld problems provided in the blackbox optimization competition (Loshchilov and Glasmachers 2015). This paper employs a simulated masterslave parallel computing environment proposed in the work of Zăvoianu et al. (2015) to measure the computational time of the competitive methods. We simulate two settings of the evaluation time in the experiment to investigate the influence of the different evaluation time characteristics. One evaluation time is determined by the normal distribution with the different variances, while another evaluation time correlates to the objective function value. We employ a hypervolume (HV) indicator (Zitzler and Thiele 1998) to assess the quality of obtained solutions.
The remaining of this paper is organized as follows. Sect. 2 firstly presents the description of a multiobjective optimization problem and explains the detail of MOEA/D. Section 3 shows the detail of ELM and ELMOEA/D. Section 4 briefly explains PEAs and proposes integrated algorithms of SPELMOEA/D and APELMOEA/D. Section 5 describes the experimental setup to compare the synchronous and asynchronous ELMOEA/D on multiobjective optimization problem (MOP) benchmarks and realworld problems. Section 6 shows the results on MOP benchmarks with the normallydistributed evaluation time, while Sect. 7 shows the one with the fitnesscorrelated evaluation time. Section 8 shows the results on realworld problems. Finally, Sect. 9 concludes this research and shows future work.
2 Multiobjective evolutionary algorithms
This section firstly explains what multiobjective optimization problem is, and then, we show a detailed algorithm of MOEA/D (Zhang and Li 2007), which is an underlying algorithm of ELMOEA/D.
2.1 Multiobjective optimization problem
A multiobjective optimization problem (MOP) is a problem of minimizing or maximizing m mutually competing objective functions \({\varvec{f}}(\mathbf{x})\) (Deb 2001) expressed as follows:
where \({{\mathbf {x}}}=(x_1,\ldots ,x_{D})\) is a D dimensional decision variable. \(x_{L,d}\) and \(x_{U,d}\) are the lower and upper bounds of the dth variable, respectively. \({\varvec{f}}:\varOmega \rightarrow {\mathbb {R}}^m\) is a vector having m objective functions and maps the decision variable space \(\varOmega\) to the objective space \({\mathbb {R}}^m\).
It is difficult to obtain a single optimal solution in MOPs because each objective function is in a tradeoff relationship. Therefore, the goal of MOPs is to acquire the Pareto optimal solution set. The Pareto optimal solution set is defined with the dominance relationship of solutions in MOPs denoted as follows:
where \({{\mathbf {x}}} \prec {{\mathbf {y}}}\) means \({{\mathbf {x}}}\) dominates \({{\mathbf {y}}}\). The Pareto optimal solution set consists of solutions that are not dominated by any other solutions, and it is denoted as follows:
2.2 MOEA/D
MOEA/D (Zhang and Li 2007) is a multiobjective evolutionary algorithm based on decomposition. It decomposes multiple objectives into many single objective subproblems based on an aggregation function associated with the weight vector uniformly distributed on the objective space. MOEA/D simultaneously performs many singleobjective searches with genetic operations within a local region determined by the neighborhood of the weight vectors. This paper adopts MOEA/DDE (Li and Zhang 2009) that introduces a differential evolution (DE) operator (Storn and Price 1997; Das and Suganthan 2011) into MOEA/D as a crossover operator.
Algorithm 1 describes the detailed algorithm of MOEA/DDE. Firstly, the weight vector sets W are generated uniformly on the objective space. W consists of N (the population size) weight vectors \(\varvec{\lambda }_i\,(i=1, \ldots , m)\). The weight vector is used to decompose multiple objectives to a single objective with an aggregation function (detailed later). Then, the neighborhood set \({\mathbf {B}}(i)\) is generated, consisting of T weight vectors closest to the weight vector \(\varvec{\lambda }_i\). Next, the initial population is generated and evaluated. The ideal point \({\mathbf {z}}^{*}\) is calculated from the objective function values of the initial population. The parent solutions for the DE operator are selected from the neighborhood set \({\mathbf {B}}(i)\) with the probability of \(\delta\), while selected from the entire population with the probability of \((1\delta )\). After applying the DE operator and the polynomial mutation (PM) (Deb and Goyal 1996), a newly generated solution is evaluated, and the ideal point \({\mathbf {z}}^*\) is updated. Finally, the population is updated according to the aggregation function value of a new solution. These procedures are repeated until the stopping criteria are satisfied.
Although previous works have proposed several aggregation functions, this research uses the Penalty Boundary Intersection (PBI) aggregation function. PBI is defined as Eqs. (6, 7 and 8):
where \(\theta\) is the penalty factor.
3 Extreme learning surrogateassisted MOEA/D
This section introduces the original extreme learning assisted MOEA/D, ELMOEA/D (Pavelski et al. 2016). ELMOEA/D uses an extreme learning machine (ELM) (Huang et al. 2004) as a surrogate model and generates promising solutions by using MOEA/DDE (Li and Zhang 2009) according to objective function values estimated by a constructed ELM model. In the following subsections, we first explain the overview of ELM, and then we provide the detail of ELMOEA/D.
3.1 Extreme learning machine (ELM)
ELM is a kind of machine learning technique (Huang et al. 2004). ELM is constructed as a single layer feedforward neural network that consists of L hidden layer neurons weighted by \({\mathbf {a}}_j\,(j=1, \ldots , L)\), output weights \(\varvec{\beta }\), and an activation function \(G({{\mathbf {x}}}, {{\mathbf {a}}}, b)\). \({{\mathbf {a}}}\) indicates weights to a hidden neuron, while b indicates a bias value. The output of ELM is calculated as \(Y={\mathbf {H}}\varvec{\beta }\), where \(\varvec{\beta }\) is a learned parameter, while \({\mathbf {H}}\) is a matrix of the activation function as \({\mathbf {H}}=\{G({\mathbf {x}}_i, {\mathbf {a}}_j, b_j)\}\).
The most notable feature of ELM is that the hidden layer weights \({{\mathbf {a}}}\) and biases b are randomly assigned and are not learned. The output weights \({\varvec{\beta }}\) are only parameters to be learned from N distinct training samples of ndimensional inputs \({{\mathbf {x}}}_i\) with m dimensional outputs \({{\mathbf {t}}}_i\). In particular, \({\varvec{\beta }}\) is calculated as \(\varvec{\beta }=\left( {\mathbf {I}}/C +{\mathbf {H}}{\mathbf {H}}^T\right) ^{1}{\mathbf {H}}^TT\), where C is a regularization parameter, \({{\mathbf {I}}}\) indicates an \(N\times N\) unit matrix, and \({{\mathbf {T}}}=\left[ {{\mathbf {t}}}_1, \ldots , {\mathbf{t}}_N\right]\) is the matrix of desired outputs.
ELM can use any piecewise nonlinear continuous activation functions in the hidden layer, i.e., \(G({{\mathbf {x}}}, {{\mathbf {a}}}, b)\). The original work of ELMOEA/D used the following three activation functions:

Sigmoid (SIG):
$$\begin{aligned} G_{SIG}({{\mathbf {x}}}, {{\mathbf {a}}}, b)=\frac{1}{1+\exp \left( \left( {{\mathbf {a}}}\cdot {{\mathbf {x}}} + b\right) \right) } \end{aligned}$$(9) 
Gaussian (GAU):
$$\begin{aligned} G_{GAU}({{\mathbf {x}}}, {{\mathbf {a}}}, b)=\exp \left( b{\mathbf{x}}{{\mathbf {a}}}^2\right) \end{aligned}$$(10) 
Multiquadric (MQ):
$$\begin{aligned} G_{MQ}({{\mathbf {x}}}, {{\mathbf {a}}}, b)=\left( {{\mathbf {x}}}  {\mathbf{a}}^2+b^2\right) ^{\frac{1}{2}} \end{aligned}$$(11)
The advantages of ELM to neural networks and support vector machines (SVMs) are summarized as follows (Huang et al. 2004; Huang 2015):

The learning speed of ELM is extremely fast. It can train singlelayer feedforward networks much faster than classical learning algorithms.

Unlike traditional classic gradientbased learning algorithms, which intend to reach minimum training error but do not consider the magnitude of weights, ELM tends to reach not only the smallest training error but also the smallest norm of weights. Thus, ELM tends to have better generalization performance for feedforward neural networks.

Unlike the traditional classic gradientbased learning algorithms that only work for differentiable activation functions, the ELM learning algorithm can be used to train singlelayer feedforward networks with nondifferentiable activation functions.

Unlike traditional classic gradientbased learning algorithms facing several issues like local minimum, improper learning rate, and overfitting, etc, ELM tends to reach the solutions straightforward without such trivial issues. The ELM learning algorithm looks much simpler than most learning algorithms for feedforward neural networks.

Unlike SVM, which does not consider the feature representation and functioning roles of each inner hidden layer, ELM studies feature representations in each layer.
3.2 Algorithm of ELMOEA/D
Algorithm 2 briefly describes the algorithm of ELMOEA/D (see detail in the work of (Pavelski et al. 2016)).
ELMOEA/D uses Latin hypercube sampling (LHS) (McKay et al. 1979) as the sampling method in the initialization step. ELMOEA/D uses two weight vector sets, one is used for the evolution of MOEA/D, while another is used to select promising solutions. A weight vector set W used for the evolution of MOEA/DDE is generated that uniformly spread on the objective space. The weight vector set consists of N weight vectors (\(W=\{{\varvec{\lambda }}_1, \ldots , {\varvec{\lambda }}_N\}\)). Subsequently, a selector weight vector set \(W_s\) used for the selection of \(N_s\) promising solutions is generated, which consists of \(N_s\) selector weight vectors \(W^s=\{{\varvec{\lambda }}^{s}_{1}, \ldots , {\varvec{\lambda }}^{s}_{N_s}\}\subset W\). Finally, a neighborhood vector set for the promising solution selection \({\mathbf {B}}^s(i)=\{{\varvec{\lambda }}_{i,1},\ldots ,{\varvec{\lambda }}_{i,K_s}\}\,(\varvec{\lambda }_{i, j}\in W)\) is associated with each selector weight vector \(\varvec{\lambda }_i^s\in W_s\). Each neighborhood vector set \({\mathbf {B}}^s(i)\) consists of \(K_s(=N/N_s)\) weight vectors in W that are closest to each selector weight vector \(\varvec{\lambda }_i^s\).
After the initialization, ELMOEA/D constructs an ELM surrogate model by using a training set \(P_t\). ELMs are trained with different activation functions and different normalization parameters C in this step. The best ELM model that performs the smallest mean square error (MSE) for the training set is selected and used as a surrogate model.
The constructed ELM model is used for evaluating solutions during the MOEA/DDE procedure. The initial population is randomly selected from the nondominated solutions in the archive population \(P_a\) if its size exceeds the population size N. Otherwise, LHS generates the remaining solutions in the bound of \([\varvec{\mu }\varvec{\sigma }, \varvec{\mu }+\varvec{\sigma }]\), where \({\varvec{\mu }}\) and \({\varvec{\sigma }}\) are the mean and the standard deviation calculated from the nondominated solutions in \(P_a\).
The promising solutions for the actual evaluation are selected from the final population after the optimization of MOEA/DDE. The aggregation function values of solutions in the final population are calculated for each \({\varvec{\lambda }}_{i, j} \in {\mathbf{B}}^s(i)\) based on the estimated objective values by the surrogate model. Then, the best solution is selected from each neighborhood vector set \({\mathbf {B}}^s(i)\), and totally, \(N_s\) promising solutions are selected for the actual evaluation.
The previous work (Pavelski et al. 2016) reported that ELMOEA/D has the following advantages:

Since ELMOEA/D uses a simpler surrogate model with less userspecific parameters, it consumes less computational resources and is faster than the conventional surrogateassisted EA.

ELMOEA/D showed promising results on the MOP benchmarks with high dimensional design variables compared with ParEGO and MOEA/DRBF.
For these advantages, our research employs ELMOEA/D.
4 Parallelization of ELMOEA/D
This paper considers two schemes of ELMOEA/D parallelization, a synchronous parallel ELMOEA/D (SPELMOEA/D), and an asynchronous parallel ELMOEA/D (APELMOEA/D). This section firstly introduces parallel EAs briefly, and then, we explain the detailed description of SPELMOEA/D and APELMOEA/D. We finally introduce two selection mechanisms for APELMOEA/D.
4.1 Parallel evolutionary algorithm
When applying EAs to realworld optimization problems, solution evaluations may take much computational time because of, for example, a requirement of physical simulation. Previous works have studied parallel EAs (PEAs) (Alba and Tomassini 2002; Alba et al. 2013) to speed up the optimization process of EAs by using multiple computing resources. In recent years, parallel MOEAs have been studied to reduce the computing time to approximate the Pareto front (Talbi 2019).
A masterslave parallelization is one of the most typical and straightforward approaches of PEAs. A masterslave PEA performs the main procedure of EAs like initialization, selection, variation, and replacement on a single master computing node. At the same time, many slave computing nodes execute fitness evaluations in parallel. Such a masterslave parallelization has been applied to solving realworld problems. For example, the work of Barbera et al. (2018) applied the massively parallel EA to the phylogenetic placement problem. The work of Strofylas et al. (2018) proposed a parallel differential evolution method for calibrating a secondorder macroscopic traffic flow.
Masterslave PEAs can be classified into two schemes, a synchronous evolution and an asynchronous one. Synchronous PEAs (SPEAs) is similar to generation based EAs, where the new population is generated after waiting for all solution evaluations. On the other hand, asynchronous PEAs (APEAs) is similar to steadystate EAs. A new solution is immediately generated whenever one solution evaluation completes without waiting for other solution evaluations. In general, an SPEA has a higher search capability than an APEA because an SPEA can utilize many evaluation information simultaneously when generating a new population. On the other hand, when the evaluation time of solutions differs from each other, i.e., the variance of the evaluation time is large, an APEA shows a higher search efficiency than in terms of the computational time (Harada and Takadama 2013, 2014; Scott and De Jong 2015a, b). This is because APEAs can reduce the idling time of slave nodes that quickly complete the evaluation tasks, unlike SPEAs need to wait for the slowest solution evaluations.
4.2 Synchronous parallel ELMOEA/D
Algorithm 3 shows a masterslave synchronous parallel ELMOEA/D (SPELMOEA/D). SPELMOEA/D is an extension of ELMOEA/D, and the underlined texts denote the difference from Algorithm 2. At the initialization procedure, all generated solutions are sent to slave nodes in Step 1–2, and their evaluations are firstly waited for in Step 1–3. Then, the remaining initialization procedure is executed. After evolving solutions on MOEA/DDE with an ELM surrogate, \(N_s\) promising solutions are selected, and they are sent to slave nodes in Step 5–1. SPELMOEA/D waits for all evaluations from slave nodes in Step 5–2 and updates the archive population and the training set. SPELMOEA/D repeats these procedures until the terminal condition is satisfied.
4.3 Asynchronous parallel ELMOEA/D
APELMOEA/D evaluates solutions in parallel as same as SPELMOEA/D but waits for only one solution evaluation every step. When one solution evaluation completes, it is appended to the archive population, and MOEA/DDE with the ELM surrogate model generates a new promising solution. The differences between APELMOEA/D and SPELMOEA/D are in Steps 4 and 5 in Algorithm 3. Concretely, these procedures are modified as Algorithm 4.
First, APELMOEA/D waits for only one evaluation from a slave node in Step 5’–2, unlike SPELMOEA/D waits for all \(N_s\) solution evaluations. In response to this, Steps 4’ and 5’ of APELMOEA/D are modified to address one solution. In Step 4–1 in Algorithm 3, SPELMOEA/D selects \(N_s\) promising solutions in each generation and actually evaluates them in parallel. On the other hand, APELMOEA/D selects only one promising solution in Step 4’–3. For this selection, APELMOEA/D needs to select an index \(k_s\) of the selector weight vector in Step 4’–2 instead of considering all \(N_s\) selector vectors. We take this into account in the next subsection. In addition to this, APELMOEA/D addresses only one solution in the update of the archive population \(P_a^{g+1}\) and the training set \(P_t^{g+1}\) in Steps 5’–3 and 5’4. Besides, because only one solution is selected for an actual evaluation every MOEA/DDE optimization step, the surrogate model is constructed every \(N_s\) actual evaluations in APELMOEA/D not to increase the number of the ELM training.
4.4 Promising solution selection of APELMOEA/D
When applying the asynchronous evolution scheme to ELMOEA/D, we need to determine how to select one promising solution from the population optimized by MOEA/DDE on the surrogate evaluation model. In particular, an index \(k_s\) should be chosen Step 4’ in Algorithm 4 whenever one solution evaluation completes. This paper explores simple two selection mechanisms of an index \(k_s\): the index order based selection and the random order selection.
4.4.1 Index order based selection:
A straightforward way to select an index \(k_s\) is to use the order of the index of the neighborhood vector set. This paper denotes APELMOEA/D with the promising selection using the order of the index of the neighborhood vector set as APELMOEA/DIO. APELMOEA/DIO selects a promising solution depending on the order of the index of the neighborhood vector set. APELMOEA/DIO initially set an index \(k_s\) to 1, and whenever a promising solution is selected, \(k_s\) is incremented as \(k_s\leftarrow k_s+1\). If \(k_s\) exceeds \(K_s\), which is the number of the neighborhood vector set, \(k_s\) is reset to 1.
4.4.2 Random order selection
The second way is to choose an index \(k_s\) every selection procedure randomly. This paper denotes this as APELMOEA/DRO. Whenever an index \(k_s\) is selected, it is uniformly chosen from 1 to \(N_s\) without considering the previous index, i.e., consecutive selection of the same index is allowed. Since an actual implementation of MOEA/D often uses a random permutation to select the aggregation vector during the optimization Fan et al. (2019); Nebro et al. (2015), the random order selection can be a valid strategy in APELMOEA/D.
5 Experiment
We conduct an experiment to compare three different parallel ELMOEA/D variants, i.e., SPELMOEA/D, APELMOEA/DIO, and APELMOEA/DRO, to clarify which parallelization scheme is suitable for ELMOEA/D. Additionally, we compare the sequential ELMOEA/D (i.e., run on a single CPU) to confirm the effectiveness of the parallelization. The following subsections explain the benchmark problems used in the experiment, the experimental setting, and the evaluation criteria.
5.1 Problem instance
We use wellknown multiobjective benchmarks, the ZDT test suite (Zitzler et al. 2000), the WFG test suite (Huband et al. 2005), and the DTLZ test suite (Deb et al. 2002). This experiment considers biobjective minimization for all test suites. The dimension of the decision variable D is as:

ZDT1–3: \(D=30\)

ZDT4 and 6: \(D=10\)

WFG1–9: \(D=24\)

DTLZ1–7: \(D=5\).
Additionally, we use four realworld problems provided by the black box optimization competition (Loshchilov and Glasmachers 2015). In particular, three are from the engineering design, a heat exchanger problem, a hydrodynamics problem, and a vibrating platform problem, while one is from the operational research, a facility placement problem. All of them are biobjective minimization, and \(D=16\) for the heat exchanger problem, \(D=6\) for the hydrodynamics problem, \(D=5\) for the vibrating platform problem, and \(D=20\) for the facility placement problem.
5.2 Experimental setting
We conduct the experiment on the pseudo (simulated) masterslave parallel computing environment that measures the computational time according to the model proposed in the work of Zăvoianu et al. (2015). In the experiment, we define the unit of the computing time as “simulation time”. In particular, we define one unit simulation time as the time to complete the sequential tasks on the master node, i.e., the ELM learning and the MOEA/D optimization with the surrogate evaluation.
We test two settings of evaluation time of solutions. One determines the evaluation time of any solutions by the normal distribution of \({\mathcal {N}}(t_p, c_v\times t_p)\) simulation time on the distributed slave nodes. \(t_p\) (\(\gg 1\)) is the mean evaluation time, while the random variance is added according to the normal distribution with the standard deviation \(c_v\times t_p\), where \(c_v\,(c_v\ge 0)\) is a parameter to determine the variance of the evaluation time. In this setting, the evaluation time does not depend on the feature of evaluated solutions. We call this evaluation time normallydistributed evaluation time.
Another setting defines the evaluation time of solutions correlates to the objective function value. In particular, the evaluation time is calculated as:
where \(f_1({\mathbf {x}})\) indicates the first objective function value of a solution \({\mathbf {x}}\), \(t_p\) (\(\gg 1\)) is the base evaluation time when \(f_1({\mathbf {x}})=1\), while \(c_g\) denotes the parameter to decides the gradient of the evaluation time. If \(c_g>0\), the evaluation time positively correlates to the first objective function value, i.e., the greater the objective function value, the longer the evaluation time. If \(c_g<0\), on the other hand, the evaluation time negatively correlates to the first objective function value, i.e., the greater the objective function value, the shorter the evaluation time. Since the evaluation time depends on the objective function value, the search direction of the APELMOEA/D variants has the possibility to biased toward the search space where solutions quickly complete their evaluations (Scott and De Jong 2015a).
This experiment uses the parameters shown in Table 1, most of which are the same as the original work (Pavelski et al. 2016). Notably, we use the MOEA/DDE parameter settings that the original MOEA/DDE work recommended (Li and Zhang 2009). Ten slave nodes are used for solution evaluations, which means \(N_s=10\) selected promising solutions can be evaluated simultaneously. To clarify how the variance of the evaluation time on the parallel computing environment affects the search performance, we consider different magnitudes of the variance \(c_v=\{0.02, 0.05, 0.10, 0.20\}\) in the normallydistributed evaluation time. On the other hand, in the fitnesscorrelated evaluation time, we use the different gradient parameters \(c_g=\{\pm 0.2, \pm 0.5\}\) to clarify the influence of the evaluation time gradient.
This paper uses the hypervolume (HV) indicator (Zitzler and Thiele 1998) to assess the quality of the achieved solution set by the methods. The HV indicator can evaluate both the convergence and the diversity of obtained solutions. For the benchmark problems, the scale of HV value differs in each benchmark problem. To clearly display the results in the table, we normalize the HV value by the ideal HV value calculated from the true Pareto front for each benchmark problem. The normalized HV value \({\widetilde{HV}}\) is defined as follows:
where HV indicates the HV value obtained by each method, while \(HV_{ideal}\) indicates the ideal HV value. On the other hand, for the realworld problems, since the ideal HV value is unknown, we use raw HV values.
The reference point, which is the extreme point of the area to be covered by acquired solutions, is required to calculate the HV value. We use the following reference points, which all are the same as the original work:

ZDT1–3, 6, WFG1–9, DTLZ2, 4–6: \(\{5.0, 5.0\}\)

ZDT4, DTLZ1, 3, 7: \(\{100.0, 100.0\}\)
For the realworld problems, on the other hand, the reference point is set to \(\{1.0, 1.0\}\), which is used in (Loshchilov and Glasmachers 2015).
We assess three methods from the viewpoint of the number of actual evaluations and the elapsed simulation time. The HV values of 30 independent trials are calculated for the maximum number of actual evaluations and the maximum elapsed simulation time. We set the maximum number of the actual evaluations as 2000 for all ZDT instances, 1000 for all WFG instances, and 200 for all DTLZ instances. For the realworld problems, we set the maximum number of the actual evaluations as 500 for the heat exchanger problem, 200 for the vibrating platform problem, and 1000 for the hydrodynamics and the facility placement problems. The maximum elapsed simulation time, on the other hand, is set to \(1.5\times 10^5\) simulation time for the ZDT instances, \(7.0\times 10^4\) simulation time for the WFG instances, and \(1.0\times 10^4\) simulation time for the DTLZ instances. For the realworld problems, we set the maximum elapsed simulation time to \(3.0\times 10^4\) simulation time for the heat exchanger problem, \(1.2\times 10^4\) simulation time for the vibrating platform problem, and \(7.0\times 10^4\) simulation time for the hydrodynamics and the facility placement problems. These maximum simulation times are chosen because all asynchronous schemes complete their executions before these computational times, though the synchronous one needs more computational time to complete its execution.
6 Results on benchmark problems with the normallydistributed evaluation time
This section shows the experimental results on the benchmark problems with the use of the normallydistributed evaluation time. First, we compare the HV values of the parallel ELMOEA/D variants at the maximum number of the actual evaluations. Then we compare them from the viewpoint of the elapsed simulation time.
6.1 Hypervolume at the maximum number of the actual evaluations
Table 2 shows the mean normalized HV values and their standard deviations of the competitive methods at the maximum number of the actual evaluations. This table does not include the sequential ELMOEA/D because its result is the same as SPELMOEA/D at the same number of the actual evaluations. For each benchmark and each \(c_v\) value, the maximum mean values are shown in the bold style. We conduct a statistic test with the Mann–Whitney U test with a 5\(\%\) significance level. If a method is significantly better than another, it is specified as the superscript character in the “Mean” column. “1” indicates that a corresponding method is significantly better than SPELMOEA/D, and similarly “2” indicates APELMOEA/DIO, and “3” indicates APELMOEA/DRO. These results indicate that APELMOEA/DIO has the highest mean HV values in most cases, although the difference is not significant for many benchmarks. On the other hand, SPELMOEA/D significantly outperforms the APELMOEA/D variants in ZDT3 and WFG9. There is no trend in the variance of the evaluation time.
Table 3 shows an aggregation of the statistic test of the pairwise comparison. Each group of three columns indicates the compared two methods, while each column shows the count of the results of the statistic test. The “1” column indicates the first method is significantly better than the second one, while the “2” column indicates the second method is significantly better than the first one. If there is no significant difference, “\(\approx\)” is counted. “Sync.” indicates SPELMOEA/D, “IO” indicates APELMOEA/DIO, and “RO” indicates APELMOEA/DRO. This result firstly indicates that APELMOEA/DIO is significantly better than SPELMOEA/D in 22 experimental cases of total 84 cases, while SPELMOEA/D wins in 5 cases. Comparing SPELMOEA/D and APELMOEA/D, APELMOEA/DRO is significantly better than SPELMOEA/D in 13 experimental cases, while SPELMOEA/D wins in 6 cases. Comparing APELMOEA/DIO and APELMOEA/DRO, there is no significant difference in most cases, but in 10 cases, APELMOEA/DIO is significantly better than APELMOEA/DRO, while APELMOEA/DRO wins in only 2 cases.
These results reveal that the integration of the surrogateassisted EA with the asynchronous evolution scheme improves the search capability, even taking no account of the computational time.
6.2 Hypervolume at the maximum elapsed simulation time
The mean normalized HV values at the maximum elapsed simulation time are shown in Table 4. The maximum mean values are shown in the bold style, and the results of the statistic test are specified as the superscript character. Both of them are the same as Table 2. Table 5 aggregates the statistic test of the pairwise comparison as same as Table 3.
First, let us compare the results of the parallel and the sequential ELMOEA/Ds. It is confirmed that the parallel ELMOEA/Ds significantly outperforms the sequential ELMOEA/D for all benchmarks and all \(c_v\) values. From this, the parallelization of ELMOEA/D contributes to reducing the computational time for the optimization.
Then, let us compare the parallelization schemes, i.e., the synchronous and the asynchronous parallel ELMOEA/Ds. We can confirm a similar tendency to Table 2 from these results. The HV value of APELMOEA/DIO is significantly better than or equal to SPELMOEA/D in all benchmark problems. On the other hand, APELMOEA/DRO is also significantly better than or equal to SPELMOEA/D especially when the variance of the evaluation time is large. In particular, APELMOEA/DIO performs the highest mean HV values in most experimental cases. The results in Table 5 indicate that APELMOEA/DIO is significantly better than SPELMOEA/D in 30 experimental cases of total of 84 cases. In comparison, APELMOEA/DRO is significantly better than SPELMOEA/D in 15 experimental cases. Comparing APELMOEA/DIO and APELMOEA/DRO, APELMOEA/DIO is significantly better in 15 experimental cases.
Next, we compare the difference in performance as the variance of the evaluation time changes. When the variance of the evaluation time is small, the difference between SPELMOEA/D and the APELMOEA/D variants is small. In particular, SPELMOEA/D outperforms APELMOEA/DRO in a few benchmarks. In contrast, when the variance of the evaluation time is large, the cases of APELMOEA/Ds outperforming SPELMOEA/D increase. This result shows that as the variance of the evaluation time increases, the APELMOEA/D variants shows higher performance.
From these results, it is revealed that the asynchronous evolution scheme also improves the search efficiency in terms of computational time. However, in some benchmark problems, there is no significant difference between SPELMOEA/D and the APELMOEA/D variants even though the evaluation time variance is large. The next subsection discusses the details of the transition of the HV values over the elapsed simulation time. In particular, we focus on ZDT4, ZDT6, WFG8, and DTLZ4, because there is no significant difference among SPELMOEA/D and the APELMOEA/D variants in these problems.
6.3 Transition of HV over the elapsed simulation time
Figures 1–4 show the transition of the mean normalized HV values over the elapsed simulation time. These figures include two parts: the top part shows the mean normalized HV value transition, while the bottom part shows the results of the statistic test at every simulation time. In the top part, the horizontal axis shows the elapsed simulation time. The vertical axis shows the normalized mean HV value. The solid line shows SPELMOEA/D, the dashed line shows APELMOEA/DIO, and the dashdot line shows APELMOEA/DRO. In the bottom part, on the other hand, the horizontal axis is common, while the vertical axis indicates the pair of the comparison, where “1”, “2”, and “3” indicate SPELMOEA/D, APELMOEA/DIO, and APELMOEA/DRO, respectively. The “+” mark indicates the first method of the pair significantly outperforms the second one at the corresponding elapsed simulation time. In contrast, the “” mark indicates the second one outperforms the first one. For example, if the yaxis of the bottom part is “1 v 2” and the mark is “”, APELMOEA/DIO significantly outperforms SPELMOEA/D at the corresponding elapsed simulation time.
There is no significant difference over time between any pairs of the parallel ELMOEA/D variants in WFG8 and DTLZ4. On the other hand, in ZDT4, the significant difference between APELMOEA/DIO and SPELMOEA/D, and that between APELMOEA/DRO and SPELMOEA/D can be found in the early and the middle term of the evolution when the variance of the evaluation time is low, though there is no significant difference in the final part of the evolution. This tendency can be found in the high variance in ZDT6, where APELMOEA/DRO significantly outperforms SPELMOEA/D in the middle term of the evolution. This result indicates that APELMOEA/DIO and APELMOEA/DRO reach to the final HV value quicker than SPELMOEA/D, even though the final HV value is almost the same.
These results reveal that the convergence speed of the APELMOEA/D variants is faster than SPELMOEA/D even though they finally achieve a similar quality of solutions.
7 Results on benchmark problems with the fitnesscorrelated evaluation time
This section shows the experimental results on the benchmark problems with the fitnesscorrelated evaluation time. As same as the previous section, we first compare the HV values of the parallel ELMOEA/D variants at the maximum number of the actual evaluations, then we compare them from the viewpoint of the elapsed simulation time.
Note that in this experiment, we exclude the results on the benchmarks of DTLZ1 and DTLZ3. This is because the range of the first objective function value of these benchmarks is vast (the maximum value is about 100.0), and the evaluation time calculated by Eq. (12) is not appropriate.
7.1 Hypervolume at the maximum number of the actual evaluations
Tables 6 and 7 show the mean normalized HV values and their standard deviations of the competitive methods at the maximum number of the actual evaluations with the fitnesscorrelated evaluation time. Table 6 shows the result of the positively correlated evaluation time. In contrast, Table 7 shows the result of the negatively correlated evaluation time. These tables do not include the sequential ELMOEA/D for the same reason as Section 6.1. Tables 8 and 9 show aggregations of the statistic test of the pairwise comparison of the results in Tables 6 and 7. The notations in these tables are the same as in Sect. 6.1.
The experimental results at the maximum number of the actual evaluations show that SPELMOEA/D and the APELMOEA/D variants are less significantly different. In particular, compared with the previous section, the number of cases in which the APELMOEA/D variants are significantly better than SPELMOEA/D decreases. At the same time, SPELMOEA/D significantly outperforms the APELMOEA/D variants in some benchmarks. For example, SPELMOEA/D outperforms in the benchmarks of ZDT6, wFG9, and DTLZ3 when the evaluation time positively correlates to the first objective function value, while in ZDT3 when the evaluation time negatively correlates. However, APELMOEA/DIO comprehensively shows the highest average HV values in many of the benchmarks, and it is statistically equivalent to or better than the other two methods. These results indicate that although APELMOEA/DIO is affected by the evaluation time bias, it still performs high search capability.
7.2 Hypervolume at the maximum elapsed simulation time
Tables 10 and 11 show the mean normalized HV values and their standard deviations of the competitive methods at the maximum elapsed simulation time with the fitnesscorrelated evaluation time. Table 10 shows the result of the positively correlated evaluation time. In contrast, Table 11 shows the result of the negatively correlated evaluation time. Tables 12 and 13 show aggregations of the statistic test of the pairwise comparison of the results in Tables 10 and 11. The notations in these tables are the same as in Sect. 6.2.
First, comparing the sequential ELMOEA/D and the parallel ELMOEA/Ds, it is shown that the parallel variants have significantly better performance than the sequential method in all benchmarks and all evaluation time setting. This result shows that the parallelization of ELMOEA/D contributes to shortening the computational time of the optimization even when the evaluation time is biased.
Next, comparing the three parallel ELMOEA/Ds, the APELMOEA/D variants are significantly better than SPELMOEA/D in several benchmarks. However, the number of them is reduced compared to the case of the normallydistributed evaluation time. This is because the search direction of APELMOEA/D is biased to the solution region with a short evaluation time if the evaluation time correlates to the objective function value. This causes to decrease the search capability of APELMOEA/D. However, in terms of the computation time, the waiting time of SPELMOEA/D increases because the fitnesscorrelated evaluation time induces the evaluation time variance depending on the objective function value, thus its computational efficiency decreases. Therefore, although SPELMOEA/D is not affected by the evaluation time bias in search, it cannot outperform the APELMOEA/D variants at the maximum elapsed evaluation time.
In this paper, we use ten slave nodes for the evaluation. However, APEAs are less affected by the evaluation time bias when the number of the slave nodes is small. Therefore, the performance deterioration of APELMOEA/Ds was suppressed in our experiment. However, if we use more slave nodes, it may deteriorate the performance of the APELMOEA/D variants.
8 Results on realworld problems
This section shows the experimental results on realworld problems. As same as the previous section, we first compare the HV values of the parallel ELMOEA/D variants at the maximum number of the actual evaluations, then we compare them from the viewpoint of the elapsed simulation time.
8.1 Hypervolume at the maximum number of the actual evaluations
Table 14 shows the mean HV values and their standard deviation of the competitive methods at the maximum number of the actual evaluations with the normallydistributed evaluation time. We represent each problem as an abbreviation. “HX” denotes the heat exchanger problem, “HD” denotes the hydrodynamics problem, “VP” denotes the vibrating platform problem, and “FP” denotes the facility placement problem. The best results are shown in the bold style. As same as the results on the benchmark problems, we conduct a statistic test with the Mann–Whitney U test with a 5% significance level, and its result is shown in this table as the same style in Table 2.
These results show that, in the heat exchanger problem and the facility placement problem, the APELMOEA/D variants show better performance than SPELMOEA/D. Especially in some cases, the asynchronous variants significantly outperform the synchronous one. On the other hand, in the hydrodynamics problem and the vibrating platform problem, APELMOEA/DRO shows worse performance than the other two methods. In particular, SPELMOEA/D shows the best mean HV values in the hydrodynamics problem for all evaluation time variances.
Table 15 shows the mean HV values and their standard deviation of the competitive methods at the maximum number of the actual evaluations with the fitnesscorrelated evaluation time. The notation in this table is same as Table 14. Even when the evaluation time correlates to the objective function value, the tendency is almost the same as the results with the normallydistributed evaluation time. In particular, in the heat exchanger problem and the facility placement problem, the APELMOEA/D variants show better performance than SPELMOEA/D In the hydrodynamics problem and the vibrating platform problem, on the other hand, APELMOEA/DRO is worse than the other two methods.
These results indicate that no clear difference is found in the realworld problems at the maximum number of the actual evaluations. However, APELMOEA/DIO shows averagely better performance than the other two methods, i.e., APELMOEA/DIO is never significantly worse.
8.2 Hypervolume at the maximum elapsed simulation time
Table 16 shows the mean HV values and their standard deviation of the competitive methods at the maximum elapsed simulation time with the normallydistributed evaluation time. On the other hand, Table 17 shows the result with the fitnesscorrelated evaluation time. The notations of these tables are the same as Tables 14 and 15. These results include the sequential ELMOEA/D.
First, from the result of the sequential ELMOEA/D, it is shown that the parallel variants all significantly outperform the sequential ELMOEA/D in all problems and both of the evaluation time settings. This result indicates that the parallelization of ELMOEA/D is even useful in realworld problems.
From the comparison of the parallel variants of ELMOEA/D in the normallydistributed evaluation time, no significant difference between three parallel ELMOEA/Ds is found in the heat exchanger and the vibrating platform problems. Note that only in \(c_v=0.2\) in the vibrating platform problem, APELMOEA/DIO significantly outperforms SPELMOEA/D, where the variance of the evaluation time is enormous. On the other hand, in the facility placement problem, the asynchronous variants significantly outperform SPELMOEA/D regardless of the variance of the evaluation time. In the hydrodynamics problem, SPELMOEA/D significantly outperforms the APELMOEA/D variants when the variance of the evaluation time is low, i.e., \(c_v=\{0.02, 0.05\}\). Meanwhile, SPELMOEA/D decreases its performance as the variance of the evaluation time increases even in the hydrodynamics problem. On the other hand, the APELMOEA/D variants keep their performance as the variance increases. Especially, APELMOEA/DIO shows the highest mean HV values in most cases and significantly outperforms the other two methods in some of the cases.
Finally, we compare the parallel variants of ELMOEA/D in the fitnesscorrelated evaluation. First, there is no significant difference between SPELMOEA/D and APELMOEA/DIO in all cases except for the facility placement problem with \(c_g=0.5\). On the other hand, although APELMOEA/DRO shows better performance only in the heat exchanger problem, it is inferior to APELMOEA/DIO in the other problems. In total, APELMOEA/DIO shows the highest mean HV values in most cases and significantly outperforms the other two methods in some of the cases.
These results reveal that APELMEOA/DIO shows high search performance in all realworld problems and all evaluation time settings. Although SPELMOEA/D shows the high performance when the variance of the evaluation time is low, its performance decreases as the variance increases.
9 Conclusion
This paper introduced a parallel evaluation scheme into a surrogateassisted multiobjective evolutionary algorithm (MOEA) for further reducing the optimization time. As a surrogateassisted MOEA, this paper used extreme learning assisted MOEA/D (ELMOEA/D) that uses an extreme learning machine (ELM) as a surrogate model and MOEA/DDE as an optimizer. On the other hand, we considered two different schemes of the parallelization, i.e., a synchronous evolution scheme and an asynchronous evolution scheme, and We proposed the parallelization methods of ELMOEA/D. One is a synchronous parallel ELMOEA/D (SPELMOEA/D) that generates new promising solutions after waiting all solution evaluations. On the other hand, another is an asynchronous parallel ELMOEA/D (APELMOEA/D) that generates a new solution immediately after waiting for one solution evaluation. For APELMOEA/D, we proposed two selection scheme of a promising solution. One is the index order selection that selects a promising solution depending on the order of the index of the neighborhood vector set. Another is the random order selection that randomly select the index of the neighborhood vector set.
We conducted the experiment on multiobjective optimization problems to compare the sequential ELMOEA/D, SPELMOEA/D, and the APELMOEA/D variants. The experiment used multiobjective benchmarks and realworld problems. The experiment was conducted on the pseudo (simulated) parallel computing environment and considered two settings of evaluation time of solutions; the normallydistributed evaluation time and the fitnesscorrelated evaluation time.
The experimental results first revealed that the parallelization of ELMOEA/D significantly reduces the computational time compared with the sequential one. From the comparison of the synchronous and the asynchronous parallelization schemes, the asynchronous evolution scheme outperforms the synchronous one, in particular, the asynchronous one converges faster even if the variance of the evaluation time is enormous. Even though the evaluation time is biased, the asynchronous evolution scheme has an advantage to the synchronous one. This is because the synchronous evolution scheme still decreases its computational efficiency due to the difference of the evaluation time. Moreover, although two promising solution selection strategies are not significantly different, the promising solution selection based on the index order showed the best performance in most test problems.
In the future, we will conduct experiments using not only ELMOEA/D but also other surrogateassisted EAs. For example, the method proposed in (Habib et al. 2019) proposes a novel surrogateassisted MOEA/D that uses multiple surrogate models. We can apply the parallelization scheme we have proposed to such MOEA/Dbased approaches. For further improvement of the proposed method, we will introduce a preferencebased EA like RVEA (Cheng et al. 2016) to APELMOEA/D instead of MOEA/D. Since APELMOEA/D selects only a single promising solution, we can improve the quality of the promising solution by concentrating the search only around the target weight vector.
References
Akinsolu MO, Liu B, Grout V, Lazaridis PI, Mognaschi ME, Barba PD (2019) A parallel surrogate model assisted evolutionary algorithm for electromagnetic design optimization. IEEE Trans Emerg Top Comput Intell 3(2):93–105
Alba E, Tomassini M (2002) Parallelism and evolutionary algorithms. IEEE Trans Evol Comput 6(5):443–462. https://doi.org/10.1109/TEVC.2002.800880
Alba E, Luque G, Nesmachnow S (2013) Parallel metaheuristics: recent advances and new trends. Int Trans Oper Res 20(1):1–48. https://doi.org/10.1111/j.14753995.2012.00862.x
Barbera P, Kozlov AM, Czech L, Morel B, Darriba D, Flouri T, Stamatakis A (2018) EPAng: massively parallel evolutionary placement of genetic sequences. Syst Biol 68(2):365–369. https://doi.org/10.1093/sysbio/syy054
Cheng R, Jin Y, Olhofer M, Sendhoff B (2016) A reference vector guided evolutionary algorithm for manyobjective optimization. IEEE Trans Evol Comput 20(5):773–791. https://doi.org/10.1109/TEVC.2016.2519378
Das S, Suganthan PN (2011) Differential evolution: A survey of the stateoftheart. IEEE Trans Evol Comput 15(1):4–31. https://doi.org/10.1109/TEVC.2010.2059031
Davis JBA, Shayeghi A, Horswell SL, Johnston RL (2015) The Birmingham parallel genetic algorithm and its application to the direct dft global optimisation of irn (n = 10–20) clusters. Nanoscale 7:14032–14038. https://doi.org/10.1039/C5NR03774C
Deb K (2001) Multiobjective optimization using evolutionary algorithms. Wiley, New York
Deb K, Goyal M (1996) A combined genetic adaptive search (GeneAS) for engineering design. Comput Sci Inform 26:30–45
Deb K, Thiele L, Laumanns M, Zitzler E (2002) Scalable multiobjective optimization test problems. In: Proceedings of the 2002 congress on evolutionary computation. CEC’02 (Cat. No.02TH8600), 1:825–830 vol.1, https://doi.org/10.1109/CEC.2002.1007032
Devos O, Downey G, Duponchel L (2014) Simultaneous data preprocessing and svm classification model selection based on a parallel genetic algorithm applied to spectroscopic data of olive oils. Food Chem 148:124–130. https://doi.org/10.1016/j.foodchem.2013.10.020
Fan Z, Li W, Cai X, Huang H, Fang Y, You Y, Mo J, Wei C, Goodman E (2019) An improved epsilon constrainthandling method in moea/d for cmops with large infeasible regions. Soft Comput 23(23):12491–12510. https://doi.org/10.1007/s0050001903794x
Habib A, Singh HK, Chugh T, Ray T, Miettinen K (2019) A multiple surrogate assisted decompositionbased evolutionary algorithm for expensive multi/manyobjective optimization. IEEE Trans Evol Comput 23(6):1000–1014. https://doi.org/10.1109/TEVC.2019.2899030
Haftka RT, Villanueva D, Chaudhuri A (2016) Parallel surrogateassisted global optimization with expensive functions—a survey. Struct Multidiscip Optim 54(1):3–13. https://doi.org/10.1007/s0015801614323
Harada T, Takadama K (2013) Asynchronous evaluation based genetic programming: comparison of asynchronous and synchronous evaluation and its analysis. In: Krawiec K, Moraglio A, Hu T, EtanerUyar A, Hu B (eds) Genetic programming, vol 7831. Lecture notes in computer science. Springer, Berlin, pp 241–252. https://doi.org/10.1007/9783642372070_21
Harada T, Takadama K (2014) Asynchronously evolving solutions with excessively different evaluation time by referencebased evaluation. In: Proceedings of the 2014 annual conference on genetic and evolutionary computation, ACM, New York, USA, GECCO ’14, pp 911–918, https://doi.org/10.1145/2576768.2598330
Huang GB (2015) What are extreme learning machines? filling the gap between frank Rosenblatt’s dream and John von Neumann’s puzzle. Cogn Comput 7(3):263–278
Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE International joint conference on neural networks (IEEE Cat. No.04CH37541), vol 2, pp 985–990 vol.2, https://doi.org/10.1109/IJCNN.2004.1380068
Huband S, Barone L, While L, Hingston P (2005) A scalable multiobjective test problem toolkit. In: Coello Coello CA, Hernández Aguirre A, Zitzler E (eds) Evolutionary multicriterion optimization. Springer, Berlin, pp 280–295
Jin Y (2011) Surrogateassisted evolutionary computation: recent advances and future challenges. Swarm Evolut Comput 1(2):61–70. https://doi.org/10.1016/j.swevo.2011.05.001
Knowles J (2006) ParEGO: a hybrid algorithm with online landscape approximation for expensive multiobjective optimization problems. IEEE Trans Evol Comput 10(1):50–66. https://doi.org/10.1109/TEVC.2005.851274
Li H, Zhang Q (2009) Multiobjective optimization problems with complicated pareto sets, MOEA/D and NSGAII. IEEE Trans Evol Comput 13(2):284–302. https://doi.org/10.1109/TEVC.2008.925798
Liu B, Akinsolu MO, Ali N, AbdAlhameed R (2019) Efficient global optimisation of microwave antennas based on a parallel surrogate modelassisted evolutionary algorithm. IET Microw Antennas Propag 13(2):149–155
Loshchilov I, Glasmachers T (2015) Black box optimization competition. https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/index.html, Online
McKay MD, Beckman RJ, Conover WJ (1979) A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245
Nebro AJ, Durillo JJ, Vergne M (2015) Redesigning the jmetal multiobjective optimization framework. In: Proceedings of the companion publication of the 2015 annual conference on genetic and evolutionary computation, Association for computing machinery, New York, NY, USA, GECCO Companion ’15, p 1093–1100, https://doi.org/10.1145/2739482.2768462
Obayashi S, Jeong S, Shimoyama K, Chiba K, Morino H (2010) Multiobjective design exploration and its applications. Int J Aeronaut Space Sci 4(4):247–265. https://doi.org/10.5139/IJASS.2010.11.4.247
Oyama A, Kohira T, Kemmotsu H, Tatsukawa T, Watanabe T (2017) Simultaneous structure design optimization of multiple car models using the k computer. In: 2017 IEEE symposium series on computational intelligence (SSCI), pp 1–4, https://doi.org/10.1109/SSCI.2017.8285350
Pavelski LM, Delgado MR, Almeida CP, Gonçalves RA, Venske SM (2014) ELMOEA/DDE: Extreme learning surrogate models in multiobjective optimization based on decomposition and differential evolution. In: 2014 Brazilian conference on intelligent systems, pp 318–323, https://doi.org/10.1109/BRACIS.2014.64
Pavelski LM, Delgado MR, Almeida CP, Gonçalves RA, Venske SM (2016) Extreme learning surrogate models in multiobjective optimization based on decomposition. Neurocomputing 180:55–67. https://doi.org/10.1016/j.neucom.2015.09.111
Roberge V, Tarbouchi M, Okou F (2014) Strategies to accelerate harmonic minimization in multilevel inverters using a parallel genetic algorithm on graphical processing unit. IEEE Trans Power Electron 29(10):5087–5090. https://doi.org/10.1109/TPEL.2014.2311737
Scott EO, De Jong KA (2015a) Evaluationtime bias in asynchronous evolutionary algorithms. In: Proceedings of the companion publication of the 2015 annual conference on genetic and evolutionary computation, ACM, New York, NY, USA, GECCO Companion ’15, pp 1209–1212, https://doi.org/10.1145/2739482.2768482
Scott EO, De Jong KA (2015b) Understanding simple asynchronous evolutionary algorithms. In: Proceedings of the 2015 ACM conference on foundations of genetic algorithms XIII, ACM, New York, NY, USA, FOGA ’15, pp 85–98, https://doi.org/10.1145/2725494.2725509
Shayeghi A, Gotz D, Davis JBA, Schafer R, Johnston RL (2015) Poolbcga: a parallelised generationfree genetic algorithm for the ab initio global optimisation of nanoalloy clusters. Phys Chem Chem Phys 17:2104–2112. https://doi.org/10.1039/C4CP04323E
Soufan O, Kleftogiannis D, Kalnis P, Bajic VB (2015) Dwfs: a wrapper feature selection tool based on a parallel genetic algorithm. PLoS ONE 10(2):1–23. https://doi.org/10.1371/journal.pone.0117988
Storn R, Price K (1997) Differential evolution—a simple and efficient Heuristic for global optimization over continuous spaces. J Global Optim 11(4):341–359. https://doi.org/10.1023/A:1008202821328
Strofylas G, Porfyri K, Nikolos I, Delis A, Papageorgiou M (2018) Using synchronous and asynchronous parallel differential evolution for calibrating a secondorder traffic flow model. Adv Eng Softw 125:1–18. https://doi.org/10.1016/j.advengsoft.2018.08.011
Talbi EG (2019) A unified view of parallel multiobjective evolutionary algorithms. J Parallel Distrib Comput 133:349–358. https://doi.org/10.1016/j.jpdc.2018.04.012
Wang J, Clark SC, Liu E, Frazier PI (2016) Parallel Bayesian global optimization of expensive functions. arXiv:1602.05149
Zapotecas Martínez S, Coello Coello CA (2013) MOEA/D assisted by RBF networks for expensive multiobjective optimization problems. In: Proceedings of the 15th annual conference on genetic and evolutionary computation, ACM, New York, NY, USA, GECCO ’13, pp 1405–1412, https://doi.org/10.1145/2463372.2465805
Zhang Q, Li H (2007) MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans Evol Comput 11(6):712–731. https://doi.org/10.1109/TEVC.2007.892759
Zitzler E, Thiele L (1998) Multiobjective optimization using evolutionary algorithms—a comparative case study. In: Eiben AE, Bäck T, Schoenauer M, Schwefel HP (eds) Parallel problem solving from nature—PPSN V. Springer, Berlin, pp 292–301
Zitzler E, Deb K, Thiele L (2000) Comparison of multiobjective evolutionary algorithms: empirical results. Evol Comput 8(2):173–195
Zăvoianu AC, Lughofer E, Koppelstätter W, Weidenholzer G, Amrhein W, Klement EP (2015) Performance comparison of generational and steadystate asynchronous multiobjective evolutionary algorithms for computationallyintensive problems. KnowlBased Syst 87(C):47–60. https://doi.org/10.1016/j.knosys.2015.05.029
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work was supported by Japan Society for the Promotion of Science GrantinAid for Young Scientists Grant Number JP19K20362.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Harada, T., Kaidan, M. & Thawonmas, R. Comparison of synchronous and asynchronous parallelization of extreme surrogateassisted multiobjective evolutionary algorithm. Nat Comput 21, 187–217 (2022). https://doi.org/10.1007/s11047020098062
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11047020098062