Effectiveness and efficiency of nondominated sorting for evolutionary multi and manyobjective optimization
 907 Downloads
 2 Citations
Abstract
Since nondominated sorting was first adopted in NSGA in 1995, most evolutionary algorithms have employed nondominated sorting as one of the major criteria in their environmental selection for solving multi and manyobjective optimization problems. In this paper, we focus on analyzing the effectiveness and efficiency of nondominated sorting in multi and manyobjective evolutionary algorithms. The effectiveness of nondominated sorting is verified by considering two popular evolutionary algorithms, NSGAII and KnEA, which were designed for solving multi and manyobjective optimization problems, respectively. The efficiency of nondominated sorting is evaluated by comparing several stateoftheart nondominated sorting algorithms for multi and manyobjective optimization problems. These results provide important insights to adopt nondominated sorting in developing novel multi and manyobjective evolutionary algorithms.
Keywords
Nondominated sorting Evolutionary algorithm Multiobjective optimization Manyobjective optimizationIntroduction
Multiobjective optimization problems (MOPs) refer to those consisting of multiple contradictory objectives to be optimized simultaneously, which widely exist in realworld applications [1, 2, 3, 4, 5]. MOPs with more than three objectives are also called manyobjective optimization problems (MaOPs) [6]. Due to the fact that the objectives are in conflict with each other, there usually exists a set of tradeoff solutions instead of one single optimal solution for an MOP. In the past 2 decades, evolutionary algorithms have demonstrated the superiority in solving MOPs, and a large number of multiobjective evolutionary algorithms (MOEAs) have been developed, such as NSGAII [7], SPEA2 [8], PESAII [9], IBEA [10], and MOEA/D [11].
For solving MOPs, one of the most important problems that should be addressed is how to distinguish the quality of solutions consisting of multiple objective values. To this end, Goldberg [12] suggested the use of the Pareto dominance [13] to sort a set of solutions for MOPs, and the sorting procedure is called nondominated sorting. Briefly, the nondominated sorting aims to divide a solution set into a number of disjoint subsets or ranks, by means of comparing their values of the same objective. After nondominated sorting, solutions in the same rank are viewed equally important, and solutions in a smaller rank are better than those in a larger rank. Since NSGA [14] first adopted nondominated sorting in 1995, it has become a widely adopted strategy in MOEAs. For nondominated sorting, the effectiveness and efficiency are two important issues that have been concerned in MOEAs.
For MOPs, the effectiveness of nondominated sorting has long been recognized and most of the existing MOEAs adopted this strategy, e.g., NSGAII [7], PESAII [9], GDE3 [15], SMPSO [16], EAGMOEA/D [17], MOEA/IGDNS [18], etc. In the existing MOEAs, nondominated sorting is mainly performed in environmental selection, where solutions are divided into several ranks by nondominated sorting and those in the same rank are distinguished by additional criteria, such as crowding distance in NSGAII, GDE3, SMPSO, and EAGMOEA/D, the regionbased metric in PESAII, and the enhanced IGD in MOEA/IGDNS. As for MaOPs, nondominated sorting has also been favored by researchers in developing MOEAs despite that its effectiveness considerably deteriorates on MaOPs due to the Pareto dominance resistance phenomenon [19]. Some representative MOEAs that directly adopted nondominated sorting to handle MaOPs include GrEA [20], NSGAIII [21], KnEA [22], and LMEA [23]. There are also some recent works reported that nondominated sorting is also an effective strategy to which did not use nondominated sorting in their original versions, e.g., decompositionbased MOEAs with dominance, MOEA/DD [24], and BCEMOEA/D [25].
The efficiency of nondominated sorting is another issue in MOEAs, since it is often criticized due to the high computational cost. Taking the NSGAII as an example, nondominated sorting consumes more than 70% of the runtime in NSGAII when a population size of 1000 and a maximum generation of 500 are adopted to solve a 2objective DTLZ1. The computational cost will become much higher for larger population size or/and larger number of objectives. To address this issue, a lot of nondominated sorting algorithms have been developed to improve the efficiency of nondominated sorting, e.g., Jensen’s sort [26], nondominated ranking approach [27], deductive sort [28], and efficient nondominated sort (ENS) [29]. There are also some nondominated sorting methods specially tailored for solving MaOPs, such as corner sort [30], TENS [23], and AENS [31].
Despite that the effectiveness and efficiency of nondominated sorting were widely concerned in the past years, a little work has been reported to analyze them in MOEAs, especially for solving MaOPs. In this paper, we empirically investigate the effectiveness and efficiency of nondominated sorting in MOEAs for both MOPs and MaOPs. We verify the effectiveness of nondominated sorting by considering two popular MOEAs developed for solving MOPs and MaOPs, NSGAII and KnEA. Some rectifications to enhance the effectiveness of nondominated sorting for manyobjective optimization are also discussed. The efficiency of nondominated sorting is evaluated by comparing eight existing nondominated sorting algorithms in solving MOPs and MaOPs. These results will be helpful for researchers to develop new nondominated sortingbased MOEAs which are both effective and efficient.
The rest of the paper is organized as follows. In “Basic concepts of multiobjective optimization and nondominated sorting” section, some basic concepts of multiobjective optimization and nondominated sorting are given. In “Effectiveness of nondominated sorting for multi and manyobjective optimization” section, the effectiveness of nondominated sorting is analyzed with some discussions on the existing rectifications for manyobjective optimization. Afterwards, in “Efficiency of nondominated sorting for multi and manyobjective optimization” section, the efficiency of nondominated sorting is investigated by comparing several nondominated sorting methods in dealing with MOPs and MaOPs. Finally, the conclusion is given in “Conclusion” section.
Basic concepts of multiobjective optimization and nondominated sorting

Step 1) Initialize the index i to 1.

Step 2) Find all solutions which are not dominated by any solution in P and move them from P to \(F_i\); \(i=i+1\).

Step 3) If P is empty, stop; otherwise, go to Step 2.
With nondominated sorting, the quality of solutions in a population can be considerably distinguished, and this strategy has been widely adopted in MOEAs. In the next two sections, we will discuss the effectiveness and efficiency of nondominated sorting in MOEAs, respectively.
Effectiveness of nondominated sorting for multi and manyobjective optimization
Effectiveness for multiobjective optimization
Figure 2 presents the convergence profiles of inverted generational distance (IGD) values obtained by NSGAII with nondominated sorting and the variant without nondominated sorting on 2objective ZDT1 and ZDT4, and 3objective DTLZ1 and DTLZ2, averaged over 30 independent runs, where the population size is set to 100 and the remaining parameters are set to the same values recommended in [7]. IGD is a popular metric for measuring the quality of a solution set in terms of both convergence and distribution. ZDT1, ZDT4, DTLZ1, and DTLZ2 are widely used benchmark MOPs with multi and unimodal properties, respectively [33, 34]. From the figure, the following two observations can be obtained.
First, nondominated sorting plays a crucial role in guiding the population of NSGAII to approximate the Pareto fronts of MOPs. On both uni and multimodel MOPs, the population of NSGAII is far from converging to the Pareto fronts in a maximum of 300 generations without using nondominated sorting. Second, the crowding distance is a little helpful for the convergence of populations of NSGAII on 2objective MOPs, but the population cannot converge to the Pareto fronts of MOPs without nondominated sorting. On 3objective MOPs, it seems that the crowding distance cannot promote populations of NSGAII towards the Pareto fronts. As the number of generations increases, the IGD values obtained by the NSGAII without nondominated sorting increase on 3objective DTLZ1 and DTLZ2. The second observation can be confirmed by Fig. 3, where the minimal distance of solutions in population to Pareto fronts is presented at different iterations for the variant of NSGAII without nondominated sorting on 2objective ZDT1 and ZDT4 and 3objective DTLZ1 and DTLZ2.
To further illustrate the role of nondominated sorting in MOEAs, Fig. 4 presents the number of solutions in different fronts determined by nondominated sorting for next population of NSGAII on 2objective ZDT1 and ZDT4 and 3objective DTLZ1 and DTLZ2, where the number of candidate solutions is 200 and the size of next population is 100. From the figure, it can be seen clearly that nondominated sorting is a very effective strategy to distinguish the quality of solutions in population evolution of NSGAII for solving MOPs. For the combined population consisting of parent and offspring populations, nondominated sorting can determine a large number of candidate solutions unsuitable for surviving for next population when NSGAII is used to solve MOPs. It can also be found that the number of solutions which are considered unsuitable for surviving considerably decreases on 3objective MOPs, which implies the decrement of ability of nondominated sorting in distinguishing the quality of solutions as the number of objectives increases.
Effectiveness for manyobjective optimization
Figure 5 presents the convergence profiles of IGD values obtained by KnEA with nondominated sorting and the variant without nondominated sorting on 5 and 10objective DTLZ1 and DTLZ2, where the population size is set to 100 and the expected rate of knee points is set to 0.1 for DTLZ1 and 0.5 for DTLZ2. From the figure, the following three observations can be obtained. First, nondominated sorting is very helpful for KnEA to achieve a set of nondominated solutions with better quality for solving MaOPs. The KnEA with nondominated sorting can always obtain better IGD values than the variant of KnEA without nondominated sorting under different iterations, on all tested MaOPs, especially for 5objective DTLZ2. To illustrate the reason for the enhanced quality of solution set, Fig. 6 gives the generational distance (GD) values obtained by KnEA with nondominated sorting and the variant without nondominated sorting at different iterations on DTLZ1 and DTLZ2 with 5 and 10 objectives. The GD is a widely used metric for measuring the convergence of a solution set. As shown in the figure, nondominated sorting can clearly help population of KnEA to better converge to the Pareto fronts on MaOPs, especially for DTLZ1 which has a large number of local Pareto fronts.
Second, the role of nondominated sorting in promoting population of KnEA to converge to the Pareto fronts degenerates on MaOPs in comparison to that on MOPs. The main reason is attributed to a phenomenon called dominance resistance [19], since the number of solutions will considerably increase as the number of objectives increases. Taking two random solutions in Mdimensional objective space as an example, the probability that one solution dominates the other one is \((\frac{1}{2})^{M1}\), as shown in Fig. 7. It is clear that the probability decreases rapidly with the increasing number of objectives, and it becomes almost impossible that solutions with more than 12 objectives in a random population can be dominated [35]. It is necessary to note that nondominated sorting can still determine a few candidate solutions in the combined population unsuitable for surviving for next population in solving MaOPs, which is helpful for promoting population of KnEA to converge to the Pareto fronts.
Third, knee point selection plays a key role in enabling the population of KnEA to converge to the Pareto fronts for MaOPs. On all tested MaOPs, KnEA only using knee point selection can achieve a competitive performance in terms of IGD and GD, especially for DTLZ2 with 5 and 10 objectives. This result shows that KnEA is still effective on most MaOPs when the knee points are selected from the whole population instead of each nondominated front of the population, despite that the performance has a little deterioration. It seems that nondominated sorting is more helpful in KnEA on DTLZ1 with a large number of local Pareto fronts, which can considerably enhance the convergence of population of KnEA in terms of GD.
Therefore, we can summarize that nondominated sorting is also important for developing a promising MOEA to solve MaOPs, especially for some complex MaOPs, such as those with multiple local Pareto fronts.
Rectifications of Pareto dominance for manyobjective optimization
As shown in the above subsection, the effectiveness of nondominated sorting degenerates in MOEAs for MaOPs due to the decrement of selection pressure of the Pareto dominance. To address this issue, a large number of enhanced versions of the Pareto dominance have been proposed based on different ideas.
The second idea for enhancing the effectiveness of the Pareto dominance is based on the expansion of the dominance area. For two random solutions with M objectives, the probability that one solution dominates the other one is \((\frac{1}{2})^{M1}\), and hence, it is rare for one solution to dominate the other one in highdimensional space. The controlling dominance area of solutions (CDAS) method [39] expands the dominance area of each solution by a specified angle on each objective, and thus, the probability of dominance and the selection pressure increase. An adaptive version of CDAS was suggested in [40], where the expanding angle was adaptively estimated according to the extreme solutions. It is worth noting that the CDAS expands the dominance area by modifying the objective values of solutions; there also exist some dominance relations which expand the dominance area by modifying the definition of dominance, e.g., \(\alpha \)dominance [41] and generalized Pareto optimality (GPO) [42].
The third idea is to adopt the concept of fuzzy logic to develop novel dominance relations, such as (1k)dominance [43], Ldominance [44], and fuzzy dominance [45, 46]. In the Pareto dominance, one solution dominates another one only if all the objective values of the former are smaller than or equal to those of the latter, whereas in fuzzy logicbased dominance relations, one solution may dominate another one if the majority of the objectives of the former are smaller than those of the latter. In this case, a solution can dominate those which have much worse values than it on most objectives and slightly better values than it on a few objectives, and thus, the quality of solutions can be distinguished.
The fourth idea enhances the effectiveness of the Pareto dominance by means of a set of uniformly distributed reference vectors as suggested in decompositionbased MOEAs [47, 48]. \(\theta \)Dominance [48] is a dominance relation belonging to this category, where each solution is associated with its nearest reference vector, and a solution is said to dominate another one if and only if the two solutions are associated with the same reference vector and the former has better convergence and diversity than the latter. \(\theta \)dominance relation aims to make each solution converge to the same direction of one reference vector, which can enable the population to hold a good convergence and diversity.
It is necessary to stress that there are also some other interesting ideas which enhance the effectiveness of nondominated sorting by combining it with additional convergence or diversityrelated metrics, instead of directly modifying the definition of the Pareto dominance. Some representatives belonging to this category include hypervolume [49], knee point [22], enhanced IGD [18], and shiftbased density [50].
Efficiency of nondominated sorting for multi and manyobjective optimization
Main nondominated sorting methods and their time complexity
In the past 2 decades, a large number of interesting algorithms have been developed to address the high computational cost of nondominated sorting. In what follows, we only recall several nondominated sorting algorithms which are widely used in literature. The interested readers can refer to [51] for a more detailed list of nondominated sorting algorithms.
In 2003, Jensen [26] suggested a nondominated sorting method, called Jensen’s sort, based on the divideandconquer strategy. Jensen’s sort is an improved version of the nondominated sorting method developed by Kung et al. [52], where the time complexity has been reduced to \(O(N\mathtt{ln}^{M1}N)\) from \(O(N^2\mathtt{ln}^{M1}N)\) (M and N are hereafter denoted as the number of objectives and the population size, respectively), considerably outperforming the time complexity \(O(MN^2)\) of fast nondominated sort (FNS) developed in NSGAII. Compared to the FNS, Jensen’s sort also has a significant improvement in space complexity. The space complexity of Jensen’s sort is O(N), whereas FNS holds a space complexity of \(O(N^2)\). Hence, Jensen’s sort is computationally more efficient than FNS on MOPs with a small number of objectives, especially for those with two or three objectives. In spite of the high computational efficiency, Jensen’s sort suffers from the restriction that two compared solutions should not have the same value in any of their objectives.
To address the weakness of Jensen’s sort, Fang et al. [53], in 2008, proposed a new divideandconquerbased nondominated sorting algorithm, where a new data structure called dominance tree was adopted to reduce the number of redundant comparisons in FNS. This algorithm is called the divideandconquer based sort. Empirical results demonstrated that the divideandconquerbased sort holds a time complexity close to \(O(MN\mathtt{ln}N)\) for twoobjective MOPs and approaches asymptotically to the upper bound \(O(MN^2)\) as the number of objectives increases. Another improved version of Jensen’s sort, termed generalized Jensen’s sort, was also developed by Fortin et al. in [54]. As reported in [54], the generalized Jensen’s sort can well address the weakness of Jensen’s sort without increasing the time complexity and space complexity.
In 2012, McClymont and Keedwell [28] developed a novel nondominated sorting method, called deductive sort, by exploiting the properties of Pareto optimality, dominance, and nondominance, as well as the possible inherent inferences that can be made based on the nature of these relationships. Although deductive sort holds a time complexity of \(O(MN^2)\) in the worst case which is the same as that of FNS, empirical evidence indicated that it can save a large number of comparisons between solutions.
The corner sort suggested by Wang and Yao [30] in 2013 is also a computationally very efficient nondominated sorting algorithm. In corner sort, a nondominated solution is first selected from the corner solutions, and then, the solutions dominated by it will be ignored to save comparisons between solutions. As reported in [30], corner sort is more suited for solving MaOPs than FNS and deductive sort, and the more objectives an MaOP has, the more objective comparisons it can save. Corner sort holds a time complexity of \(O(MN^2)\) in the worst case, but in some best cases, its time complexity can be reduced to \(O(MN\sqrt{N})\).
Zhang et al. [29] also developed an efficient nondominated sorting method in 2015, called ENS, which has been shown to well suit for solving MOPs with a small number of objectives, especially for MOPs with two or three objectives. The high efficiency of ENS is attributed to the fact that an operation of presort was suggested for population according to one of the objectives, since in the sorted population, a solution can never be dominated by solutions ranked behind it, thereby saving a large number of comparisons between solutions. It is worth noting that the superiority of ENS will decrease as the number of objectives increases, despite that it still outperforms several existing nondominated sorting methods, such as FNS and deductive sort as indicated in [29]. To solve this problem, a treebased nondominated sorting method, termed TENS, was suggested by Zhang et al. in [23] based on the ENS framework. Empirical evaluations demonstrated that TENS is computationally very efficient for MaOPs and its computational cost almost keeps the same as the number of objectives increases. TENS holds a worst case time complexity of \(O(MN^2)\) (the same with that of ENS) and a best case time complexity of \(O(MN\mathtt{ln}N/\mathtt{ln}M)\), which is better than \(O(MN\mathtt{ln}N)\) of ENSBS and \(O(MN\sqrt{N})\) of ENSSS.
Number of objective comparisons of the eight nondominated sorting methods when they are embedded into NSGAII for solving twoobjective and threeobjective DTLZ2
Methods  \(N=100\)  \(N=500\)  

\(M=2\)  \(M=3\)  \(M=2\)  \(M=3\)  
FNS  1.6E+7  1.9E+7  4.0E+8  4.7E+8 
Generalized Jensen’s sort  5.0E+5  1.1E+6  3.8E+6  7.9E+6 
Divideandconquer sort  5.6E+6  8.0E+6  1.2E+8  2.0E+8 
Deductive sort  5.2E+6  7.7E+6  1.2E+8  1.9E+8 
Corner sort  4.6E+6  6.6E+6  10.0E+7  1.5E+8 
Mfront  3.8E+6  5.1E+6  8.8E+7  1.1E+8 
TENS  3.1E+6  1.5E+6  6.9E+7  2.0E+7 
ENSSS  3.9E+5  4.2E+6  2.6E+6  9.8E+7 
Efficient nondominated sorting for multiobjective optimization
In the following, we verify the computational efficiency of eight widely used nondominated sorting methods when they are adopted for handling MOPs. The eight methods under consideration include fast nondominated sort (FNS) [7], generalized Jensen’s sort [54], divideandconquerbased sort [53], deductive sort [28], corner sort [30], Mfront [55], ENSSS [29], and TENS [23].
In the experiments, the computational efficiency of these nondominated sorting methods is considered on random populations and MOEAs. The first scenario is used to mimic the situation in the early search stages of MOEAs, and the second scenario is adopted to test the computational efficiency of these nondominated sorting methods when they are embedded into MOEAs for solving MOPs. In the second scenario, all components of an embedded MOEA are identical with the only exception of the nondominated sorting methods adopted in it.
Figures 8 and 9 present the number of objective comparisons and runtime(s) of the eight nondominated sorting methods in the first scenario for twoobjective and threeobjective optimization, averaged over 30 random populations with the same size. From the figures, the following two results can be observed. First, in terms of number of objective comparisons, the generalized Jensen’s sort performs the best on random populations for both twoobjective and threeobjective optimization. ENSSS and divideandconquerbased sort need slightly more objective comparisons than generalized Jensen’s sort. The rest five nondominated sorting methods underperform the generalized Jensen’s sort, ENSSS, and divideandconquerbased sort.
Second, in terms of runtime, ENSSS always achieves the best efficiency on random populations for twoobjective and threeobjective optimization. The superiority of ENSSS over the generalized Jensen’s sort in runtime may be partly attributed to the fact that ENSSS holds a space complexity of O(1), whereas the space complexity of generalized Jensen’s sort is O(N). It is worth noting that deductive sort also achieves a competitive efficiency on random populations in terms of runtime despite that it uses more objective comparisons.
Ratio of runtime of the eight nondominated sorting methods to that of ENSSS when they are embedded into NSGAII for solving twoobjective and threeobjective DTLZ2
Methods  \(N=100\)  \(N=500\)  

\(M=2\)  \(M=3\)  \(M=2\)  \(M=3\)  
FNS  18.98  4.18  93.84  4.59 
Generalized Jensen’s sort  16.98  21.13  16.73  4.82 
Divideandconquer sort  14.02  2.91  41.41  2.39 
Deductive sort  6.94  1.82  30.47  1.98 
Corner sort  15.60  3.65  46.74  2.78 
Mfront  43.18  9.39  66.51  3.41 
TENS  10.25  1.26  33.95  0.51 
ENSSS  1.00  1.00  1.00  1.00 
For the number of objective comparisons, the generalized Jensen’s sort still performs the best for threeobjective DTLZ2, but ENSSS can achieve the best efficiency in solving twoobjective DTLZ2. As for the runtime, ENSSS takes much less cost than the other seven nondominated sorting methods for solving twoobjective and threeobjective DTLZ2, in case a population size of 100 is used. When the population size increases to 500, ENSSS takes the least runtime for solving twoobjective DTLZ2, whereas TENS will achieve the best for threeobjective DTLZ2.
From the tables, it can also be seen that the computational efficiency of ENSSS will be enhanced when a large population size of NSGAII is used to solve MOPs, especially for solving MOPs with two objectives. Take the wellknown nondominated sorting method FNS as a comparison, the runtime taken by FNS is about 19 times of that of ENSSS if a population size of 100 is used to solve twoobjective DTLZ2. This ratio will be incremented to 94 as the population size becomes 500.
On the basis of the above empirical comparisons, we can conclude that ENSSS is computationally more efficient than the stateoftheart nondominated sorting methods for multiobjective optimization, which can significantly improve the computational efficiency of an MOEA when it is embedded into the algorithm to solve MOPs. In the case of solving MOPs with three objectives, TENS will be strongly suggested to be adopted if a large population size is used in the MOEA.
Efficient nondominated sorting for manyobjective optimization
Number of objective comparisons of the eight nondominated sorting methods when they are embedded into KnEA for solving 5objective and 10objective DTLZ2
Methods  \(N=100\)  \(N=500\)  

\(M=5\)  \(M=10\)  \(M=5\)  \(M=10\)  
FNS  2.1E+7  2.8E+7  5.3E+8  6.6E+8 
Generalized Jensen’s sort  4.6E+6  6.7E+6  4.7E+7  8.8E+7 
Divideandconquer sort  1.0E+7  1.4E+7  2.5E+8  3.1E+8 
Deductive sort  1.0E+7  1.4E+7  2.5E+8  3.0E+8 
Corner sort  9.2E+6  1.4E+7  1.9E+8  2.5E+8 
Mfront  1.2E+7  2.5E+7  2.3E+7  5.5E+8 
ENSSS  6.5E+6  1.0E+7  1.5E+8  2.1E+8 
TENS  1.7E+6  2.8E+6  1.8E+7  2.4E+7 
Ratio of runtime of the eight nondominated sorting methods to that of TENS when they are embedded into KnEA for solving 5objective and 10objective DTLZ2
Methods  \(N=100\)  \(N=500\)  

\(M=5\)  \(M=10\)  \(M=5\)  \(M=10\)  
FNS  3.97  4.17  11.15  10.57 
Generalized Jensen’s sort  58.97  71.43  57.60  92.60 
Divideandconquer sort  2.67  2.67  6.09  5.65 
Deductive sort  1.91  2.06  5.27  4.90 
Corner sort  3.66  3.51  7.27  6.08 
Mfront  10.53  13.20  11.10  14.13 
ENSSS  1.13  1.36  3.14  3.32 
TENS  1.00  1.00  1.00  1.00 
In this subsection, we test the computational efficiency of the above eight nondominated sorting methods on MaOPs.
The experiments are conducted on three different scenarios which are often encountered in manyobjective optimization. In the first scenario, we test the efficiency of the nondominated sorting methods on random populations for manyobjective optimization; in the second scenario, the computational costs of the nondominated sorting methods are compared by embedding them into the MOEAs which are specially tailored to solve MaOPs; in the third scenario, we compare the efficiency of the nondominated sorting methods when they are used to obtain a set of reference points uniformly distributed on the true PFs, which is required for calculating some performance indicators, such as GD [56] and IGD [57].
Number of objective comparisons of the eight nondominated sorting methods for obtaining a set of reference points uniformly distributed on the true PF of DTLZ7 under different numbers of sampled points
Methods  1000 Points  10,000 Points  

\(M=5\)  \(M=10\)  \(M=5\)  \(M=10\)  
FNS  2.9E+6  3.0E+6  2.9E+8  3.0E+8 
Generalized Jensen’s sort  2.3E+5  3.5E+5  5.4E+6  1.3E+7 
Divideandconquer sort  8.9E+5  1.5E+6  5.0E+7  1.4E+8 
Deductive sort  7.5E+5  1.5E+6  3.3E+7  1.4E+8 
Corner sort  5.9E+5  1.2E+6  2.7E+7  1.0E+8 
Mfront  9.0E+5  2.6E+6  5.4E+7  2.1E+8 
ENSSS  4.9E+5  1.0E+6  2.7E+7  9.4E+7 
TENS  1.7E+5  3.7E+5  9.1E+6  2.5E+7 
Ratio of runtime of the eight nondominated sorting methods to that of TENS when they are used to obtain a set of reference points uniformly distributed on the true PF of DTLZ7 under different numbers of sampled points
Methods  1000 Points  10,000 Points  

\(M=5\)  \(M=10\)  \(M=5\)  \(M=10\)  
FNS  7.98  4.61  12.14  4.69 
Generalized Jensen’s sort  36.69  32.19  11.02  10.99 
Divideandconquer sort  3.32  2.59  3.31  2.45 
Deductive sort  2.40  2.35  1.88  2.28 
Corner sort  2.99  2.94  2.59  3.01 
Mfront  5.95  6.77  4.04  4.82 
ENSSS  2.00  1.46  2.37  1.39 
TENS  1.00  1.00  1.00  1.00 
Tables 3 and 4 list the experimental results of the eight nondominated sorting methods when they are embedded into KnEA with a population size of 100 and 500 to solve 5objective and 10objective DTLZ2, respectively. The parameter settings of KnEA are the same as those recommended in [22] and the reported results are averaged over 30 independent runs. From the tables, we can find that TENS performs much more efficient than the other seven nondominated sorting methods, in case that they are embedded into MOEAs to solve MaOPs. ENSSS achieves the second best efficiency in terms of runtime when they are embedded into KnEA to solve MaOPs. It can also be seen that the superiority of TENS over the compared methods will be enhanced as the population size and the number of objectives increase in KnEA. These empirical results show that TENS is more suited to deal with largescale MaOPs, since a large population is often needed for solving largescale optimization problems.
Tables 5 and 6 present the experimental results of the eight nondominated sorting methods for obtaining a set of reference points uniformly distributed on the true PF of DTLZ7 under 1000 and 10,000 sampled points, respectively. As can be seen from the tables, the generalized Jensen’s sort achieves the fewest objective comparisons in obtaining a set of reference points uniformly distributed on the true PF of DTLZ7, except in the case of 1000 sampled points for 5objective DTLZ7, where TENS performs the fewest objective comparisons. In terms of runtime, TENS takes the least among all eight considered nondominated sorting methods, despite that it consumes more objective comparisons than the generalized Jensen’s sort. It can also be found that ENSSS is the second less timeconsuming nondominated sorting method except in the case of 10,000 sampled points for 10objective DTLZ7, where deductive sort performs the second best.
From the empirical results shown in the above three scenarios, we can conclude that TENS is more suited for manyobjective optimization, whose computational efficiency is much more competitive than that of the existing nondominated sorting methods, especially when a large population size is adopted.
Approximate nondominated sorting for manyobjective optimization
Ratio of runtime of AENS to that of TENS when they are embedded into KnEA and Two_Arch2 for solving DTLZ1–DTLZ7 and WFG1–WFG9 with 5 and 10 objectives
Problems  KnEA  Two_Arch2  

\(M=5\)  \(M=10\)  \(M=5\)  \(M=10\)  
DTLZ1  0.59  0.53  0.64  0.53 
DTLZ2  0.82  0.68  0.75  0.71 
DTLZ3  0.60  0.46  0.63  0.49 
DTLZ4  0.74  0.68  0.79  0.76 
DTLZ5  0.55  0.47  0.39  0.44 
DTLZ6  0.54  0.56  0.43  0.52 
DTLZ7  0.59  0.5  0.50  0.44 
WFG1  0.60  0.56  0.66  0.55 
WFG2  0.59  0.54  0.65  0.63 
WFG3  0.65  0.58  0.52  0.50 
WFG4  0.75  0.72  0.74  0.76 
WFG5  0.76  0.69  0.79  0.81 
WFG6  0.73  0.69  0.74  0.76 
WFG7  0.76  0.71  0.79  0.81 
WFG8  0.65  0.62  0.68  0.67 
WFG9  0.69  0.72  0.80  0.77 
HV values obtained by KnEA with accurate nondominated sorting method and AENS on DTLZ1–DTLZ7 and WFG1–WFG9 with 5 and 10 objectives
Problems  \(M=5\)  \(M=10\)  

TENS  AENS  TENS  AENS  
DTLZ1  \(4.5969\hbox {E}{}1\pm 8.82\hbox {E}{}2\)  \(\mathbf{5.0734E }{}\mathbf{1 }\pm \mathbf{1.25E }{}\mathbf{1 }\)  \(\mathbf{2.7163E }{}\mathbf{1 }\pm \mathbf{1.41E }{}\mathbf{1 }\)  \(7.9948\hbox {E}{}2\pm 1.02\hbox {E}{}1\) 
DTLZ2  \(\mathbf{6.1657E }{}\mathbf{1 }\pm \mathbf{6.15E }{}\mathbf{3 }\)  \(6.0946\hbox {E}{}1\pm 8.69\hbox {E}{}3\)  \(8.1704\hbox {E}{}1\pm 6.24\hbox {E}{}2\)  \(\mathbf{8.4099E }{}\mathbf{1 }\pm \mathbf{1.18E }{}\mathbf{2 }\) 
DTLZ3  \(5.6254\hbox {E}{}2\pm 1.04\hbox {E}{}1\)  \(\mathbf{2.6701E }{}\mathbf{1 }\pm \mathbf{7.27E }{}\mathbf{2 }\)  \(3.9095\hbox {E}{}3\pm 1.26\hbox {E}{}2\)  \(\mathbf{6.3055E }{}\mathbf{2 }\pm \mathbf{7.43E }{}\mathbf{2 }\) 
DTLZ4  \(\mathbf{6.1858E }{}\mathbf{1 }\pm \mathbf{6.89E }{}\mathbf{3 }\)  \(6.1811\hbox {E}{}1\pm 6.61\hbox {E}{}3\)  \(8.0266\hbox {E}{}1\pm 2.05\hbox {E}{}2\)  \(\mathbf{8.3580E }{}\mathbf{1 }\pm \mathbf{1 }.\mathbf{66E }{}\mathbf{2 }\) 
DTLZ5  \(\mathbf{7.4163E }{}\mathbf{2 }\pm \mathbf{1.80E }{}\mathbf{2 }\)  \(7.3073\hbox {E}{}2\pm 2.46\hbox {E}{}2\)  \(\mathbf{3.8298E }{}{} \mathbf {2} \pm \mathbf{1.25E }{}\mathbf{2 }\)  \(3.5344\hbox {E}{}2\pm 2.08\hbox {E}{}2\) 
DTLZ6  \(2.4864\hbox {E}{}2\pm 1.87\hbox {E}{}2\)  \(\mathbf{3.4327E }{}\mathbf{2 }\pm \mathbf{2.08E }{}\mathbf{2 }\)  \(\mathbf{1.6162E }{}\mathbf{2 }\pm \mathbf{2.29E }{}\mathbf{2 }\)  \(2.8049\hbox {E}{}4\pm 8.72\hbox {E}{}4\) 
DTLZ7  \(1.3594\hbox {E}{}1\pm 6.57\hbox {E}{}3\)  \(\mathbf{1.4844E }{}\mathbf{1 }\pm \mathbf{5.74E }{}\mathbf{3 }\)  \(2.2713\hbox {E}{}2\pm 8.14\hbox {E}{}3\)  \(\mathbf{9.9292E }{}\mathbf{2 }\pm \mathbf{7.58E }{}\mathbf{3 }\) 
WFG1  \(9.7626\hbox {E}{}1\pm 2.32\hbox {E}{}2\)  \(\mathbf{9.8972E }{}\mathbf{1 }\pm \mathbf{8.03E }{}\mathbf{3 }\)  \(9.4577\hbox {E}{}1\pm 8.51\hbox {E}{}2\)  \(\mathbf{9.9397E }{}\mathbf{1 }\pm \mathbf{2.66E }{}\mathbf{3 }\) 
WFG2  \(\mathbf{9.7379E }{}\mathbf{1 }\pm \mathbf{4.23E }{}\mathbf{2 }\)  \(9.2167\hbox {E}{}1\pm 9.42\hbox {E}{}2\)  \(9.8899\hbox {E}{}1\pm 1.93\hbox {E}{}3\)  \(\mathbf{9.9603E }{}\mathbf{1 }\pm \mathbf{2.87E }{}\mathbf{3 }\) 
WFG3  \(5.3516\hbox {E}{}1\pm 1.40\hbox {E}{}2\)  \(\mathbf{5.6173E }{}\mathbf{1 }\pm \mathbf{2 }.\mathbf{11E }{}{} \mathbf {2} \)  \(5.3188\hbox {E}{}1\pm 1.73\hbox {E}{}2\)  \(\mathbf{5.5059E }{}\mathbf{1 }\pm \mathbf{1.33E }{}\mathbf{2 }\) 
WFG4  \(\mathbf{5.6356E }{}\mathbf{1 }\pm \mathbf{9.52E }{}\mathbf{3 }\)  \(5.6327\hbox {E}{}1\pm 7.44\hbox {E}{}3\)  \(7.7241\hbox {E}{}1\pm 2.30\hbox {E}{}2\)  \(\mathbf{7.9026E }{}\mathbf{1 }\pm \mathbf{1.09E }{}\mathbf{2 }\) 
WFG5  \(\mathbf{5.4986E }{}\mathbf{1 }\pm \mathbf{5.80E }{}\mathbf{3 }\)  \(5.4724\hbox {E}{}1\pm 5.75\hbox {E}{}3\)  \(\mathbf{7.6269E }{}\mathbf{1 }\pm \mathbf{5.32E }{}\mathbf{3 }\)  \(7.5885\hbox {E}{}1\pm 4.79\hbox {E}{}3\) 
WFG6  \(\mathbf{5.3199E }{}\mathbf{1 }\pm \mathbf{2.63E }{}\mathbf{2 }\)  \(5.1521\hbox {E}{}1\pm 1.53\hbox {E}{}2\)  \(7.2757\hbox {E}{}1\pm 3.80\hbox {E}{}2\)  \(\mathbf{7.3183E }{}\mathbf{1 }\pm \mathbf{2.00E }{}\mathbf{2 }\) 
WFG7  \(6.1116\hbox {E}{}1\pm 7.31\hbox {E}{}3\)  \(\mathbf{6.1254E }{}\mathbf{1 }\pm \mathbf{4 }.\mathbf{73E }{}{} \mathbf {3} \)  \(8.2764\hbox {E}{}1\pm 9.16\hbox {E}{}3\)  \(\mathbf{8.2897E }{}\mathbf{1 }\pm \mathbf{1.17E }{}\mathbf{2 }\) 
WFG8  \(\mathbf{3.7022E }{}\mathbf{1 }\pm \mathbf{2.14E }{}\mathbf{2 }\)  \(3.6682\hbox {E}{}1\pm 2.72\hbox {E}{}2\)  \(\mathbf{5.9964E }{}\mathbf{1 }\pm \mathbf{4.96E }{}\mathbf{2 }\)  \(5.8005\hbox {E}{}1\pm 3.96\hbox {E}{}2\) 
WFG9  \(4.9760\hbox {E}{}1\pm 6.52\hbox {E}{}2\)  \(\mathbf{5.2891E }{}\mathbf{1 }\pm \mathbf{4.38E }{}\mathbf{2 }\)  \(6.8623\hbox {E}{}1\pm 7.20\hbox {E}{}2\)  \(\mathbf{7.0326E }{}\mathbf{1 }\pm \mathbf{4.68E }{}\mathbf{2 }\) 
HV values obtained by Two_Arch2 with accurate nondominated sorting method and AENS on DTLZ1–DTLZ7 and WFG1–WFG9 with 5 and 10 objectives
Problems  \(M=5\)  \(M=10\)  

TENS  AENS  TENS  AENS  
DTLZ1  \(\mathbf{8.6562E }{}\mathbf{1 }\pm \mathbf{2.11E }{}\mathbf{2 }\)  \(8.2886\hbox {E}{}1\pm 2.08\hbox {E}{}2\)  \(\mathbf{8.7591E }{}\mathbf{1 }\pm \mathbf{4.92E }{}\mathbf{2 }\)  \(7.9605\hbox {E}{}1\pm 4.67\hbox {E}{}2\) 
DTLZ2  \(\mathbf{5.7334E }{}\mathbf{1 }\pm \mathbf{5.88E }{}\mathbf{3 }\)  \(5.6916\hbox {E}{}1\pm 7.30\hbox {E}{}3\)  \(4.7341\hbox {E}{}1\pm 3.09\hbox {E}{}2\)  \(\mathbf{6.7170E }{}\mathbf{1 }\pm \mathbf{2.89E }{}\mathbf{2 }\) 
DTLZ3  \(\mathbf{3.0047E }{}\mathbf{1 }\pm \mathbf{4.14E }{}\mathbf{2 }\)  \(2.4114\hbox {E}{}1\pm 3.13\hbox {E}{}2\)  \(2.3976\hbox {E}{}1\pm 1.27\hbox {E}{}1\)  \(\mathbf{2.5976E }{}\mathbf{1 }\pm \mathbf{5.31E }{}\mathbf{2 }\) 
DTLZ4  \(\mathbf{5.4961E }{}\mathbf{1 }\pm \mathbf{1.05E }{}\mathbf{2 }\)  \(5.4904\hbox {E}{}1\pm 1.15\hbox {E}{}2\)  \(5.4429\hbox {E}{}1\pm 3.98\hbox {E}{}2\)  \(\mathbf{7.0587E }{}\mathbf{1 }\pm \mathbf{1.46E }{}\mathbf{2 }\) 
DTLZ5  \(1.3372\hbox {E}{}1\pm 7.98\hbox {E}{}3\)  \(\mathbf{1.3770E }{}\mathbf{1 }\pm \mathbf{8.76E }{}\mathbf{3 }\)  \(5.2870\hbox {E}{}2\pm 1.55\hbox {E}{}2\)  \(\mathbf{8.0355E }{}\mathbf{2 }\pm \mathbf{2.34E }{}\mathbf{2 }\) 
DTLZ6  \(1.1627\hbox {E}{}1\pm 1.09\hbox {E}{}2\)  \(\mathbf{1.2888E }{}\mathbf{1 }\pm \mathbf{1.81E }{}\mathbf{2 }\)  \(9.8949\hbox {E}{}3\pm 1.54\hbox {E}{}2\)  \(\mathbf{5.9620E }{}\mathbf{2 }\pm \mathbf{1.26E }{}\mathbf{2 }\) 
DTLZ7  \(1.3042\hbox {E}{}1\pm 1.65\hbox {E}{}2\)  \(\mathbf{1.4944E }{}\mathbf{1 }\pm \mathbf{1.16E }{}\mathbf{2 }\)  \(8.4290\hbox {E}{}2\pm 2.32\hbox {E}{}2\)  \(\mathbf{1.0447E }{}\mathbf{1 }\pm \mathbf{1.15E }{}\mathbf{2 }\) 
WFG1  \(9.8852\hbox {E}{}1\pm 1.22\hbox {E}{}3\)  \(\mathbf{9.8861E }{}\mathbf{1 }\pm \mathbf{1.48E }{}\mathbf{3 }\)  \(9.9069\hbox {E}{}1\pm 1.29\hbox {E}{}3\)  \(\mathbf{9.9436E }{}\mathbf{1 }\pm \mathbf{8.20E }{}\mathbf{4 }\) 
WFG2  \(9.1682\hbox {E}{}1\pm 9.08\hbox {E}{}2\)  \(\mathbf{9.5573E }{}\mathbf{1 }\pm \mathbf{6.89E }{}\mathbf{2 }\)  \(9.9005\hbox {E}{}1\pm 3.14\hbox {E}{}3\)  \(\mathbf{9.9036E }{}\mathbf{1 }\pm \mathbf{2.82E }{}\mathbf{3 }\) 
WFG3  \(\mathbf{5.6637E }{}\mathbf{1 }\pm \mathbf{1.03E }{}\mathbf{2 }\)  \(5.6437\hbox {E}{}1\pm 1.20\hbox {E}{}2\)  \(5.6233\hbox {E}{}1\pm 8.87\hbox {E}{}3\)  \(\mathbf{5.7551E }{}\mathbf{1 }\pm \mathbf{1.61E }{}\mathbf{2 }\) 
WFG4  \(5.1069\hbox {E}{}1\pm 1.00\hbox {E}{}2\)  \(\mathbf{5.1718E }{}\mathbf{1 }\pm \mathbf{9.87E }{}\mathbf{3 }\)  \(4.9015\hbox {E}{}1\pm 1.85\hbox {E}{}2\)  \(\mathbf{5.1932E }{}\mathbf{1 }\pm \mathbf{2.24E }{}\mathbf{2 }\) 
WFG5  \(4.9430\hbox {E}{}1\pm 7.43\hbox {E}{}3\)  \(\mathbf{5.0298E }{}\mathbf{1 }\pm \mathbf{6.10E }{}\mathbf{3 }\)  \(4.7617\hbox {E}{}1\pm 2.18\hbox {E}{}2\)  \(\mathbf{5.4869E }{}\mathbf{1 }\pm \mathbf{1.80E }{}\mathbf{2 }\) 
WFG6  \(\mathbf{4.8227E }{}\mathbf{1 }\pm \mathbf{1.95E }{}\mathbf{2 }\)  \(4.8073\hbox {E}{}1\pm 1.32\hbox {E}{}2\)  \(4.5144\hbox {E}{}1\pm 2.37\hbox {E}{}2\)  \(\mathbf{5.3360E }{}\mathbf{1 }\pm \mathbf{3.17E }{}\mathbf{2 }\) 
WFG7  \(5.5479\hbox {E}{}1\pm 7.28\hbox {E}{}3\)  \(\mathbf{5.6390E }{}\mathbf{1 }\pm \mathbf{7.92E }{}\mathbf{3 }\)  \(5.2117\hbox {E}{}1\pm 1.95\hbox {E}{}2\)  \(\mathbf{5.6443E }{}\mathbf{1 }\pm \mathbf{2.51E }{}\mathbf{2 }\) 
WFG8  \(3.4962\hbox {E}{}1\pm 1.44\hbox {E}{}2\)  \(\mathbf{3.6521E }{}\mathbf{1 }\pm \mathbf{2.29E }{}\mathbf{2 }\)  \(2.8372\hbox {E}{}1\pm 3.96\hbox {E}{}2\)  \(\mathbf{3.4275E }{}\mathbf{1 }\pm \mathbf{5.26E }{}\mathbf{2 }\) 
WFG9  \(\mathbf{4.9517E }{}\mathbf{1 }\pm \mathbf{7.85E }{}\mathbf{3 }\)  \(4.9494\hbox {E}{}1\pm 3.12\hbox {E}{}2\)  \(4.5549\hbox {E}{}1\pm 2.92\hbox {E}{}2\)  \(\mathbf{4.9367E }{}\mathbf{1 }\pm \mathbf{2.83E }{}\mathbf{2 }\) 
In the above two subsections, we have empirically verified the computational efficiency of eight stateoftheart nondominated sorting algorithms for multiobjective and manyobjective optimization, respectively. In this subsection, we consider another interesting idea of performing nondominated sorting for manyobjective optimization, called approximate nondominated sorting. AENS was the first algorithm developed recently based on approximate nondominated sorting for manyobjective optimization [31].
The main difference between AENS and existing nondominated sorting algorithms lies in the fact that, instead of identifying the accurate nondominated sorting result for a given population in existing nondominated sorting algorithms, AENS determines an approximate sorting result by performing at most threeobjective comparisons for each pair of solutions. This means that the existing nondominated sorting algorithms always find the same sorting result for a given population, and these algorithms distinguish themselves only in the computational efficiency. AENS obtains a sorting result different from that of the other nondominated sorting algorithms due to the errors caused by approximate sorting. In the following, we empirically verify the efficiency of AENS and the influence on performance of MOEAs by embedding it into two MOEAs, KnEA and Two_Arch2, developed recently for solving MaOPs. All reported experimental results are obtained by averaging over 30 independent runs.
Table 7 presents the computational efficiency of AENS in KnEA and Two_Arch2 on DTLZ1–DTLZ7 [58] and WFG1–WFG9 [59] with 5 and 10 objectives, in comparison with the accurate nondominated sorting algorithm TENS. The parameter settings of KnEA and Two_Arch2 are the same as recommended in [22, 60]. From the table, it can be seen clearly that AENS is more efficient than TENS in MOEAs for solving MaOPs. Compared to the accurate nondominated sorting TENS, AENS consumes roughly 70% runtime of that of TENS in both KnEA and Two_Arch2 to solve DTLZ and WFG test problems with 5 and 10 objectives. It can also be found that the superiority of AENS over TENS in computational efficiency will be enhanced as the number of objectives increases to 10. The above results demonstrate the competitiveness of approximate nondominated sorting in computational efficiency for handling manyobjective optimization.
To evaluate the influence of AENS on performance of MOEAs, Tables 8, 9 present the hypervolume (HV) values obtained by KnEA and Two_Arch2 with AENS and accurate nondominated sorting algorithm TENS on 5 and 10objective DTLZ and WFG test problems. HV is a performance indicator to measure the quality of solution sets obtained by MOEAs in terms of both convergence and diversity [61]. The method for calculating HV value is the same to that adopted in [23]. The larger the value of HV, the better the solution set. From the tables, the following three results can be observed.
First, AENS can enhance the performance of both KnEA and Two_Arch2 in solving most 5 and 10objective DTLZ and WFG test problems without considerably deterioration on all test instances under consideration. For the 32 test instances, AENS achieves better HV values than accurate nondominated sorting on 19 test instances in KnEA, and 24 instances in Two_Arch2. The enhanced performance of KnEA and Two_Arch2 may show that the errors introduced by approximate nondominated sorting are helpful for MOEAs to improve the performance in solving MaOPs.
Second, compared to the performance on 5objective MaOPs, the effectiveness of AENS is significantly enhanced on 10objective MaOPs for both KnEA and Two_Arch2. AENS obtains better HV values on 8 out of 16 test instances with 5 objectives in KnEA and 9 instances in Two_Arch2. The numbers of AENS outperforming accurate nondominated sorting increase to 11 in KnEA and 15 in Two_Arch2 on 16 test instances with 10 objectives. Third, AENS is more helpful for Two_Arch2 than KnEA in improving their performance to solve MaOPs. This implies that the idea of approximate nondominated sorting deserves further investigation by developing MOEAs well suited for approximate nondominated sorting as reported in [31].
From the above empirical results, we can conclude that approximate nondominated sorting AENS is a promising idea to perform nondominated sorting for manyobjective optimization, which cannot only improve the computational efficiency, but also enhance the performance in quality of solution set, when it is adopted in MOEAs to solve MaOPs.
Conclusion
In this paper, we have empirically analyzed the effectiveness and efficiency of nondominated sorting for evolutionary multi and manyobjective optimization. The effectiveness of nondominated sorting is verified by considering two MOEAs, NSGAII and KnEA, both of which adopted nondominated sorting as an important component, to solve MOPs and MaOPs, respectively. Experimental results obtained by NSGAII demonstrate that nondominated sorting is very important for MOEAs to converge to the Pareto fronts when they are used to solve MOPs. For MaOPs, nondominated sorting has been shown to be effective in MOEAs such as KnEA, especially for dealing with MaOPs with a large number of local Pareto fronts, despite that it suffers from the deterioration of effectiveness due to the dominance resistance phenomenon. Some enhanced variants of the Pareto dominance for manyobjective optimization have also been briefly introduced.
The efficiency of nondominated sorting is evaluated by comparing 8 stateoftheart nondominated sorting algorithms for evolutionary multi and manyobjective optimization. According to the experimental results, ENSSS performs the best in efficiency for multiobjective optimization and TENS holds the best efficiency for manyobjective optimization. The approximate nondominated sorting algorithm AENS has also been empirically discussed and experimental results have indicated that approximate nondominated sorting is a promising idea for manyobjective optimization in terms of both efficiency and effectiveness.
Notes
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 61672033, 61502004, 61502001), and the Joint Research Fund for Overseas Chinese, Hong Kong and Macao Scholars of the National Natural Science Foundation of China (Grant No. 61428302).
References
 1.Gupta A, Mańdziuk J, Ong YS (2015) Evolutionary multitasking in bilevel optimization. Complex Intell Syst 1:83–95CrossRefGoogle Scholar
 2.Nguyen S, Mei Y, Zhang M (2017) Genetic programming for production scheduling: a survey with a unified framework. Complex Intell Syst 3:1–26CrossRefGoogle Scholar
 3.Ponsich A, Jaimes AL, Coello CAC (2013) A survey on multiobjective evolutionary algorithms for the solution of the portfolio optimization problem and other finance and economics applications. IEEE Trans Evol Comput 17(3):321–344CrossRefGoogle Scholar
 4.Zhang L, Pan H, Su Y, Zhang X, Niu Y (2017) A mixed representationbased multiobjective evolutionary algorithm for overlapping community detection. IEEE Trans Cybern 47(9):2703–2716CrossRefGoogle Scholar
 5.Zhang X, Duan F, Zhang L, Cheng F, Jin Y, Tang K (2017) Pattern recommendation in taskoriented applications: a multiobjective perspective. IEEE Comput Intell Mag 12:43–53CrossRefGoogle Scholar
 6.Cheng R, Li M, Tian Y, Zhang X, Yang S, Jin Y, Yao X (2017) A benchmark test suite for evolutionary manyobjective optimization. Complex Intell Syst 3:67–81CrossRefGoogle Scholar
 7.Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGAII. IEEE Trans Evol Comput 6(2):182–197CrossRefGoogle Scholar
 8.Zitzler E, Laumanns M, Thiele L (2001) SPEA2: improving the strength Pareto evolutionary algorithm for multiobjective optimization. In: Proceedings of the 5th conference on evolutionary methods for design, optimization and control with applications to industrial problems, pp 95–100Google Scholar
 9.Corne DW, Jerram NR, Knowles JD, Oates MJ (2001) PESAII: regionbased selection in evolutionary multiobjective optimization. In: Proceedings of the 2001 conference on genetic and evolutionary computation, pp 283–290Google Scholar
 10.Zitzler E, Künzli S (2004) Indicatorbased selection in multiobjective search. In: Proceedings of the 8th international conference on parallel problem solving from nature, pp 832–842Google Scholar
 11.Zhang Q, Li H (2007) MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans Evol Comput 11(6):712–731CrossRefGoogle Scholar
 12.Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. AddisonWesley, BostonMATHGoogle Scholar
 13.Miettinen K (1999) Nonlinear multiobjective optimization. Springer, New YorkMATHGoogle Scholar
 14.Srinivas N, Deb K (1995) Multiobjective optimization using nondominated sorting in genetic algorithms. Evol Comput 2(3):221–248CrossRefGoogle Scholar
 15.Kukkonen S, Lampinen J (2005) GDE3: the third evolution step of generalized differential evolution. In: Proceedings of the 2005 IEEE congress on evolutionary computation, vol 1, pp 443–450Google Scholar
 16.Nebro AJ, Durillo JJ, GarciaNieto J, Coello CC, Luna F, Alba E (2009) SMPSO: a new PSObased metaheuristic for multiobjective optimization. In: Proceedings of the 2009 IEEE symposium on computational intelligence in multicriteria decisionmaking, pp 66–73Google Scholar
 17.Cai X, Li Y, Fan Z, Zhang Q (2015) An external archive guided multiobjective evolutionary algorithm based on decomposition for combinatorial optimization. IEEE Trans Evol Comput 19(4):508–523CrossRefGoogle Scholar
 18.Tian Y, Zhang X, Cheng R, Jin Y (2016) A multiobjective evolutionary algorithm based on an enhanced inverted generational distance metric. In: Proceedings of the 2016 IEEE congress on evolutionary computation, pp 5222–5229Google Scholar
 19.Li B, Li J, Tang K, Yao X (2015) Manyobjective evolutionary algorithms: a survey. ACM Comput Surv 48(1):13CrossRefGoogle Scholar
 20.Yang S, Li M, Liu X, Zheng J (2013) A gridbased evolutionary algorithm for manyobjective optimization. IEEE Trans Evol Comput 17(5):721–736CrossRefGoogle Scholar
 21.Deb K, Jain H (2014) An evolutionary manyobjective optimization algorithm using referencepoint based nondominated sorting approach, part I: solving problems with box constraints. IEEE Trans Evol Comput 18(4):577–601CrossRefGoogle Scholar
 22.Zhang X, Tian Y, Jin Y (2014) A knee point driven evolutionary algorithm for manyobjective optimization. IEEE Trans Evol Comput 19(6):761–776CrossRefGoogle Scholar
 23.Zhang X, Tian Y, Cheng R, Jin Y (2017) A decision variable clusteringbased evolutionary algorithm for largescale manyobjective optimization. IEEE Trans Evol Comput (99):1–1. doi: 10.1109/TEVC.2016.2600642
 24.Li K, Deb K, Zhang Q, Kwong S (2015) An evolutionary manyobjective optimization algorithm based on dominance and decomposition. IEEE Trans Evol Comput 19:694–716CrossRefGoogle Scholar
 25.Li M, Yang S, Liu X (2016) Pareto or nonPareto: bicriterion evolution in multiobjective optimization. IEEE Trans Evol Comput 20(5):645–665CrossRefGoogle Scholar
 26.Jensen MT (2003) Reducing the runtime complexity of multiobjective EAs: the NSGAII and other algorithms. IEEE Trans Evol Comput 7(5):503–515CrossRefGoogle Scholar
 27.Deb K, Tiwari S (2005) Omnioptimizer: a procedure for single and multiobjective optimization. In: Proceedings of the 3rd international conference on evolutionary multicriterion optimization, pp 47–61Google Scholar
 28.Clymont KM, Keedwell E (2012) Deductive sort and climbing sort: new methods for nondominated sorting. Evol Comput 20(1):1–26CrossRefGoogle Scholar
 29.Zhang X, Tian Y, Cheng R, Jin Y (2015) An efficient approach to nondominated sorting for evolutionary multiobjective optimization. IEEE Trans Evol Comput 19(2):201–213CrossRefGoogle Scholar
 30.Wang H, Yao X (2014) Corner sort for Paretobased manyobjective optimization. IEEE Trans Cybern 44(1):92–102CrossRefGoogle Scholar
 31.Zhang X, Tian Y, Jin Y (2016) Approximate nondominated sorting for evolutionary manyobjective optimization. Inf Sci 369:14–33CrossRefGoogle Scholar
 32.Cheng R, Jin Y, Narukawa K, Sendhoff B (2015) A multiobjective evolutionary algorithm using Gaussian process based inverse modeling. IEEE Trans Evol Comput 19(6):838–856CrossRefGoogle Scholar
 33.Deb K, Thiele L, Laumanns M, Zitzler E (2002) Scalable multiobjective optimization test problems. In: Proceedings of the 2002 IEEE congress on evolutionary computation, pp 825–830Google Scholar
 34.Zitzler E, Deb K, Thiele L (2000) Comparison of multiobjective evolutionary algorithms: empirical results. Evol Comput 8(2):173–195CrossRefGoogle Scholar
 35.Ishibuchi H, Tsukamoto N, Nojima Y (2008) Evolutionary manyobjective optimization: a short review. In: Proceedings of the 2008 IEEE congress on evolutionary computation, pp 2419–2426Google Scholar
 36.Laumanns M, Thiele L, Deb K, Zitzler E (2002) Combining convergence and diversity in evolutionary multiobjective optimization. Evol Comput 10(3):263–282CrossRefGoogle Scholar
 37.HernándezDíaz AG, SantanaQuintero LV, Coello CAC, Molina J (2007) Paretoadaptive \(\epsilon \)dominance. Evol Comput 15(4):493–517CrossRefGoogle Scholar
 38.Batista LS, Campelo F, Guimarães FG, Ramírez JA (2011) Pareto cone \(\varepsilon \)dominance: improving convergence and diversity in multiobjective evolutionary algorithms. In: Proceedings of the 2011 international conference on evolutionary multicriterion optimization, pp 76–90Google Scholar
 39.Sato H, Aguirre H, Tanaka K (2007) Controlling dominance area of solutions and its impact on the performance of MOEAs. In: Proceedings of the 2007 international conference on evolutionary multicriterion optimization, pp 5–20Google Scholar
 40.Sato H, Aguirre HE, Tanaka K (2010) Selfcontrolling dominance area of solutions in evolutionary manyobjective optimization. In: SEAL, vol 8, Springer, New York, pp 455–465Google Scholar
 41.Ikeda K, Kita H, Kobayashi S (2001) Failure of Paretobased MOEAs: does nondominated really mean near to optimal? In: Proceedings of the 2001 IEEE congress on evolutionary computation, pp 957–962Google Scholar
 42.Zhu C, Xu L, Goodman ED (2016) Generalization of Paretooptimality for manyobjective evolutionary optimization. IEEE Trans Evol Comput 20(2):299–315CrossRefGoogle Scholar
 43.Farina M, Amato P (2004) A fuzzy definition of “optimality” for manycriteria optimization problems. IEEE Trans Syst Man Cybern Part A Syst Hum 34(3):315–326CrossRefGoogle Scholar
 44.Zou X, Chen Y, Liu M, Kang L (2008) A new evolutionary algorithm for solving manyobjective optimization problems. IEEE Trans Syst Man Cybern Part B 38(5):1402–1412CrossRefGoogle Scholar
 45.Wang G, Jiang H (2007) Fuzzydominance and its application in evolutionary many objective optimization. In: Proceedings of the 2007 international conference on computational intelligence and security workshops, pp 195–198Google Scholar
 46.He Z, Yen GG, Zhang J (2014) Fuzzybased Pareto optimality for manyobjective evolutionary algorithms. IEEE Trans Evol Comput 18(2):269–285CrossRefGoogle Scholar
 47.Elarbi M, Bechikh S, Gupta A, Said LB, Ong YS (2017) A new decompositionbased NSGAII for manyobjective optimization. IEEE Trans Syst Man Cybern Syst PP(99):1–20. doi: 10.1109/TSMC.2017.2654301
 48.Yuan Y, Xu H, Wang B, Yao X (2016) A new dominance relationbased evolutionary algorithm for manyobjective optimization. IEEE Trans Evol Comput 20(1):16–37CrossRefGoogle Scholar
 49.Bader J, Zitzler E (2011) HypE: an algorithm for fast hypervolumebased manyobjective optimization. Evol Comput 19(1):45–76CrossRefGoogle Scholar
 50.Li M, Yang S, Liu X (2014) Shiftbased density estimation for Paretobased algorithms in manyobjective optimization. IEEE Trans Evol Comput 18(3):348–365CrossRefGoogle Scholar
 51.Zhang X, Tian Y, Cheng R, Jin Y (2016) Empirical analysis of a treebased efficient nondominated sorting approach for manyobjective optimization. In: Proceedings of the 2016 IEEE symposium series on computational intelligence, pp 1–8Google Scholar
 52.Kung HT, Luccio F, Preparata FP (1975) On finding the maxima of a set of vectors. J ACM 22:469–476MATHCrossRefMathSciNetGoogle Scholar
 53.Fang H, Wang Q, Tu Y, Horstemeyer MF (2008) An efficient nondominated sorting method for evolutionary algorithms. Evol Comput 16(3):355–384CrossRefGoogle Scholar
 54.Fortin FA, Grenier S, Parizeau M (2013) Generalizing the improved runtime complexity algorithm for nondominated sorting. In: Proceedings of the 15th annual conference on genetic and evolutionary computation, pp 615–622Google Scholar
 55.Drozdík M, Akimoto Y, Aguirre H, Tanaka K (2015) Computational cost reduction of nondominated sorting using Mfront. IEEE Trans Evol Comput 19(5):659–678CrossRefGoogle Scholar
 56.Veldhuizen DAV, Lamont GB (1998) Multiobjective evolutionary algorithm research: a history and analysis. Technical report, Department of Electrical and Computer Engineering. Graduate School of Engineering, Air Force Inst Technol, Wright Patterson, Tech. Rep. TR9803Google Scholar
 57.Zhou A, Jin Y, Zhang Q, Sendhoff B, Tsang E (2006) Combining modelbased and geneticsbased offspring generation for multiobjective optimization using a convergence criterion. In: Proceedings of the 2006 IEEE congress on evolutionary computation, pp 892–899Google Scholar
 58.Deb K, Thiele L, Laumanns M, Zitzler E (2005) Scalable test problems for evolutionary multiobjective optimization. Springer, New YorkMATHCrossRefGoogle Scholar
 59.Huband S, Hingston P, Barone L, While L (2006) A review of multiobjective test problems and a scalable test problem toolkit. IEEE Trans Evol Comput 10(5):477–506MATHCrossRefGoogle Scholar
 60.Wang H, Jiao L, Yao X (2015) Two_arch2: an improved twoarchive algorithm for manyobjective optimization. IEEE Trans Evol Comput 19(4):524–541CrossRefGoogle Scholar
 61.While L, Hingston P, Barone L, Huband S (2006) A faster algorithm for calculating hypervolume. IEEE Trans Evol Comput 10(1):29–38CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.