Evolutionarybased tailoring of synthetic instances for the Knapsack problem
 53 Downloads
Abstract
The assessment of strengths and weaknesses of a solver is often limited by the diversity of the cases where it is tested upon. As such, it is paramount to have a versatile tool which finds the problem instances where such a solver excels/fails. In this manuscript, we propose to use an evolutionary algorithm for creating this tool. To validate our approach, we conducted several tests on four heuristics for the knapsack problem. Although, the process can be extended to other domains with relatively few changes. The tests cover different sets of instances, both favoring the performance of one heuristic while hindering that of the remaining ones, and vice versa. To further test our evolutionarybased model, we also apply it on a recent approach that combines the strengths of different heuristics to improve its performance (usually referred to as a hyperheuristic). We show that it is possible to tailor instances in which even this more complex model excels/fails. Throughout our approach, a researcher can test a solver under different kinds of scenarios, delving deeper into the conditions that make it perform well/poorly. Therefore, we recommend using the proposed approach as a means to grasp better insights about strengths and weaknesses of different solvers.
Keywords
Evolutionary computation Knapsack problem Instance generationList of abbreviations
 MIPLIB
Mixed integer programming library
 Def
Default heuristic
 MaP
Max profit heuristic
 MPW
Max profit per Weight heuristic
 MiW
Minimum weight heuristic
 LHS
Latin hypercube sampling
 HH
Hyperheuristic
1 Introduction
The knapsack problem (KP) is a fundamental and extensively studied problem in combinatorial optimization, not only for its theoretical interest but also because of its many practical applications. This problem is classified as NPhard, and several exact and approximate algorithms have been developed to solve it (Gao et al. 2014; Furini et al. 2017; Mavrotas et al. 2015). KP has multiple variants, as described in the works of Martello and Toth (1990) and Kellerer et al. (2004).
Knapsack problems are important, among other things, because they can be used to model certain kinds of realworld problems (Zitzler and Thiele 1999; Samavati et al. 2017), including budgeting, cargo loading and cutting stock (Martello and Toth 1990). A recent example of the many applications of this problem is described in the work of Simon et al. (2017), where a group of people must decide which items to carry and how to distribute them. The authors modeled their problem as multidimensional (dKP) and multiple knapsack problems. Li et al. (2016) illustrate another feasible scenario, using a multiple knapsack model to solve the optimization problem of a sensor network. A third example, in yet another field, rests in the work of Szkaliczki et al. (2014), where solutions related to the KP were introduced to study a problem related to streaming of layered video contents over peertopeer networks.
Finding new strategies for solving the knapsack problem, as well as other problems, requires a set of suitable instances where these methods can be tested. For example, Marinakis and Marinaki (2014) used two different benchmarks to test the performance of a metaheuristic on the openvehicle routing problem: a traditional one, proposed by Christofides (1979); and a more recent one, proposed by Li et al. (2007). Similarly, Ariyasingha and Fernando (2015) tested different versions of the multiobjective ant colony optimization (MOACO) algorithm on instances of the traveling salesman problem presented by Reinelt (1991).
Although the past years have witnessed a remarkable improvement in the quality of the solving methods, benchmark instances have remained almost unaltered. Having access to challenging instances may help researchers to identify critical areas where improvements to solvers can take place. In this regard, realworld scenarios are an ideal but scarce source of test problems. Nonetheless, another feasible source rests in the usage of synthetic instances.
Some of the instances that have been used as benchmarks in the knapsack problem include the work of Martello and Toth (1990). Here, the authors introduced a set of 0/1 KP instances with different numbers of items, and a high correlation between weights and profits of the items. Another important benchmark source is the ORLibrary, which was created by Beasley (1990) for distributing test data for operations research (OR) problems, including the multiple knapsack problem. Azad et al. (2014) recently used this library for assessing the performance of a metaheuristic in 0/1 multidimensional knapsack problems. Yet another example of benchmarks is presented in the work by Zitzler and Thiele (1999), where the authors generated data for the multiobjective KP, by using random integers for weights and profits. This dataset has been widely used in other works on evolutionary multiobjective optimization (Zitzler et al. 2001; Lust and Teghem 2012; Knowles and Corne 2000).
In (Martello et al. 2000), the generated instances were divided into different classes. Some of those instances contained specific ratios of different kinds of items, while others had items created by generating random numbers within specific ranges. For one specific class –“allfill”, as the authors named it—they used a recursive process to create instances starting from the solution and going all the way back to the items. Similarly, Pisinger (2005) produced several sets of instances, considered hard, by using a random generator that manipulates the correlation between profit and weight of the items. Another example corresponds to the mixed integer programming library (MIPLIB) (Koch et al. 2011), originally proposed in 1992. The current version dates back to 2010 and contains several types of domains, including KP. More recently, Petursson and Runarsson (2016) implemented an instance generator for a multidimensional KP. Although the authors included parameters such as the tightness ratio for controlling the properties of the generated instances, they still relied on a uniform distribution for the generation of the weights and profits of the items.
As mentioned before, synthetic instance generation for various types of KP instances has been explored in the past. However, in most of the approaches available in the literature, parameters such as the profit and the weight of items are based on randomly distributed values. Moreover, empirical evidence suggests that it may not be enough to generate instances with arbitrary properties, but to generate them in such a way that they can be used to obtain valuable information from the behavior of specific solvers. This cannot be done with the aforementioned techniques, since instance generation is independent from any solving strategy.
We consider that the discussed approach deals with the problem in a reactive way, by creating problems and then assessing if solvers were able to perform well/poorly. Thus, in this manuscript we propose a more proactive approach. Our idea considers an evolutionarybased model that finds instances in which the solvers exhibit a desired behavior. This approach may allow for future research about the strengths and weaknesses of said solvers.
This need for reliable test instances that exploit specific weaknesses of the solving methods is not new and has received an increasing attention in recent years. A recurring idea in different domains is to use an ‘intelligent’ generator based on evolutionary computation, creating models that facilitate the analysis of each method. This model has led to interesting results. A clear example found in the literature relates to constraint satisfaction problems (CSPs), where van Hemert (2003, 2006) showed how to use an evolutionary algorithm to detect hardtosolve instances. His evolutionary algorithm maintained a population of binary CSPs. Their structure changed over time, and the genetic operators altered conflicting pairs between two values of two variables. The set of variables and their domains were kept unchanged. SmithMiles et al. (2010); SmithMiles and van Hemert (2011) proposed an evolutionary algorithm for producing distinct classes of traveling salesman problem (TSP) instances that are intentionally easy or hard for certain algorithms. In their analysis, a comprehensive set of features was used to characterize the instances. After analyzing the performance of those algorithms over the set of newly created instances, the authors proposed a highlevel algorithm for predicting the search effort, as well as the algorithm likely to perform best over a set of unseen instances, which exhibited high accuracy.
Even so, and to the best of our knowledge, the use of evolutionary computation to generate instances tailored for specific solvers of the KP remains unexplored. Based on this, we asked whether it was possible to use a genetic algorithm for producing synthetic 0/1 KP instances where one or more solvers outperform the others. By doing so, we can fill the knowledge gap regarding the generation of tailored KP instances, while providing a tool for creating test beds with increased/decreased difficulty for a particular kind of solver. Therefore, we developed an evolutionary model powered by a genetic algorithm. We used such a model within the KP, striving to generate sets of instances tailored to four different heuristics, as well as to a highlevel solver that combines the four of them.
Research carried out in this work yields an important contribution: a reliable technique for generating KP instances that exploit specific weaknesses or strengths of particular solvers. We thoroughly explore the behavior of this approach by analyzing the performance of four traditional heuristics on a large number of 0/1 KP instances. To further this exploration, we implement the recently proposed highlevel solver known as a hyperheuristic, which combines other solvers for tackling each problem, and we test our generation technique on it. By doing so, we aim to generate both, (1) instances where the hyperheuristic excels and single heuristics fail and (2) instances where the hyperheuristic fails but single heuristics excel. This generation is performed to determine whether the proposed approach can adapt to a more complex solution method.
The remainder of this paper is organized as follows. Section 2 describes the fundamental ideas that support this work. The evolutionarybased generator of KP instances is detailed in Sect. 3. Section 4 is reserved for presenting the methodology followed in this work. In Sect. 5, we present our experiments and main results, along with their discussion. Finally, the conclusions and overview of future work is presented in Sect. 6.
2 Fundamentals
2.1 Instances considered for this work
 Traditional instances:

Refer to those widely available in the literature and that can be used as a reference point for comparing our model. In this work, we selected some of the instances discussed by Pisinger (2005)^{1}. The author generated such instances through a random model based on the correlation of the weights and profits of items within an instance.
 Tailored instances.

Relate to those generated in this work, following the proposed model. We generated several sets of instances with specific behaviors^{2} (for details, please refer to Sect. 4). In this way, we can isolate a given solver, providing insights for a better understanding of the conditions where it performs appropriately. It is worth remarking that the generated instances were created in such a way that they could be solved through dynamic programming, so that their global optimum can be known.
2.2 Instance characterization
We require a set of features that allow us to characterize and understand the nature of a given instance. Thus, in this work we considered seven features, calculated over the set of unpacked items in the instance: \(\bar{w}\) (mean weight value, divided by the maximum weight), \(\widetilde{w}\) (median weight value, divided by the maximum weight), \(\sigma _{w}\) (standard deviation of the weights, divided by the maximum weight), \(\bar{p}\) (mean profit value, divided by the maximum profit), \(\widetilde{p}\) (median profit value, divided by the maximum profit), \(\sigma _{p}\) (standard deviation of the profits, divided by the maximum profit) and r (weight–profit correlation, divided by two and shifted upwards by 0.5). It is worth noting that all these features were selected for this work based on empirical findings.
The values that these features can take lie in the [0, 1] range. For example, consider a subset of remaining items whose weights are (2, 2, 3, 4) and whose profits are (10, 5, 6, 15), respectively. Thus, each of the aforementioned features (in order) will take the values of \((0.69,0.63,0.24,0.60,0.53,0.30,0.69)\). Moreover, features are dynamic, i.e., they change as the instance is solved.
2.3 Available solvers
In this work, we focused on two kinds of solvers: lowlevel heuristics, which are fast and simple but do not offer any way of adapting to problems; and highlevel ones, known as hyperheuristics, which strive to learn the behavior of a problem so they can predict how to best solve it. In the first case, we included four approaches that operate over the set of unpacked items in the KP instance: Default (Def), which selects the first available item; Max Profit (MaP), which selects the item with the largest profit; Max Profit per Weight (MPW), which selects the item with the largest profit over weight ratio; and min weight (MiW), which selects the item with the minimum weight.
A recent approach for problem solving is the use of selection hyperheuristics (Burke et al. 2013): algorithms for selecting algorithms. The idea behind these highlevel solvers is that a problem can be characterized by a set of domainspecific features, so that each instance of the problem can be placed as a point within the hyperspace. Furthermore, hyperheuristics benefit when the following happens: first, each instance can be solved best with one specific lowlevel heuristic (or with some combination of them), creating zones of action for each solver; second, such behavior can be mapped back to the features. Thus, a selection module can learn the pattern of a specific domain, creating a set of rules for dividing the hyperspace into zones where a given algorithm must be applied. There are different variants and kinds of hyperheuristics, but a detailed description of them is beyond the scope of this manuscript. We will only mention that the type of hyperheuristic we considered for this work begins solving a problem with a specific heuristic (e.g., Def), and may switch to another one later on (e.g., MiW). For more details, the interested reader is welcomed to consult (Burke et al. 2013; Drake et al. 2014) and (Drake et al. 2016).
Throughout this work, we used hyperheuristics that could select among these four single heuristics (Def, MaP, MPW, and MiW). We followed the hyperheuristic model previously described in (OrtizBayliss et al. 2016), where a messy genetic algorithm finds a set of rules that determine when to use one particular heuristic, based on a set of features that characterize the current problem state. The idea in this model is to minimize the error associated to the set of rules. Thus, at each iteration, the genetic algorithm evaluates a candidate solution (a set of rules) over a training subset of instances and determines the corresponding error level. Using this information, the algorithm improves the population and moves on to the next iteration. At the end, the algorithm reports a set of rules which perform well over the training set, and that can be used for solving new instances. Details regarding the inner workings of the genetic algorithm are presented below.
2.4 Genetic algorithms
Genetic algorithms (GAs) are an optimization tool inspired in how evolution occurs in nature (Goldberg 1989; Holland 1975). One element of particular interest in the nature of GAs is their competitive behavior, since members within the population are always struggling to determine who is worthy of progressing to the next phase of evolution. The generic version of a steadystate genetic algorithm works as follows. The algorithm produces a randomly initialized population of individuals (potential solutions or configurations) and runs for a given number of iterations. In our case, individuals in the population are represented as binary strings. The initial population is evaluated according to a fitness function. At each iteration, two individuals are randomly selected from the population and the one with the best fitness is selected as the first parent. This procedure is repeated to select a second parent. Both parents are then used for creating two offspring by using genetic operators (crossover and mutation). The offspring is appended to the population, and the two individuals with the worst fitness are removed from the population. This process is repeated until a termination criterion is met. When the process is over, the GA returns the individual with the best fitness value.
3 The evolutionarybased instance generator
The generator proposed in this investigation relies on a GA that evolves the structure of potential KP instances, guided by a fitness function. The GA modifies the weight and profit of all items until it finds a configuration where the solvers exhibit the desired behavior.
To produce an instance, four arguments are required: the capacity of the knapsack, W; the number of items, n; the maximum weight per item, w; and the maximum profit per item, p. The process starts with a population of randomly initialized binary strings. From this point on, we will refer to these binary strings as chromosomes.
Each chromosome encodes a KP instance of n items. Each item in the encoded instance is represented by \(l_{w} + l_{p}\) bits, where \(l_{w} = \lceil \log _{2}(w)  1 \rceil \) and \(l_{p} = \lceil \log _{2}(p)  1 \rceil \). Therefore, the length of the chromosome, l, is given by \(n \times (l_{w} + l_{p})\). Figure 1 depicts an example of a chromosome generated with \(n=5\), \(w=4\), and \(p=8\), which encodes a KP instance of five items, with weights and profits of (3, 1, 2, 1, 4) and (8, 6, 4, 5, 2), respectively.
3.1 Genetic operators
For this investigation, we have considered three genetic operators: selection, crossover and mutation. The selection process involves a tournament selection of size TS. For each iteration, the algorithm randomly selects TS chromosomes and keeps the one with the highest fitness. The process is repeated to select a second parent. With a probability CR, the crossover operator is applied to the parents to produce two offspring. To perform crossover, one point (\(c_{x}\)) is randomly selected. The offspring will be formed according to the following strategy: the first offspring will contain the first \(c_{x}\) bits of the first parent and the last \(l  c_{x}\) bits of the second one. Conversely, the second offspring will contain the first \(c_{x}\) bits of the second parent and the last \(l  c_{x}\) bits of the first one. After crossover has occurred, mutation may be applied. With a probability MR, the mutation operator is applied to each of the offspring. When mutation takes place, one randomly selected position of the chromosome is flipped.
3.2 The fitness function
Throughout this work, we consider the generation of two kinds of instances: easytosolve and hardtosolve. The generation, thus, focuses on favoring (or hindering) the performance of a given solver (in this case, one of the lowlevel heuristics from Sect. 2.3), relative to the remaining ones. Nonetheless, our decision does not limit the scope of the proposal, since the kind of instances generated mainly depends on the functions described herein. Therefore, different behaviors can be implemented so that instances with a different nature can be generated.
4 Methodology
We divided our research into a threestage work, as shown in Fig. 2. A description of each stage is provided below. It is important to remark that, since each instance may have a different optimum, we opted for analyzing the quality of the solution relative to the best known solution. (In this work, the optimum solution was obtained via dynamic programming.) Therefore, results for the next section will range from zero to one, depending on how close they were from the known optimum.
4.1 Preliminary tests
Throughout this stage, we focused on testing out the feasibility of our approach. Thus, experiments were designed to cover a wide range of behaviors, but at the same time keeping in mind that tests could not be exhaustive. Therefore, we generated instances for all heuristics with an arbitrary problem configuration (knapsack capacity of 50 and 20 items, where each item could have maximum profit and weight of 100 and 10 units, respectively), and with fixed parameters for the generator (mutation rate of 0.1, crossover rate of 1.0, population size of 10 and tournament size of 2). We ran 60 repetitions for each configuration and demanded instances that were easy (and hard) to solve with each heuristic. This yielded a total of 480 different instances (60 runs, 8 configurations), which were then solved with each heuristic to verify that each heuristic performed appropriately over their corresponding sets. Afterward, we used principal component analysis (PCA) over all the data (considering all seven features), creating a twodimensional plot of the sets and trying to determine trends. Furthermore, we included the data from four reference sets, extracted from the literature (Pisinger 2005) and which are considered to be hardtosolve and mapped them into the reduced feature space to verify their location relative to the one of recently generated instances.
4.2 Initial testing
As a second stage, we decided to improve upon the parameters of the genetic algorithm. Therefore, we used different configurations and mapped the performance of our generation model for each of them, seeking to select a proper set of parameters for each problem scenario. Since our model is stochastic in nature, it must be repeated several times to determine an average performance. However, the idea is that this procedure can also be used for future research (so that parameters of the generation model can be tuned) and, therefore, its computational cost should not be too high. Hence, we decided to run the instance generator 30 times for each configuration. To calculate the average performance of the generator, many more instances must be generated. In fact, the number of runs of the genetic algorithm for this stage considerably increased.
Ranges and number of points considered for each parameter of the evolutionary strategy. The total number of combinations is 3200
Parameter  Range  Number of points 

Population size (PS)  [10, 150]  8 
Crossover rate (CR)  (0, 1.00]  10 
Mutation rate (MR)  (0, 0.02]  10 
Tournament size (TS)  [2, 5]  4 
With the aim of delving deeper within the adaptive capabilities of the proposed model, for this stage we requested instances with the following problem characteristics: knapsack capacity of 25 and 40 items, with maximum profit of 100 and maximum weight of 20 each. We used, once again, PCA to reduce the number of dimensions from seven to two. Nonetheless, we carried out a new analysis since the configuration of the problem was altered.
4.3 Advanced tests
The final stage of testing focused on exploring the scalability of the proposed instance generator. Thus, we selected the best configuration yielded by the previous stage and used it to generate 200 instances per heuristic. Again, all eight scenarios were considered (four heuristics, with easy and hard instances for each one), so a total of 1600 instances were created. In this case, the problem configuration of the previous stage was preserved, as we strove to determine feature regions where each heuristic performs poorly and/or appropriately.
Aside from analyzing feature distribution across all instances (using PCA), in this stage we also analyzed the performance of a modern highlevel solver. For this work, we decided to use a hyperheuristic model able to select among all base heuristics (see Sect. 2.3). In theory, during the learning phase the solver must be able to identify which heuristic performs best for each scenario, so that a similar scenario in the test phase is solved appropriately. Moreover, we created 200 easytosolve and 200 hardtosolve instances for the hyperheuristic. This was done to analyze the location of instances that the hyperheuristic can properly solve, by combining tools that are unsuited for the task. But, it was also done to analyze instances where a combination of appropriate tools leads to an unsatisfactory result.
As a last test, we selected two hardtosolve problem configurations available in the literature (for 20 and 50 items) and generated easy and hard instances for each solver. Then, we plotted the main information of the generated instances against that of the ones used as a reference and analyzed the performance of each solver over all sets (the ones generated and the ones selected as reference).
5 Results and discussion
5.1 Preliminary tests
As mentioned in Sect. 4, the first step we took was to verify that the proposed strategy was, indeed, able to generate instances tailored to a given solver. To that end, we show the performance of each heuristic over every set of freshly generated instances (Fig. 3). There is one subplot for each targeted heuristic (highlighted bars), and each one contains two groups of bars: one focused on evolving easytosolve instances and one focused on evolving hardtosolve instances. It is interesting to see that our model performs appropriately, being able to generate instances with a performance gap across solvers and either favoring or hindering them, according to what the user desires, all while using the same problem parameters.
It is also astonishing to note that, in some cases, the generated instances can be extremely difficult for a given heuristic. For example, for the Default heuristic, case (a), the average performance was lowered below the 0.05 mark, but the remaining ones had a much better performance (above 0.95). Nonetheless, some solvers are way more robust, making it easy to find instances where they outperform the competition, but hard to find ones where they perform poorly. An example of this idea is exhibited by the Max Profit per Weight heuristic, case (c), where its average performance for easytosolve instances was more than 0.50 higher than the second best heuristic, but whose average performance for hardtosolve instances was only about 0.08 lower than the second worst heuristic.
In order to verify whether the instances we generate are different from those available in the literature, we selected four sets with different number of elements which are considered to be hardtosolve and plotted the performance of each heuristic over them. We can observe that the instances we generated allow for a better differentiation in the performance of the solvers (Fig. 4). Thus, we have evidence suggesting that the proposed approach could be a fruitful path to pursue.
5.2 Initial testing
Motivated by these results, we ran a set of tests to verify whether the performance was dependent on the problem parameters. We used the following problem configuration: knapsack capacity of 100 and 100 items, with maximum profit of 100 and maximum weight of 100 each. Data show interesting behaviors (Fig. 8). The first thing to note is that, again, it is extremely straightforward to find easy/hard instances for some heuristics and that scenarios have opposite behaviors. For example, consider the generation process of hardtosolve instances for the Def heuristic: even with the worst configuration, the genetic algorithm is capable of producing instances whose normalized average profit differed by more than 0.99 from the closest heuristic. Nonetheless, the opposite scenario (generating easytosolve instances for Def) was not as effortless, and achieving the desired behavior depended on properly selecting a parameter configuration for the solver (negative values in Fig. 8 indicate that behavior was not as desired). Moreover, if the problem parameters are not adequate (as in this example) it becomes impossible for the algorithm to perform appropriately in some scenarios, even at its best configuration (e.g., for MPW, where in all cases a negative delta was achieved).
Best and worst configurations for each heuristic, considering easy (E) and hard (H) to solve instances. Def: Default, MaP: Max Profit, MPW: Max Profit per Weight, MiW: Min Weight. PS: Population size, CR: Crossover rate, MR: Mutation rate, TS: Tournament size
Type  Best  Worst  

PS  CR  MR  TS  PS  CR  MR  TS  
Def (E)  36  0.772  0.065  5  41  0.317  0.001  3 
MaP (E)  139  0.720  0.047  3  123  0.368  0.109  3 
MPW (E)  157  0.709  0.153  4  64  0.851  0.095  4 
MiW (E)  47  0.238  0.032  4  49  0.837  0.020  4 
Def (H)  30  0.650  0.173  5  41  0.317  0.001  3 
MaP (H)  95  0.230  0.169  4  48  0.350  0.020  2 
MPW (H)  25  0.636  0.051  3  38  0.051  0.156  2 
MiW (H)  99  0.709  0.153  4  62  0.126  0.143  2 
5.3 Advanced testing
An interesting behavior that surfaces once more is the tradeoff between easytosolve and hardtosolve instances, since whenever it is easy to achieve big gaps in one of them, it becomes difficult to achieve them for the other one. But, perhaps more interesting is the behavior of MPW. This heuristic was the only one for which it was easier to create easytosolve instances, thus leading to wider gaps of performance (when compared against the remaining solvers). We consider that this can be due to the fact that this is the only heuristic that combines two metrics (profit and weight), while the other ones only consider one at a time (order, profit, or weight), or maybe because MPW is a more general purpose heuristic that works well under a broader set of scenarios. We deem necessary a deeper analysis of these ideas, which was not done here since it deviated from the focus of the manuscript.
It is also alluring to analyze that, because hard instances for one solver can be rather easy for another one, it is common that they share some portions of the feature range. Moreover, and due to the way in which PCA works, it is also possible that the easy and hardtosolve regions for a particular heuristic overlap, at least in some parts (Fig. 11). Nonetheless, one of the main ideas that can be extracted from these maps is that our proposed approach generates instances whose features are located in a zone where the selected solver performs as desired. A second one is that each solver exhibits a different behavior when migrating from easytohard instances. In the case of Def and MaP, the movement goes from left to right, upwards and downwards, respectively. In the case of MPW and MiW, the movement goes from right to left, downwards and upwards, respectively. We think that a deeper study of this phenomenon could help in designing new features for the KP.
5.3.1 Behavior of a highlevel solver
We wanted to go a bit further in our testing, so we used our approach to generate easy and hard instances for the highlevel solver. It is worth remarking that, for these tests, we used the best already trained solver and focused on generating instances for it instead of training a new solver at each iteration, which could lead to some kind of race condition between both evolutionary approaches. Nonetheless, a careful analysis of such scenario seems indeed interesting, and it could be studied in a future work.
Using our evolutionary approach, we developed instances where the best single heuristic was, on average, 15% below the theoretical solution while the highlevel solver was virtually perfect. This information is available in Fig. 12a. In fact, such a hyperheuristic was able to find the theoretical optimum for 196 instances (i.e., 98% of the instances). On the other hand, an ideal heuristic selector (i.e., an Oracle that is able to perfectly predict the best heuristic for each instance) was unable to perfectly solve even a single instance. This means that no single heuristic is good at solving any of those instances, while combining them leads to finding a perfect solution most of the time. In fact, the average normalized profit for the Oracle sat at 0.8473, while that of the hyperheuristic sat at 0.9994, meaning that the latter yielded about 15% more of the theoretical profit than the former. We also evolved, at will, sets where the opposite happened, having a heuristic with virtually perfect performance (i.e., MPW, which was able to find an average profit of 98.8% of the theoretical one.), but where the highlevel solver fell 50% short, on average, of the best possible profits.
We also noted an interesting behavior of instances, since the migration from easytosolve instances to hardtosolve ones implied a movement from right to left in the reduced feature space (as with MPW and MiW). However, as seen in Fig. 12b, there was almost no movement in the vertical direction. Instead, hardtosolve instances distributed themselves over a more elongated region, as if trying to circle around easytosolve instances. We analyzed the distribution of features for each subset of instances, and we found that only the first three, which were related to the profit, varied. This indicates that by changing the profit of items within instances, it can become easier (or harder) for the highlevel solver to behave properly.
5.4 An example about how to transfer these ideas to other domains
6 Conclusion and future work
Throughout this work we focused on using an evolutionary approach as the cornerstone for creating a tool that may help in better understanding solvers from a particular problem domain. In particular, our proposed tool can generate easytosolve and hardtosolve instances for a given heuristic. This decision, though, does not limit the generality of our approach, since other objective functions can be used for evolving instances with diverse behaviors (see Sect. 3.2). In the first case, i.e., easytosolve instances, our tool provided instances where the desired solver virtually reached the theoretical optimum. In fact, the worst case scenario was for the min weight (MiW) heuristic, which yielded an average profit of 98.9% of the theoretical profit (calculated through dynamic programming). In the second case, i.e., hardtosolve instances, our tool exhibited a more varied performance. The best case scenario was for the Default (Def) heuristic, which was only able to achieve a 5.7% of the maximum theoretical profit. The worst case scenario, however, was for the Max Profit per Weight heuristic, where we could only hinder performance in little more than 5% of the maximum theoretical profit.
Features are not required by our proposed approach, even though they could be used in case the solver requires them (e.g., in the case of hyperheuristics). Even so, we did use seven different features for analyzing the behavior of each set of generated instances. We found interesting patterns. For example, in the case of the Default (Def) and Max Profit (MaP) heuristics, increasing the difficulty of instances implied a shift from the left side of the feature space to the right one, as well as upwards and downwards displacements (respectively). However, for the two remaining heuristics, the migration implied a shift in the opposite direction (i.e., from right to left), as well as a steeper vertical displacement (downwards for Max Profit per Weight and upwards for Min Weight). What seems more interesting about these patterns is that they remained even after changing the problem configuration to two previously reported ones. We deem as relevant an indepth study of this phenomenon (see Sect. 5.3), e.g., by using the approach we proposed in this work to generate instances with different levels of difficulty for each heuristic.
After testing our generation model with problem configurations readily available in the literature, we noticed that it was able to improve upon the already published data. For the two examples we selected, we evolved instances where each solver became virtually perfect (they found the theoretical optimum in all tested cases). We also evolved instances where most solvers were unable to yield a profit higher than 30% of the theoretical value. (The only exception being for the Max Profit per Weight heuristic, where performance was hindered in about 15%.) These data are certainly interesting, since this implies that our generation model can be coupled with previously published ideas to further improve the instances.
We also noticed that certain problem configurations undermined the evolution of instances with the desired behavior (even after selecting a suitable solver configuration). For example, we observed that capacity of the knapsack was an important factor for achieving hard instances for the Max Profit per Weight (MPW) heuristic, since it modified the relationship between the number of items to be evolved and the maximum number of items that could be packed, hence affecting the shape of the objective function. Thus, carrying out an indepth study of parameter dependence is paramount for better understanding the knapsack problem. Furthermore, having a tool such as the one presented in this work could make the task an easier and more structured process, so we recommend using this approach for such an endeavor. As an illustrative case of what could be done in a future work, the reader may consider modifying the objective function to force instances with specific parameter ratios, or allowing for the evolutionary algorithm to directly optimize the capacity, or even allowing for some kind of coevolutionary scheme. Another important avenue for future work, besides expanding the generator to other domains, resides in furthering the study of scalability for the presented generator, by creating considerably larger instances (with thousands of items). Likewise, the generator can be extended to include different versions of the knapsack problem, e.g., by considering multiple knapsacks and/or multiple objectives.
In light of the recent literature, thinking about highlevel solvers most of the time implies dealing with features of a specific problem domain. Therefore, the tool presented herein could also be used to improve the performance of highlevel solvers and advance that field, by generating instances with specific behaviors and with given feature values. This will facilitate an analysis of the efficiency with which features can map instances to specific solvers, and will undoubtedly lead to a better understanding of the elements that make a given heuristic perform well or poorly. For example, we evolved instances where a highlevel solver was able to perfectly solve 98% of the instances, even though the base heuristics that composed it were unable to perfectly solve any instance at all. Thus, it is evident that there is a benefit from dynamically combining different solvers for tackling each instance. Moreover, the tool presented here can be extrapolated to other domains, so their benefits could be ripped across different optimization problems.
Our approach exhibits some points in favor, but it also has some drawbacks. Regarding the former, we have shown that our model successfully tailors instances that enhance or lessen performance of a given solver. But, it does not stop there. Our approach can evolve instances for more complex solvers. As an example, we targeted hyperheuristics, which combine different solvers into a single one. Additionally, since we use an evolutionary approach for tailoring instances, users can provide different objective functions, customizing instances with different behaviors. Another benefit of using our approach is that different genetic operators can be used to expand its benefits. Also, our approach can be easily expanded to other problem domains. Regarding drawbacks, they mainly arise from not including the knapsack capacity nor the number of items into the encoding of the chromosome. They also cover the fact that the user cannot decide the degree of difficulty that an instance exhibits (though this could be solved by changing the objective function). Nonetheless, we made these decisions because they simplified the approach. We also considered that, by tailoring instances this way, it may be easier to arrive at conclusions about the influence that profit and weight hold over each solver. Future works should focus on improving these drawbacks so that we can provide a more robust tool.
Footnotes
 1.
These instances can be downloaded from http://www.diku.dk/~pisinger/.
 2.
These instances, including their optimum, are available upon request.
Notes
Acknowledgements
This research was supported in part by Consejo Nacional de Ciencia y Tecnología (CONACyT) Basic Science Project [Grant Number 241461 and 287479] and ITESM Research Group with Strategic Focus in intelligent Systems. C.A. Coello Coello gratefully acknowledges support from CONACyT grant no. 2016011920 (Investigacion en Fronteras de la Ciencia 2016).
Compliance with ethical standards
Conflict of interest
Luis Fernando PlataGonzález declares that he has no conflict of interest. Ivan Amaya declares that he has no conflict of interest. José Carlos OrtizBayliss declares that he has no conflict of interest. Santiago Enrique ConantPablos declares that he has no conflict of interest. Hugo TerashimaMarín declares that he has no conflict of interest. Carlos A. Coello Coello declares that he has no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
References
 Amaya I, OrtizBayliss JC, ConantPablos SE, TerashimaMarín H, Coello Coello CA (2018) Tailoring instances of the 1D bin packing problem for assessing strengths and weaknesses of its solvers. In: Auger A, Fonseca CM, Lourenço N, Machado P, Paquete L, Whitley D (eds) Parallel problem solving from nature PPSN XV, lecture notes in computer science, vol 11101. Springer, Cham, pp 373–384. https://doi.org/10.1007/9783319992594_30 CrossRefGoogle Scholar
 Ariyasingha IDID, Fernando TGI (2015) Performance analysis of the multiobjective ant colony optimization algorithms for the traveling salesman problem. Swarm Evol Comput 23:11–26. https://doi.org/10.1016/j.swevo.2015.02.003 CrossRefGoogle Scholar
 Azad MAK, Rocha AMAC, Fernandes EMGP (2014) Improved binary artificial fish swarm algorithm for the 0–1 multidimensional knapsack problems. Swarm Evol Comput 14:66–75. https://doi.org/10.1016/j.swevo.2013.09.002 CrossRefGoogle Scholar
 Beasley J (1990) ORlibrary: distributing test problems by electronic mail. J Oper Res Soc 41(11):1069–1072CrossRefGoogle Scholar
 Burke EK, Hyde M, Kendall G, Ochoa G (2013) Hyperheuristics : a survey of the state of the art. J Oper Res Soc 64(12):1695–1724CrossRefGoogle Scholar
 Christofides N (1979) The vehicle routing problem. Comb Optim 315–338Google Scholar
 Drake JH, Hyde M, Ibrahim K, Ozcan E (2014) A genetic programming hyperheuristic for the multidimensional knapsack problem. Kybernetes 43(9/10):1500–1511. https://doi.org/10.1108/K0920130201 CrossRefGoogle Scholar
 Drake JH, Özcan E, Burke EK (2016) A case study of controlling crossover in a selection hyperheuristic framework using the multidimensional knapsack problem john. Evol Comput 24(1):113–141CrossRefGoogle Scholar
 Furini F, Ljubić I, Sinnl M (2017) An effective dynamic programming algorithm for the minimumcost maximal knapsack packing problem. Eur J Oper Res 262(2):438–448MathSciNetCrossRefzbMATHGoogle Scholar
 Gao J, He G, Liang R, Feng Z (2014) A quantuminspired artificial immune system for the multiobjective 0–1 knapsack problem. Appl Math Comput 230:120–137MathSciNetzbMATHGoogle Scholar
 Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Addison Wesley, BostonzbMATHGoogle Scholar
 Holland JR (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann ArborGoogle Scholar
 Kellerer H, Pferschy U, Pisinger D (2004) Knapsack problems, vol 1. Springer, New YorkCrossRefzbMATHGoogle Scholar
 Knowles JD, Corne DW (2000) Approximating the nondominated front using the pareto archived evolution strategy. Evol Comput 8(2):149–172CrossRefGoogle Scholar
 Koch T, Achterberg T, Andersen E, Bastert O, Berthold T, Bixby RE, Danna E, Gamrath G, Gleixner AM, Heinz S, Lodi A, Mittelmann H, Ralphs T, Salvagnin D, Steffy DE, Wolter K (2011) MIPLIB 2010. Math Program Comput 3(2):103–163. https://doi.org/10.1007/s1253201100259 MathSciNetCrossRefGoogle Scholar
 Li F, Golden B, Wasil E (2007) The open vehicle routing problem: algorithms, largescale test problems, and computational results. Comput Oper Res 34(10):2918–2930. https://doi.org/10.1016/j.cor.2005.11.018 CrossRefzbMATHGoogle Scholar
 Li H, Yao T, Ren M, Rong J, Liu C, Jia L (2016) Physical topology optimization of infrastructure health monitoring sensor network for highspeed rail. Measurement 79:83–93CrossRefGoogle Scholar
 Lust T, Teghem J (2012) The multiobjective multidimensional knapsack problem: a survey and a new approach. Int Trans Oper Res 19(4):495–520MathSciNetCrossRefzbMATHGoogle Scholar
 Marinakis Y, Marinaki M (2014) A bumble bees mating optimization algorithm for the open vehicle routing problem. Swarm Evol Comput 15:80–94. https://doi.org/10.1016/j.swevo.2013.12.003 CrossRefzbMATHGoogle Scholar
 Martello S, Toth P (1990) Knapsack problems: algorithms and computer implementations. John Wiley & Sons, HobokenzbMATHGoogle Scholar
 Martello S, Pisinger D, Vigo D (2000) The threedimensional bin packing problem. Oper Res 48(2):256–267MathSciNetCrossRefzbMATHGoogle Scholar
 Mavrotas G, Florios K, Figueira JR (2015) An improved version of a core based algorithm for the multiobjective multidimensional knapsack problem: a computational study and comparison with metaheuristics. Appl Math Comput 270:25–43MathSciNetGoogle Scholar
 OrtizBayliss JC, TerashimaMarín H, ConantPablos SE (2016) Combine and conquer: an evolutionary hyperheuristic approach for solving constraint satisfaction problems. Artif Intell Rev 46(3):327–349CrossRefGoogle Scholar
 Petursson KB, Runarsson TP (2016) An evolutionary approach to the discovery of hybrid branching rules for mixed integer solvers. In: Proceedings—2015 IEEE symposium series on computational intelligence, SSCI 2015. pp 1436–1443Google Scholar
 Pisinger D (2005) Where are the hard knapsack problems? Comput Oper Res 32(9):2271–2284MathSciNetCrossRefzbMATHGoogle Scholar
 Reinelt G (1991) TSPLIBA traveling salesman problem library. ORSA J Comput 3(4):376–384. https://doi.org/10.1287/ijoc.3.4.376 CrossRefzbMATHGoogle Scholar
 Samavati M, Essam D, Nehring M, Sarker R (2017) A methodology for the largescale multiperiod precedenceconstrained knapsack problem: an application in the mining industry. Int J Prod Econ 193:12–20CrossRefGoogle Scholar
 Simon J, Apte A, Regnier E (2017) An application of the multiple knapsack problem: the selfsufficient marine. Eur J Oper Res 256(3):868–876MathSciNetCrossRefzbMATHGoogle Scholar
 SmithMiles K, van Hemert J (2011) Discovering the suitability of optimisation algorithms by learning from evolved instances. Ann Math Artif Intell 61(2):87–104MathSciNetCrossRefzbMATHGoogle Scholar
 SmithMiles K, van Hemert J, Lim X (2010) Understanding tspdifficulty by learning from evolved instances. In: Blum C, Battiti R(eds) Learning and intelligent optimization, Lecture notes in computer science, vol 6073. Springer, Berlin, pp 266–280Google Scholar
 Szkaliczki T, Eberhard M, Hellwagner H, Szobonya L (2014) Piece selection algorithms for layered video streaming in P2P networks. Discret Appl Math 167:269–279MathSciNetCrossRefzbMATHGoogle Scholar
 van Hemert JI (2003) Evolving binary constraint satisfaction problem instances that are difficult to solve. In: Proceedings of the 2003 IEEE congress on evolutionary computation (CEC’03), IEEE Press, pp 1267–1273Google Scholar
 van Hemert JI (2006) Evolving combinatorial problem instances that are difficult to solve. Evol Comput 14(4):433–462CrossRefGoogle Scholar
 Zitzler E, Thiele L (1999) Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Trans Evol Comput 3(4):257–271CrossRefGoogle Scholar
 Zitzler E, Laumanns M, Thiele L (2001) SPEA2: improving the strength pareto evolutionary algorithm. TIK–report, vol 103, pp 95–100. https://doi.org/10.3929/ethza004284029
Copyright information
OpenAccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.