Efficient Order Batching Optimization Using Seed Heuristics and the Metropolis Algorithm

Order Picking in warehouses is often optimized using a method known as Order Batching, which means that one vehicle can be assigned to pick a batch of several orders at a time. There exists a rich body of research on Order Batching Problem (OBP) optimization, but one area which demands more attention is computational efficiency, especially for optimization scenarios where warehouses have unconventional layouts and vehicle capacity configurations. Due to the NP-hard nature of the OBP, computational cost for optimally solving large instances is often prohibitive. In this paper, we compare the performance of two approximate optimizers designed for maximum computational efficiency. The first optimizer, Single Batch Iterated (SBI), is based on a Seed Algorithm, and the second, Metropolis Batch Sampling (MBS), is based on a Metropolis algorithm. Trade-offs in memory and CPU-usage and generalizability of both algorithms is analyzed and discussed. Existing benchmark datasets are used to evaluate the optimizers on various scenarios. On smaller instances, we find that both optimizers come within a few percentage points of optimality at minimal CPU-time. For larger instances, we find that solution improvement continues throughout the allotted time but at a rate which is difficult to justify in many operational scenarios. SBI generally outperforms MBS and this is mainly attributed to the large search space and the latter’s failure to efficiently cover it. The relevance of the results within Industry 4.0 era warehouse operations is discussed.


Introduction
Order Picking is the process in which sets of products (orders) are retrieved from locations in a warehouse. Order Batching is a method in which vehicles can be assigned to pick several orders at a time. Order Batching can be formulated as an optimization problem known as the Order Batching Problem (OBP) [13] or the Joint Order Batching and Picker Router Problem (JOBPRP) [35], where the Picker Router Problem is a Traveling Salesman Problem (TSP) applied in a warehouse environment [28]. We consider the OBP and JOBPRP versions equivalent if solutions to the OBP are assumed to include TSP solutions (henceforth we use the term OBP to refer to this version). There are several other versions and focus areas in OBP's, including dynamicity, traffic congestion, depot setups and obstacle layouts. One focus area is optimization aimed toward maximum computational efficiency. As will be laid out in "Literature Review" section, computational efficiency has both direct and indirect impacts on the quality of warehouse operations. Authors generally consider it to be important, but there are significant differences in how CPUtimes and timeouts are used. Although the variability of OBP versions and corresponding results concerning computational efficiency is high, we believe more research in this domain is warranted. We delimit our work to OBP's where the objective is to minimize aggregate distances, and as measurement of computational efficiency we use the rate with which aggregate distance is reduced through CPU-time. We use the following two OBP optimizers: Single Batch Iterated (SBI) [25,26] and Metropolis Batch Sampling (MBS) (introduced in this paper). We compare the aggregate distance result between the two optimizers, and also compare against results on public OBP benchmark datasets. We use a distance based OBP cost because this is the predominant Key Performance Indicator (KPI) in benchmark datasets. Although a KPI based on capital cost is what is mostly sought by warehouse management, it is more complex: there are a multitude of features that can go into capital, such as time-based aspects of work, traffic congestion, maintenance, ergonomics etc. A distance based KPI allows for a simpler model and a more generalized way in which to reproduce benchmark data and results.
We only work with CPU-times in the range 0-300 s. Results are compared with previous work by Aerts et al. [1] and Henn and Wäscher [16], who have proposed approximate optimization results for sets of smaller instance sizes. For smaller instances, we also assess results against optimal results on the Foodmart dataset [6]. For larger instances we use L09_251 [26]. As far as we are aware, there exists no standard benchmark format in OBP research, rendering experiment reproducibility difficult. Further discussions on how to represent key OBP features in reproducible data is highly relevant. Our research contributions are as follows: 1. An investigation into the importance of computational efficiency in OBP optimization. 2. Experiments regarding computational efficiency of two OBP optimizers on existing test-instances.

Literature Review
In this section, we first present how the OBP and some of its key features are formulated in the literature. Then we present commonly used OBP optimization algorithms and heuristics. Finally, we present how computational efficiency has been motivated and evaluated for different OBP models.
As several studies have pointed out, the Order Batching Problem (OBP) shares significant similarities with the more well-known Vehicle Routing Problem (VRP) [10,34,35]. Aerts et al. [1] distinguish three points of separation between the OBP and a common VRP: 1. Order-integrity constraint: In the OBP, products belonging to an order may only be picked by one vehicle, whereas there exists no concept of orders or order-integrity in the common VRP. 2. Number of visits constraint: In the OBP, the same location may be visited several times by various vehicles, whereas a location may only be visited once in the common VRP. 3. Obstacle-layout: In the OBP it is assumed that there exists an obstacle layout, whereas there is no such assumption in the common VRP.
Concerning the latter point, most of the research on the OBP assumes that the warehouse uses a conventional layout, which means racks are arranged with parallel aisles (between racks) and parallel cross-aisles (between sections of racks) [23]. If these conditions are not met the layout is unconventional (see Fig. 1). Fig. 1 Examples of the conventional (top) and a unconventional (bottom) layout warehouse, and OBP's with four orders from Oxenstierna et al. [26]. The colored diamonds denote origin and destination locations. The colored dots denote products and the orders which they belong to. In the solutions (right of the arrows), one vehicle is assigned to pick the red and lime orders and a second vehicle is assigned to pick the blue and green orders

SN Computer Science
Aerts et al. argue that the OBP can be modeled as a Clustered VRP (CluVRP) with weak cluster constraints. Weak cluster constraints mean that a vehicle may visit the locations in several clusters of locations in any sequence. The CluVRP was first introduced by Defryn and Sörensen [11] and according to Aerts et al. [1] it is equivalent to the OBP since clusters can be mapped as orders. In experiments they utilize this problem on a conventional layout warehouse and on OBP scenarios involving up to 100 orders.
For conventional layouts, proposed optimization algorithms include integer programming [35], clustering [20], datamining [9], dynamic programming [6] meta-heuristics and heuristics. Examples of meta-heuristics include Variable Neighborhood Search [1], Tabu Search [16], Ant Colony Optimization [21] and Genetic Algorithms [8]. The heuristic algorithms can be divided into three categories: priority rule-based algorithms, savings algorithms and seed algorithms [15]. Priority-rule-based algorithms build batches by sorting orders according to a heuristic, for example First-Come-First-Serve, First-Fit or Best-Fit. In savings algorithms batches with single orders are first initialized and evaluated. Then, pairs, triplets and larger batches of orders are constructed and the combination with the best total result is retrieved [16]. In seed algorithms batches are generated in two phases: Seed-selection and order addition. In the first phase, an initial seed order is selected. In order-addition orders are then added to the seed order. There are many choices for seed algorithms, with corresponding advantages and disadvantages depending on the usecase [17,20,31]. One example is the Sequential Minimal Distance (SMD) heuristic [32], where the sum of minimal distances between products in the seed order and remaining orders is computed: where s denotes a seed order in batch b , where o denotes an order which does not exist in b , and where i and j denote products in order s and o , respectively.
Whenever there are more than two products in a batch, some form of TSP optimization is often used within the OBP optimizer. For conventional layouts, the highly efficient S-shape or Largest Gap algorithms are commonly used [14,30]. We are not aware of any attempts to extend these to unconventional layouts. Given a distance matrix is provided, however, TSP's can be optimized reasonably fast using OR-tools [19] or Concorde [2,3]. Concorde, for example, is almost guaranteed to find the shortest path of a TSP with 100 nodes in less than one second [3].
OBP models can be divided into static and dynamic: dynamic models are generally more realistic than static ones (incoming orders are there assumed to be known beforehand). The literature still tends to model OBP's as static since dynamicity incurs more complexity [31]. The importance of computational efficiency in OBP optimization can be derived from two types of factors. The first type has an immediate impact on operations in the warehouse.
As an example, optimization should ideally be faster than the time it takes a vehicle to finish a picking round [14,31]. Otherwise, vehicles must wait in an idle state at the depot while optimization finishes.
The second type concerns a larger perspective with flexible 4.0 industry-era integration and business utility. As an example, if an OBP optimization module is deployed on the cloud as a 3rd party software service (SaaS), a Warehouse Management System (WMS) client may be more interested in buying it if it is safe and simple to integrate. Longer CPUtimes generally make it harder to set up a system (at least as a microservice) so that these two conditions are fulfilled [12]. Furthermore, rental and electricity cost of servers can naturally be assumed to rise with CPU-times [24].
In the broader literature on the OBP, the second type of factors are rarely discussed. CPU times are chosen to be "tolerable" [20], "reasonable" [5] or "acceptable" [1,31], but often lack in concrete explanations of what these terms entail. Some examples are provided below for how researchers have used CPU-times and timeouts in optimization experiments with OBP's.
For approximate optimization, Henn and Wäscher [16] use timeouts between 1 and 180 s for a heuristic optimizer and OBP's where 40-100 unassigned orders are to be batched. Aerts et al. [1], use timeouts between 1 and 60 s on the same instance set and propose a meta-heuristic algorithm specifically designed to terminate at around 60 s, since solution improvement is found to be insignificant beyond that point. Both Aerts et al. and Henn and Wäscher's algorithms come to within 5% of the best solution overall within the first 10% of optimization time. Scholz et al. [31] experiment with instances of similar size but in a dynamic setting and report a much lower efficiency: 70% of maximum allowed CPU-time is necessary to reach within 5% of best solution overall. Efficiency also decreases non-linearly with instance size in their results: for 10 orders, their optimizer needs 2 s, for 100 orders it needs 11 min, and for 200 orders 60 min. Henn [14] also presents an algorithm for dynamic OBP's and sets it to self-terminate after 60 s, partly due to operational considerations (to avoid vehicles from idling at the depot). Many publications do not present concrete results for timeouts or rate of solution improvement, or a low number of experiments [4,7,18]. Kulak et al. [20] and Li et al. [21], for example, present highly efficient meta-heuristic optimizers, but only on 5-10 instances, and do not include rate of solution improvement in their results. For authors presenting algorithms capable of finding optimal solutions to static OBP's, Henn and Wäscher [16] set timeouts between 2 and 1328 s for instances with up to 60 orders. Gademann et al. [13], set timeouts to 10-30 min for up to 100 orders. Valle et al. [35] and Briant et al. [6], on the Foodmart dataset, present timeouts in the range 300 s to 2 h to obtain optimal results for 20-45 orders.
These examples show that computational efficiency in OBP optimization is difficult to judge generally. Choice of static or dynamic modeling, optimal versus approximative optimization, experimental setup, instance sizes and the technology level of used software and hardware, are all aspects that can have a complex effect on results in this regard.

Problem Formulation
We define the OBP objective as the assignment of batches to vehicles such that the aggregate distance needed to pick the batches is minimized. Each batch b consists of a set of one or several orders b ∈ 2 O , where each o ∈ O is a subset of products o ∈ 2 P . Each product p ∈ P is a set which includes a unique product identifier, an order identifier, weight w and volume vol , w, vol ∈ ℝ + . The sum of weight, volume or number of orders in a batch can be retrieved with function q(b), q ∈ w, vol, k . The x , y location coordinates of all products is defined as set L P , and the location of a product is retrievable with function l(p) . The locations of the products in an order are retrievable with function l(o) = ∪ p∈o l(p), and all locations in a batch are retrievable with function l(b) = ∪ o∈b l(o) . We define a single origin location for all vehicles l s , a single destination location l d and a set of polygonal obstacle location sets L U . The aggregate of all locations is We build undirected graph G = (V, E). Each vertex in V represents a unique location in L and function v(l) gives a vertex for a location. The vertices in batch b include the origin and desti- represents the set of all Euclidean edges between all locations that circumvent obstacles in L U . Distance matrix D and shortest paths between all edges is computed using the Floyd-Warshall algorithm. How E and shortest paths can be constructed with polygonal obstacles is beyond the scope of this paper; for details see Ref. [29]. We also permit a surjective relationship of products to locations, i.e., several types of products can be stored at the same location and the location represents several real locations in the warehouse. This can be useful to help reduce the memory footprint of G . The path to pick batch b is retrievable with the following function: and represents the solution to a Traveling Salesman Problem (TSP). The distance of T(b) is retrievable with function Vehicles are defined as m ∈ M where each vehicle has capacities expressed in weight w , volume vol and number of orders k . The scenario where a vehicle m is assigned a batch, order, and/or product location is defined with binary variables x mb , x mo and x ml , respectively. We then use the following OBP formulation: where (4) states the objective, i.e., minimize distances for all generated batches B , where (5) enforces order-integrity, where (6) enforces all locations in all orders to be visited at least once and where (7) ensures vehicle capacities are never exceeded. Since this OBP is highly intractable we also use a less ambitious objective in the single batch OBP: where the aim is to find a single batch for an already selected vehicle. For this case we also enforce the single batch to come as close as possible to vehicle capacity:

Single Batch Iterated (SBI)
SBI (Algorithm 1) is a heuristic multi-phase optimizer. In the core of the algorithm unassigned orders O are iteratively sent as input to the SMD (Sequential Minimal Distance) function, together with distance matrix D , a randomly chosen available vehicle and a semi-stochastic seed order index. The SMD function builds a single batch b by first selecting a seed order according to the seed index and then adds orders to it according to minimal distances (Eq. 1). Batch b is then removed from the set of unassigned orders and the procedure repeats until all orders have been batched into B . An approximate solution to the OBP (Eq. 4) is thus obtained by pre-selecting vehicles and approximately solving a single batch OBP for each vehicle (Eq. 8).
Algorithm 1: Single Batch Iterated (SBI) Number of iterations ( N ) is used here for brevity and in the implementation ("Experiments" section) a time-based condition is used to stop the outer loop. The purpose of the i index is to reduce the probability that the same solution is obtained multiple times, and, if used, it can be set to N = |O| , for example. The paths to visit all locations in batches b ∈ B , T(b) and their distance, ∑ b∈B D(b)x mb , is computed using the ORtools TSP optimization suite 1 (in the TSP function). OR-tools is set to finish quickly using a number-of-iterations parameter, which is set to grow linearly with number of vertices in the TSP. In the update_best function, the aggregate distance between the new OR-tools cost ( y ) is compared against the best OR-tools cost obtained so far ( ŷ ). If the new cost is lower, the TSP's are optimally solved using Concorde 2 and if this is better than the previous Concorde best, the result is stored as the new best in y * .
Since the number of SMD computations between orders is approximately cubic to number of orders, , we use an SMD order-order enumerated matrix which is populated through the optimization procedure: if SMD between two orders does not exist in the matrix, it is computed and pushed to the matrix. Once the value is stored it is subsequently queried. Caching SMD's reduces number of calls to SMD from cubic to square, at an insignificant increase in memory usage (~ 25 megabytes for 5000 orders assuming 8 bits per cell in the matrix). It should be noted that this only works for an SMD algorithm where the seed is defined as a single order, which cannot provide more than a noisy estimate of the subsequent TSP solution distance for batches with more than two orders. We still deem pairwise order-order SMD caching suitable, since distance estimates are inaccurate even if SMD's for larger collections of orders are computed (TSP optimization is required for accurate estimates). Caching could also be used to store all generated single batches and their solved TSP's in a hash tree or equivalent, to prevent the same TSP to be optimized twice (memoization).
We leave an implementation of this for future work. One potential issue with SBI is its reliance on the SMD heuristic. Although SMD makes sure the distance between orders is always minimized for a given batch, the number of orders to select from decreases through the single-batch while-loop in Algorithm 1. Hence, the last batch which is created in the while-loop can be assumed to be of worse quality in terms of distance minimization relative to the first.

Metropolis Batch Sampling (MBS)
MBS is a heuristic multi-phase optimizer which uses distance matrix D , the Concorde TSP solver (in the TSP function below) and the SMD heuristic to compute distance between orders. The main difference between SBI and MBS is that the latter only uses the SMD function to produce an initial solution. A Metropolis algorithm [22] is then used to improve on it using the following procedure: Algorithm 2: Metropolis Batch Sampling The upper while-loop is equivalent to the one in Algorithm 1. The lower for-loop consists of a Metropolis algorithm where each new sample is drawn from a previous one. The function new_sample(B i ) uses the following stationary distribution to describe the probability for a given new sample: where C and P are constants and where the H d B i , B i+1 function denotes the number of swapping operations needed to obtain B i+1 from B i . A swapping operation is defined as a switch of position of two orders in an enumerated set of batches. Since number of swaps to go from B i to B i+1 is always equal to number of swaps to go from B i+1 to B i , the q distribution is symmetrical, i.e., q B i+1 |B i = q B i |B i+1 . A swap is only permitted if vehicle weight and volume capacity constraints are not broken.
The TSP's of the batches in the new sample are then solved using Concorde ("Literature Review" section) in the TSP function, and the aggregate cost is stored in y i+1 . The accept probability α is computed based on the following balance condition [33]: If y i+1 < y i the new sample is always accepted. If y i+1 ≥ y i the sample may still be accepted if a uniform random value is less than α . α depends on the ratio y i ∕y i+1 .
Contrary to SBI, the search space of MBS is guaranteed to include the global optimum, provided the sampling function can output any B that does not break constraints and enough computational time. This may just as well be a liability, however, since the search space may be too large for the algorithm to see optimization gains within reasonable time. We add bias parameter γ ∈ ℝ + to allow for experiments where the search space of the algorithm is more restricted. Without the use of γ , the probability is high that the algorithm steps away from the SBI local minimum B 1 in the very first iteration (which is likely to happen if y 1 ∕y 2 is close to one).
The best sample is assumed to be stored throughout the optimization procedure (sample storage is omitted from the pseudo-code). Just as with Algorithm 1, a number of iterations parameter N ∈ ℤ + is shown as stopping condition in Algorithm 2 for brevity, but in the experiments in "Experiments" section, a CPU-time condition is used instead. Establishing a suitable N for converge is possible by studying covariance of samples, but it is challenging in the OBP case: B i is a set of orders where the orders may contain a variable number of products at variable locations. Heuristics would thus be needed to quantify covariance between two samples.

Benchmark Datasets
The publicly shared OBP datasets Foodmart, 3 L6_203 4 and L09_251 5 are used for experimentation. Foodmart was introduced by Valle et al. [35] and models a warehouse with a conventional layout and a maximum of 8 aisles and 3 crossaisles. A feature in Foodmart is that vehicles carry bins and that vehicle capacity is expressed as a volume unit per bin. If an order cannot fit in a single bin, splitting it between different bins is permitted. SBI and MBS are not designed for this feature (it constitutes an extra bin packing problem within the OBP), so a greedy heuristic module is attached to the optimizers for the Foodmart experiment (for details see Ref. [25]). L6_203 and L09_251 model scenarios for one conventional and up to six unconventional warehouse layouts and multiple depots. In these instances, vehicle capacity is expressed in number of orders and total number of orders generally range between 4 and 50. All but 6 Foodmart instances also fit within this range. For total number of orders in the range 50-1000 we use L09_251. Number

Experiment Results
Aggregations of all results are presented in Figs. 2, 3, Tables 1 and 2 (Appendix). In Fig. 2, the relative improvement rates from the baseline are shown for the two optimizers and four instance size classes. The shades around the lines represent 95% confidence intervals.
In terms of rate of solution improvement, SBI performs stronger than MBS across all instance size classes, and the difference grows with instance size. The MBS results shown are for parameter values C = 0.1 , P = 1 , γ = 10 ("Metropolis Batch Sampling (MBS)" section), retrieved from early testing. A γ value of 1 (the standard Metropolis algorithm), yields weaker results.

SN Computer Science
Overall, the solution improvement rates for the two smaller instance classes (blue and orange) corroborate those of Henn and Wäscher [16] and Aerts et al. [1]: improvements are significant in the initial stage of optimization (1-4% improvement over baseline within the first 10% of optimization) and then taper off. In our case all instances with up to 100 orders require no more than 2 s to obtain the SBI baseline.
The Foodmart instances (all except 6) fit within the smallest class and there we compare our strongest results against optimal results in Briant et al. [6]: on average a gap to optimality of 2.3% is achieved after a maximum of 10 s. The average gap between the baseline solution and the best solution found is 3.2% on Foodmart. On generated instances in L6_203 the corresponding gap is 3.5%. For the larger two instance classes (> 4 s to find a baseline solution), the pattern is similar, but more time is needed to reach the same percentage improvement over the baseline.
In terms of absolute distance rate of improvement, we first standardize the data such that the average pick round is of similar length between the three datasets. The absolute distance improvements for the four instance size classes are shown in Fig. 3. One observable pattern in Fig. 3 is that larger OBP's tend to see more solution improvement. One explanation for this is that the probability of finding a strong baseline decreases with larger instances, and possible optimization gains can therefore be assumed to also be larger. The red curves (> 7 s), for example, yield weaker results in the beginning and stronger results in the end, relative to the green curves (4-7 s). It takes around 4 min to get there, however. Since there are only 56 instances in the red class, more data would be needed to investigate this pattern further and to narrow the confidence intervals.  The best achieved rates of solution improvement decrease to less than < 1%/minute after the initial gains taper off (after 30-60 s for the larger instances in Fig. 2). In terms of standardized distance, this is on average equivalent to around 18% of the length of a single batch TSP solution (~ 12 standardized distance units).
As discussed in "Literature Review" section, judgment of results in light of previous work is challenging due to the high variability of OBP models. Overall, we believe 1% / min is a slow rate of improvement and that it would be difficult to justify in many OBP scenarios, especially when considering the advantages of short CPU-times discussed in "Literature Review" section.
There are several possible reasons why MBS performs worse than SBI. One is that the relatively general application of the Metropolis algorithm faces a search space which is far too large for it to adequately sample within the allotted time. Even with a high amount of extra bias, imposed through manually added parameter , MBS is not able to find and improve on samples faster than SBI. The purely heuristic and more biased SBI optimizer has no global optimum guarantee due to the SMD heuristic, but it instead guarantees that each sample is a relatively strong local minimum. Using the semistochastic seed index within the SMD function ("Single Batch Iterated (SBI)" section) also makes sure that the SBI local minimums are uncorrelated to some extent. Another possible advantage of SBI is that it uses an approximate TSP optimizer to filter out promising samples before solving them optimally. Tests show that both TSP optimizers (Concorde and OR-tools) perform relatively similar in terms of CPU-time on the given instance set: its TSP's are often quite short (5-20 locations) and there is a significant amount of static CPU-time software overhead relative to the actual TSP optimization for these cases. The main optimization result is that the SMD heuristic proves useful, at least in terms of computational efficiency, the way it is used within the SBI optimizer and for the OBP version at hand.

Conclusion
We investigated computational efficiency in approximate Order Batching Problem (OBP) optimization. In previous work, computational efficiency has not been given enough attention, especially when considering unconventional warehouse layouts and vehicle types [1]. It is an important topic that affects operational costs both directly and indirectly, however. In experiments we studied the computational efficiency of two approximate optimizers, Single Batch Iterated (SBI) and Metropolis Batch Sampling (MBS). They both begin by obtaining an initial solution using the Sequential Minimal Distance (SMD) heuristic. SBI then improves on this solution by rerunning the SMD selection procedure using a semi-stochastic seed-order index, whereas MBS improves on the initial solution using a Metropolis algorithm.
For OBP instances with up to 100 orders and a few seconds of CPU-time, both optimizers yield distances only a few percentage points higher than results obtained at timeout (or optimal results where such are available). The result corroborates previous research claims: fast approximate optimization is a practicable choice in common OBP scenarios [5,20].
For larger instances, with 100-1000 orders, more time is required to obtain similar optimization gains. The standardized absolute distance saved through the optimization procedure grows similarly for all instance sizes. In SBI's case this can be explained since weak batches (with products located far from each other) are only constructed whenever there are few orders left to select from (SMD prevents this in other cases). This phenomenon occurs an equal number of times regardless of instance size and the amount of possible solution improvement in larger instances is thus relatively low. MBS does not face this particular issue, but on the other hand it has no mechanism to reduce the vast search space. MBS generally performs worse than SBI within the 5 min timeout, particularly on larger instances.
Regardless of instance size, we conclude that that the value in spending significantly more CPU-time to obtain a result a few percentage points better than a baseline, must be weighed against the less measurable and indirect costs that come with lower computational efficiency. Although several authors have problematized large CPU-time requirements for OBP optimization [5,20,34], it is challenging to judge optimization efficiency generally due to the large variability of OBP use cases ("Literature Review" section).
For future work, we believe the investigation can be widened to include more optimizers. MBS could be replaced by similar but more biased MCMC algorithms, such as Simulated Annealing [27] or Basin Hopping [36]. Heuristics to add even more bias to these algorithms might be needed, however. Examples include mode-jumping [33] and restarts [37], which prevent convergence on local minima. Also, the number of required samples ( N ) for convergence could be estimated for various MCMC algorithms by calculating covariance between generated samples. That way, maximum CPU-time can be set in a more informed manner. For SBI, the Sequential Minimal Distance (SMD) heuristic could be replaced by alternatives which may be more suitable for the unconventional layout. We believe there are significant savings to be made in optimization if more memory is allocated to store and reuse parts of expensive computations. Modeling of OBP's and data-driven performance evaluation are also of primary importance. Currently, there exists no standard format for OBP benchmark datasets and this poses a serious threat to scientific reproducibility. Since there are many possible versions of OBP's, the community needs to discuss how a standard format for OBP SN Computer Science benchmark data can be designed to balance realism with simplicity and reproducibility.