Deep Policy Dynamic Programming for Vehicle Routing Problems

Abstract

Routing problems are a class of combinatorial problems with many practical applications. Recently, end-to-end deep learning methods have been proposed to learn approximate solution heuristics for such problems. In contrast, classical dynamic programming (DP) algorithms guarantee optimal solutions, but scale badly with the problem size. We propose Deep Policy Dynamic Programming (DPDP), which aims to combine the strengths of learned neural heuristics with those of DP algorithms. DPDP prioritizes and restricts the DP state space using a policy derived from a deep neural network, which is trained to predict edges from example solutions. We evaluate our framework on the travelling salesman problem (TSP), the vehicle routing problem (VRP) and TSP with time windows (TSPTW) and show that the neural policy improves the performance of (restricted) DP algorithms, making them competitive to strong alternatives such as LKH, while also outperforming most other ‘neural approaches’ for solving TSPs, VRPs and TSPTWs with 100 nodes.

Appendices

Appendix 1 The Graph Neural Network Model

For the TSP, we use the exact model from [32], which we describe here for self-containment. The model uses node input features and edge input features, which get transformed into initial representations of the nodes and edges. These representations then get updated sequentially using a number of graph convolutional layers, which exchange information between nodes and edges, after which the final edge representation is used to predict whether the edge is part of the optimal solution.

Input Features and Initial Representation. The model uses input features for the nodes, consisting of the (x, y)-coordinates, which are then projected into H-dimensional initial embeddings $\mathbf {x}_i^{0}$ ($H = 300$). The initial edge features $\mathbf {e}_{ij}^{0}$ are a concatenation of a $\frac{H}{2}$-dimensional projection of the cost (Euclidean distance) $c_{ij}$ from i to j, and a $\frac{H}{2}$-dimensional embedding of the edge type: 0 for normal edges, 1 for edges connecting K-nearest neighbors ($K = 20$) and 2 for self-loop edges connecting a node to itself (which are added for ease of implementation).

Graph Convolutional Layers. In each of the $L = 30$ layers of the model, the node and edge representations $\mathbf {x}_i^{\ell }$ and $\mathbf {e}_{ij}^{\ell }$ get updated into $\mathbf {x}_i^{\ell + 1}$ and $\mathbf {e}_{ij}^{\ell +1}$ [32]:

$$\begin{aligned} \mathbf {x}_i^{\ell + 1}&= \mathbf {x}_i^{\ell } + \text {ReLU} \left( \text {BN}\left( W_1^{\ell } \mathbf {x}_i^{\ell } + \sum _{j \in \mathcal {N}(i)} \frac{\sigma (\mathbf {e}_{ij}^{\ell })}{\sum _{j' \in \mathcal {N}(i)} \sigma (\mathbf {e}_{ij'}^{\ell })} \odot W_2^{\ell } \mathbf {x}_j^{\ell } \right) \right) \end{aligned}$$

(2)

$$\begin{aligned} \mathbf {e}_{ij}^{\ell + 1}&= \mathbf {e}_{ij}^{\ell } + \text {ReLU} \left( \text {BN}\left( W_3^{\ell } \mathbf {e}_{ij}^{\ell } + W_4^{\ell } \mathbf {x}_i^{\ell } + W_5^{\ell } \mathbf {x}_j^{\ell } \right) \right) . \end{aligned}$$

(3)

Here $\mathcal {N}(i)$ is the set of neighbors of node i (in our case all nodes, including i, as we use a fully connected input graph), $\odot$ is the element-wise product and $\sigma$ is the sigmoid function, applied element-wise to the vector $\mathbf {e}_{ij}^{\ell }$. $\text {ReLU}(\cdot ) = \max (\cdot , 0)$ is the rectified linear unit and $\text {BN}$ represents batch normalization [31]. $W_1, W_2, W_3, W_4$ and $W_5$ are trainable parameter matrices, where we fix $W_4 = W_5$ for the symmetric TSP.

Output Prediction. After L layers, the final prediction $h_{ij} \in (0,1)$ is made independently for each edge (i, j) using a multi-layer perceptron (MLP), which takes $\mathbf {e}_{ij}^{L}$ as input and has two H-dimensional hidden layers with $\text {ReLU}$ activation and a 1-dimensional output layer, with sigmoid activation. We interpret $h_{ij}$ as the predicted probability that the edge (i, j) is part of the optimal solution, which indicates how promising this edge is when searching for the optimal solution.

Training. For TSP, the model is trained on a dataset of 1 million optimal solutions, found using Concorde [2], for randomly generated TSP instances. The training loss is a weighted binary cross-entropy loss, that maximizes the prediction quality when $h_{ij}$ is compared to the ground-truth optimal solution. Generating the dataset takes between half a day and a few days (depending on number of CPU cores), and training the model takes a few days on one or multiple GPUs, but both are only required once given a desired data distribution.

1.1 1.1 Predicting Directed Edges for the TSPTW

The TSP is an undirected problem, so the neural network implementation¹⁰ by [32] shares the parameters $W_4^l$ and $W_5^l$ in Eq. (3), i.e. $W_4^l = W_5^l$, resulting in $\mathbf {e}_{ij}^l = \mathbf {e}_{ji}^l$ for all layers l, as for $l = 0$ both directions are initialized the same. While the VRP also is an undirected problem, the TSPTW is directed as the direction of the route determines the times of arrival at different nodes. To allow the model to make different predictions for different directions, we implement $W_5^l$ as a separate parameter, such that the model can have different representations for edges (i, j) and (j, i). We define the training labels accordingly for directed edges, so if edge (i, j) is in the directed solution, it will have a label 1 whereas the edge (j, i) will not (for the undirected TSP and VRP, both labels are 1).

1.2 1.2 Dataset Generation for the TSPTW

We found that using our DP formulation for TSPTW, the instances by [8] were all solved optimally, even with a very small beam size (around 10). This is because there is very little overlap in the time windows as a result of the way they are generated, and therefore very few actions are feasible as most of the actions would ‘skip over other time windows’ (advance the time so much that other nodes can no longer be served)¹¹. We conducted some quick experiments with a weaker DP formulation, that only checks if actions directly violate time windows, but does not check if an action causes other nodes to be no longer reachable in their time windows. Using this formulation, the DP algorithm can run into many dead ends if just a single node gets skipped, and using the GNN policy (compared to a cost based policy as in Sect. 4.4) made the difference between good solutions and no solution at all being found.

We made two changes to the data generation procedure by [8] to increase the difficulty and make it similar to [12], defining the ‘large time window’ dataset. First, we sample the time windows around arrival times when visiting nodes in a random order without any waiting time, which is different from [8] who ‘propagate’ the waiting time (as a result of time windows sampled). Our modification causes a tighter schedule with more overlap in time windows, and is similar to [12]. Secondly, we increase the maximum time window size from 100 to 1000, which makes that the time windows are in the order of 10% of the horizon¹². This doubles the maximum time window size of 500 used by [12] for instances with 200 nodes, to compensate for half the number of nodes that can possibly overlap the time window.

To generate the training data, for practical reasons we used DP with the heuristic ‘cost heat + potential’ strategy and a large beam size (1M), which in many cases results in optimal solutions being found.

Appendix 2 Implementation

We implement the dynamic programming algorithm on the GPU using PyTorch [53]. While mostly used as a Deep Learning framework, it can be used to speed up generic (vectorized) computations.

1.1 2.1 Beam Variables

For each solution in the beam, we keep track of the following variables (storing them for all solutions in the beam as a vector): the cost, current node, visited nodes and (for VRP) the remaining capacity or (for TSPTW) the current time. As explained, these variables can be computed incrementally when generating expansions. Additionally, we keep a variable vector parent, which, for each solution in the current beam, tracks the index of the solution in the previous beam that generated the expanded solution. To compute the score of the policy for expansions efficiently, we also keep track of the score for each solution and the potential for each node for each solution incrementally.

We do not keep past beams in memory, but at the end of each iteration, we store the vectors containing the parents as well as last actions for each solution on the trace. As the solution is completely defined by the sequence of actions, this allows to backtrack the solution after the algorithm has finished. To save GPU memory (especially for larger beam sizes), we store the O(Bn) sized trace on the CPU memory.

For efficiency, we keep the set of visited nodes as a bitmask, packed into 64-bit long integers (2 for 100 nodes). Using bitwise operations with the packed adjacency matrix, this allows to quickly check feasible expansions (but we need to unpack the mask into boolean vectors to find all feasible expansions explicitly). Figure 4a shows an example of the beam (with variables related to the policy and backtracking omitted) for the VRP.

Fig. 4.

Implementation of DPDP for VRP (Color figure online)

1.2 2.2 Generating Non-dominated Expansions

A solution $\boldsymbol{a}$ can only dominate a solution $\boldsymbol{a}'$ if ${\text {visited}}(\boldsymbol{a}) = {\text {visited}}(\boldsymbol{a}')$ and ${\text {current}}(\boldsymbol{a}) = {\text {current}}(\boldsymbol{a}')$, i.e. if they correspond to the same DP state. If this is the case, then, if we denote by ${\text {parent}}(\boldsymbol{a})$ the parent solution from which $\boldsymbol{a}$ was expanded, it holds that

$$\begin{aligned} {\text {visited}}({\text {parent}}(\boldsymbol{a}))&= {\text {visited}}(\boldsymbol{a}) \setminus \{{\text {current}}(\boldsymbol{a})\} \\&= {\text {visited}}(\boldsymbol{a}') \setminus \{{\text {current}}(\boldsymbol{a}')\} \\&= {\text {visited}}({\text {parent}}(\boldsymbol{a}')). \end{aligned}$$

This means that only expansions from solutions with the same set of visited nodes can dominate each other, so we only need to check for dominated solutions among groups of expansions originating from parent solutions with the same set of visited nodes. Therefore, before generating the expansions, we group the current beam (the parents of the expansions) by the set of visited nodes (see Fig. 4). This can be done efficiently, e.g. using a lexicographic sort of the packed bitmask representing the sets of visited nodes¹³.

Travelling Salesman Problem. For TSP, we can generate (using boolean operations) the $B \times n$ matrix with boolean entries indicating feasible expansions (with n action columns corresponding to n nodes, similar to the $B \times 2n$ matrix for VRP in Fig. 4), i.e. nodes that are unvisited and adjacent to the current node. If we find positive entries sequentially for each column (e.g. by calling $\textsc {torch.nonzero}$ on the transposed matrix), we get all expansions grouped by the combination of action (new current node) and parent set of visited nodes, i.e. grouped by the DP state. We can then trivially find the segments of consecutive expansions corresponding to the same DP state, and we can efficiently find the minimum cost solution for each segment, e.g. using torch_scatter¹⁴.

Vehicle Routing Problem. For VRP, the dominance check has two dimensions (cost and remaining capacity) and additionally we need to consider 2n actions: n direct and n via the depot (see Fig. 4). Therefore, as we will explain, we check dominances in two stages: first we find (for each DP state) the single non-dominated ‘via-depot’ expansion, after which we find all non-dominated ‘direct’ expansions (see Fig. 4b).

The DP state of each expansion is defined by the expanded node (the new current node) and the set of visited nodes. For each DP state, there can be only one¹⁵ non-dominated expansion where the last action was via the depot, since all expansions resulting from ‘via-depot actions’ have the same remaining capacity as visiting the depot resets the capacity (see Fig. 4b). To find this expansion, we first find, for each unique set of visited nodes in the current beam, the solution that can return to the depot with lowest total cost (thus including the cost to return to the depot, indicated by a dashed green rectangle in Fig. 4). The single non-dominated ‘via-depot expansion’ for each DP state must necessarily be an expansion of this solution. Also observe that this via-depot solution cannot be dominated by a solution expanded using a direct action, which will always have a lower remaining vehicle capacity (assuming positive demands) as can bee seen in Fig. 4b. We can thus generate the non-dominated via-depot expansion for each DP state efficiently and independently from the direct expansions.

For each DP state, all direct expansions with cost higher (or equal) than the via-depot expansion can directly be removed since they are dominated by the via-depot expansion (having higher cost and lower remaining capacity, see Fig. 4b). After that, we sort the remaining (if any) direct expansions for each DP state based on the cost (using a segmented sort as the expansions are already grouped if we generate them similarly to TSP, i.e. per column in Fig. 4). For each DP state, the lowest cost solution is never dominated. The other solutions should be kept only if their remaining capacity is strictly larger than the largest remaining capacity of all lower-cost solutions corresponding to the same DP state, which can be computed using a (segmented) cumulative maximum computation (see Fig. 4b).

TSP with Time Windows. For the TSPTW, the dominance check has two dimensions: cost and time. Therefore, it is similar to the check for non-dominated direct expansions for the VRP (see Fig. 4b), but replacing remaining capacity (which should be maximized) by current time (to be minimized). In fact, we could reuse the implementation, if we replace remaining capacity by time multiplied by $-1$ (as this should be minimized). This means that we sort all expansions for each DP state based on the cost, keep the first solution and keep other solutions only if the time is strictly lower than the lowest current time for all lower-cost solutions, which can be computed using a cumulative minimum computation.

1.3 2.3 Finding the Top B Solutions

We may generate all ‘candidate’ non-dominated expansions and then select the top B using the score function. Alternatively, we can generate expansions in batches, and keep a streaming top B using a priority queue. We use the latter implementation, where we can also derive a bound for the score as soon as we have B candidate expansions. Using this bound, we can already remove solutions before checking dominances, to achieve some speedup in the algorithm.¹⁶

1.4 2.4 Performance Improvements

There are many possibilities for improving the speed of the algorithm. For example, PyTorch lacks a segmented sort so we use a much slower lexicographic sort instead. Also an efficient GPU priority queue would allow much speedup, as we currently use sorting as PyTorch’ top-k function is rather slow for large k. In some cases, a binary search for the k-th largest value can be faster, but this introduces undesired CUDA synchronisation points.