Can recurrent neural networks learn process model structure?

Abstract

Various methods using machine and deep learning have been proposed to tackle different tasks in predictive process monitoring, forecasting for an ongoing case e.g. the most likely next event or suffix, its remaining time, or an outcome-related variable. Recurrent neural networks (RNNs), and more specifically long short-term memory nets (LSTMs), stand out in terms of popularity. In this work, we investigate the capabilities of such an LSTM to actually learn the underlying process model structure of an event log. We introduce an evaluation framework that combines variant-based resampling and custom metrics for fitness, precision and generalization. We evaluate 4 hypotheses concerning the learning capabilities of LSTMs, the effect of overfitting countermeasures, the level of incompleteness in the training set and the level of parallelism in the underlying process model. We confirm that LSTMs can struggle to learn process model structure, even with simplistic process data and in a very lenient setup. Taking the correct anti-overfitting measures can alleviate the problem. However these measures did not present themselves to be optimal when selecting hyperparameters purely on predicting accuracy. We also found that decreasing the amount of information seen by the LSTM during training, causes a sharp drop in generalization and precision scores. In our experiments, we could not identify a relationship between the extent of parallelism in the model and the generalization capability, but they do indicate that the process’ complexity might have impact.

Appendices

Appendix A: Absolute metrics

Complementary to the metrics introduced in in Section 4, we have designed three alternative metrics, which we call the “absolute” metrics and which can be seen below. These are not well calibrated, as they highly depend on the predetermined size of the Tr+Te and the Simulated Log (in this work always taken to be 100 times the number of variants). This is because the values depend on whether a certain variants present in another log at all, without taking multiplicities into account. For this we use the Ex(v, L) function. Subsequently, F_A−PMSL corresponds with the fraction of variants present in the original Training Log that are actually replicated in the Simulated Log. P_A−PMSL then measures how many of the variants produced by the LSTM in the Simulated Log actually corresponds with correct behavior, present in the original Tr+Te Log. Finally, G_A−PMSL just counts how many of the variants in the Test Log are correctly reproduced by the LSTM. In case of the LOVOCV G_A−PMSL is equal to either 0 or 1, since there is only one variant in the Simulated Log.

$$ F_{{A-PMSL}} = \sum\limits_{v\in{Var(Tr)}} \frac{Ex\left( v, Sim\right)}{|{Var(Tr)}} $$

(4)

$$ P_{{A-PMSL}} = \sum\limits_{v\in{Var(Sim)}} \frac{Ex\left( v, Tr+Te\right)}{|{Var(Sim)}|} $$

(5)

$$ G_{{A-PMSL}} = \sum\limits_{v\in{Var(Te)}} \frac{Ex\left( v, Sim\right)}{|{Var(Te)}|} $$

(6)

$$ \text{With: } {Ex}(v, L) = \begin{cases} 1 & \text{if } v \in L\\ 0 & \text{if } v \not\in L \end{cases} $$

(7)

Next to the metrics in the paper itself, these metrics have been applied on the logs, produced by the LSTMs discussed throughout (Tables 6 and 7 and Fig. 5). Using these results we can unveil whether non-optimal results on the original metrics (F_PMSL, P_PMSL and G_PMSL) are due to the LSTM not learning certain behavior at all, or just because it displays lower probabilities to simulate these variants. The lower “absolute” precision scores (P_A−PMSL), combined with the nonetheless high P_PMSL scores, shows that a lot of the LSTMs simulate a lot of different (wrong) variants, which do not correspond with allowed process model behavior; however do this with low multiplicities.

Appendix B: Comparing with inductive miner

To compare the LSTMs’ abilities to understand the six different basic control flow elements introduced in Fig. 3 to “classical” process discovery, we have performed some additional experiments. For this purpose we have used the Inductive Miner (Leemans et al., 2013) as implemented in pm4py (Berti et al., 2019), with the default settings (corresponding to a noise threshold of 0.0) to discover Process Trees. These process Trees are further played out (simulated) with the basic play out function, implemented in pm4py (Berti et al., 2019) as well. We have repeated both the LOVOCV and the fold experiment introduced in Section 5. In the fold experiment we take out an increasing number of variants from the Training Log to group them into the Test Log. We have used both the metrics from (1)–(3) and the “absolute” metrics (4)–(6) introduced in Appendix A. The results of these experiments are visualized in Figs. 6 and 7. We can immediately notice that Model 1, the model with one parallel split, is discovered (and played out) perfectly by the Inductive Miner, and this for both the LOVOCV as for every fold. This is in contrast to the LSTM models, which seem to struggle a bit more with correctly interpreting this kind of behavior. Model 2, introducing multiple XOR splits, also does not cause any issues for the Inductive Miner, with the exception of the 2 fold experiment (where half of the control-flow variants are sampled out of the Training Log). The long-term dependency introduced in Model 3 is not picked up at all by the Process Tree discovered with the Inductive Miner (unsurprisingly). This results in near perfect absolute fitness (4) and generalization (6) scores (the Process Tree is able to play out the behavior), but frequency dependent fitness (1) and generalization (3) scores of around 0.5. Both precision scores are also around 0.5. Remarkably when performing the 2 fold experiment, fitness (and absolute precision) goes to 1.0 and generalization to 0.0. What happens is that the model completely overfits on the possible control-flow variants (one big XOR split) preventing any level of generalization. A similar type of behavior can be noticed for Model 4, which consists out of different inclusive OR splits. The longer parallel tracks of Model 5 are interpreted worse and worse, as more and more variants are left out of the Training Log, leading to a decreasing trend in all metrics. This is comparable to the results of the LSTMs trained on the same model. It is in contrast to the better interpretation of the Inductive Miner of the parallel split in Model 1 (with short 1 activity long parallel tracks). Finally the loops in Model 6 are modeled correctly by the Inductive Miner, where the imperfect frequency based fitness and precision scores are due to the actually infinite amount of possible behavior introduced by loops.

Fig. 6

The framework output, using metrics (1)-(3), as performed on the simulated Logs produced by playing out Process Trees discovered with the Inductive Miner (Leemans et al., 2013), discovered with different amounts of variants left out and put aside in the Test Log

Fig. 7

The framework output, using the alternative absolute metrics (4)-(6), as performed on the simulated Logs produced by playing out Process Trees discovered with the Inductive Miner (Leemans et al., 2013), discovered with different amounts of variants left out and put aside in the Test Log