Process Mining and the Black Swan: An Empirical Analysis of the Influence of Unobserved Behavior on the Quality of Mined Process Models

  • Jana-Rebecca RehseEmail author
  • Peter Fettke
  • Peter Loos
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 308)


In this paper, we present the epistomological problem of induction, illustrated by the metaphor of the black swan, and its relevance for Process Mining. The quality of mined models is typically measured in terms of four dimensions, namely fitness, precision, simplicity, and generalization. Both precision and generalization rely on the definition of “unobserved behavior”, i.e. traces not contained in the log. This paper is intended to analyze the influence of unobserved behavior, the potential black swan, has on the quality of mined models. We conduct an empirical analysis to investigate the relation between a system, its observed and unobserved behavior and the mined models. The results show that the unobserved behavior, mainly determined by the nature of the unknown system, can have a significant impact on the quality assessment of mined models, hence eliciting the need to explicate and discuss the assumptions underlying the notions of unobserved behavior in more depth.


Process mining Process discovery Evaluation metrics Process model quality 


  1. 1.
    van Dongen, B.F., Carmona, J., Chatain, T.: A unified approach for measuring precision and generalization based on anti-alignments. In: La Rosa, M., Loos, P., Pastor, O. (eds.) BPM 2016. LNCS, vol. 9850, pp. 39–56. Springer, Cham (2016). CrossRefGoogle Scholar
  2. 2.
    Buijs, J., van Dongen, B., van der Aalst, W.M.P.: Quality dimensions in process discovery: the importance of fitness, precision, generalization and simplicity. Int. J. Coop. Inf. Syst. 23(01), 1440001 (2014)CrossRefGoogle Scholar
  3. 3.
    Janssenswillen, G., Jouck, T., Creemers, M., Depaire, B.: Measuring the quality of models with respect to the underlying system: an empirical study. In: La Rosa, M., Loos, P., Pastor, O. (eds.) BPM 2016. LNCS, vol. 9850, pp. 73–89. Springer, Cham (2016). CrossRefGoogle Scholar
  4. 4.
    Popper, K.: The Logic of Scientific Discovery. Routledge, London (2005)zbMATHGoogle Scholar
  5. 5.
    Buijs, J.: Flexible evolutionary algorithms for mining structured process models. Ph.D. thesis, Technische Universiteit Eindhoven (2014)Google Scholar
  6. 6.
    Weber, P., Bordbar, B., Tiňo, P., Majeed, B.: A framework for comparing process mining algorithms. In: GCC Conference and Exhibition (GCC), pp. 625–628. IEEE (2011)Google Scholar
  7. 7.
    vanden Broucke, S., De Weerdt, J., Vanthienen, J., Baesens, B.: A comprehensive benchmarking framework (coBeFra) for conformance analysis between procedural process models and event logs in prom. In: Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM 2013), pp. 254–261. IEEE (2013)Google Scholar
  8. 8.
    vanden Broucke, S., De Weerdt, J., Vanthienen, J., Baesens, B.: An improved process event log artificial negative event generator. Feb research report, KU Leuven - Faculty of Economics and Business, Leuven, Belgium (2012)Google Scholar
  9. 9.
    Weijters, A., van Der Aalst, W.M.P., De Medeiros, A.: Process mining with the heuristics miner-algorithm. Technical Report 166, TU Eindhoven (2006)Google Scholar
  10. 10.
    Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Discovering block-structured process models from event logs containing infrequent behaviour. In: Lohmann, N., Song, M., Wohed, P. (eds.) BPM 2013. LNBIP, vol. 171, pp. 66–78. Springer, Cham (2014). CrossRefGoogle Scholar
  11. 11.
    van der Werf, J.M.E.M., van Dongen, B.F., Hurkens, C.A.J., Serebrenik, A.: Process discovery using integer linear programming. In: van Hee, K.M., Valk, R. (eds.) PETRI NETS 2008. LNCS, vol. 5062, pp. 368–387. Springer, Heidelberg (2008). CrossRefGoogle Scholar
  12. 12.
    van der Aalst, W.M.P., Adriansyah, A., van Dongen, B.: Replaying history on process models for conformance checking and performance analysis. Wiley Interdisc. Rev. Data Min. Knowl. Discov. 2(2), 182–192 (2012)CrossRefGoogle Scholar
  13. 13.
    vanden Broucke, S., De Weerdt, J., Vanthienen, J., Baesens, B.: Determining process model precision and generalization with weighted artificial negative events. IEEE Trans. Knowl. Data Eng. 26(8), 1877–1889 (2014)Google Scholar
  14. 14.
    Rozinat, A., van der Aalst, W.M.P.: Conformance checking of processes based on monitoring real behavior. Inf. Syst. 33(1), 64–95 (2008)CrossRefGoogle Scholar
  15. 15.
    Adriansyah, A., Munoz-Gama, J., Carmona, J., van Dongen, B.F., van der Aalst, W.M.P.: Alignment based precision checking. In: La Rosa, M., Soffer, P. (eds.) BPM 2012. LNBIP, vol. 132, pp. 137–149. Springer, Heidelberg (2013). CrossRefGoogle Scholar
  16. 16.
    van der Aalst, W.M.P: Mediating between modeled and observed behavior: the quest for the “right” process. In: IEEE International Conference on Research Challenges in Information Science (RCIS 2013), pp. 31–43 (2013)Google Scholar
  17. 17.
    Rogge-Solti, A., Senderovich, A., Weidlich, M., Mendling, J., Gal, A.: In log and model we trust? A generalized conformance checking framework. In: La Rosa, M., Loos, P., Pastor, O. (eds.) BPM 2016. LNCS, vol. 9850, pp. 179–196. Springer, Cham (2016). CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Institute for Information Systems (IWi) at the German Center for Artificial Intelligence (DFKI GmbH), Saarland UniversitySaarbrueckenGermany

Personalised recommendations