Skip to main content
SpringerLink
Log in

Evaluating Conformance Measures in Process Mining Using Conformance Propositions

  • Chapter
  • First Online:
Transactions on Petri Nets and Other Models of Concurrency XIV

Part of the book series: Lecture Notes in Computer Science ((TOPNOC,volume 11790))

Abstract

Process mining sheds new light on the relationship between process models and real-life processes. Process discovery can be used to learn process models from event logs. Conformance checking is concerned with quantifying the quality of a business process model in relation to event data that was logged during the execution of the business process. There exist different categories of conformance measures. Recall, also called fitness, is concerned with quantifying how much of the behavior that was observed in the event log fits the process model. Precision is concerned with quantifying how much behavior a process model allows for that was never observed in the event log. Generalization is concerned with quantifying how well a process model generalizes to behavior that is possible in the business process but was never observed in the event log. Many recall, precision, and generalization measures have been developed throughout the years, but they are often defined in an ad-hoc manner without formally defining the desired properties up front. To address these problems, we formulate 21 conformance propositions and we use these propositions to evaluate current and existing conformance measures. The goal is to trigger a discussion by clearly formulating the challenges and requirements (rather than proposing new measures). Additionally, this paper serves as an overview of the conformance checking measures that are available in the process mining area.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 16.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Every regular language has a unique minimal DFA according to the Myhill–Nerode theorem.

  2. 2.

    Note that the term “probability” is used here in an informal manner. Since we only have example observations and no knowledge of the underlying (possibly changing) process, we cannot compute such a probability. Of course, unseen cases can have traces that have been observed before.

References

  1. van der Aalst, W.M.P.: Mediating between modeled and observed behavior: the quest for the “Right" process. In: IEEE International Conference on Research Challenges in Information Science, RCIS 2013, pp. 31–43. IEEE Computing Society (2013)

    Google Scholar 

  2. van der Aalst, W.M.P.: Process Mining: Data Science in Action. Springer, Berlin (2016). https://doi.org/10.1007/978-3-662-49851-4

    Book  Google Scholar 

  3. van der Aalst, W.M.P.: Relating process models and event logs: 21 conformance propositions. In: van der Aalst, W.M.P., Bergenthum, R., Carmona, J. (eds.) Workshop on Algorithms & Theories for the Analysis of Event Data, ATAED 2018, pp. 56–74. CEUR Workshop Proceedings (2018)

    Google Scholar 

  4. van der Aalst, W.M.P., Adriansyah, A., van Dongen, B.: Replaying history on process models for conformance checking and performance analysis. WIREs Data Mining Knowl. Discov. 2(2), 182–192 (2012)

    Article  Google Scholar 

  5. van der Aalst, W.M.P., Weijters, A.J.M.M., Maruster, L.: Workflow mining: discovering process models from event logs. IEEE Trans. Knowl. Data Eng. 16(9), 1128–1142 (2004)

    Article  Google Scholar 

  6. Adriansyah, A., van Dongen, B., van der Aalst, W.M.P.: Conformance checking using cost-based fitness analysis. In: Chi, C.H., Johnson, P. (eds.) IEEE International Enterprise Computing Conference, EDOC 2011, pp. 55–64. IEEE Computer Society (2011)

    Google Scholar 

  7. Adriansyah, A., Munoz-Gama, J., Carmona, J., van Dongen, B.F., van der Aalst, W.M.P.: Alignment Based Precision Checking. In: La Rosa, M., Soffer, P. (eds.) BPM 2012. LNBIP, vol. 132, pp. 137–149. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36285-9_15

    Chapter  Google Scholar 

  8. Augusto, A., Armas-Cervantes, A., Conforti, R., Dumas, M., La Rosa, M., Reissner, D.: Abstract-and-Compare: A Family of Scalable Precision Measures for Automated Process Discovery. In: Weske, M., Montali, M., Weber, I., vom Brocke, J. (eds.) BPM 2018. LNCS, vol. 11080, pp. 158–175. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98648-7_10

    Chapter  Google Scholar 

  9. Buijs, J.C.A.M.: Flexible evolutionary algorithms for mining structured process models. Ph.D. thesis, Department of Mathematics and Computer Science (2014)

    Google Scholar 

  10. Buijs, J.C.A.M., van Dongen, B.F., van der Aalst, W.M.P.: On the Role of Fitness, Precision, Generalization and Simplicity in Process Discovery. In: Meersman, R., Panetto, H., Dillon, T., Rinderle-Ma, S., Dadam, P., Zhou, X., Pearson, S., Ferscha, A., Bergamaschi, S., Cruz, I.F. (eds.) OTM 2012. LNCS, vol. 7565, pp. 305–322. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33606-5_19

    Chapter  Google Scholar 

  11. Buijs, J.C.A.M., van Dongen, B.F., van der Aalst, W.M.P.: Quality dimensions in process discovery: the importance of fitness, precision, generalization and simplicity. Int. J. Coop. Inf. Syst. 23(1), 1–39 (2014)

    Article  Google Scholar 

  12. Carmona, J., van Dongen, B., Solti, A., Weidlich, M.: Conformance Checking: Relating Processes and Models. Springer, Berlin (2018). https://doi.org/10.1007/978-3-319-99414-7

    Book  Google Scholar 

  13. van Dongen, B.F., Carmona, J., Chatain, T.: A Unified Approach for Measuring Precision and Generalization Based on Anti-alignments. In: La Rosa, M., Loos, P., Pastor, O. (eds.) BPM 2016. LNCS, vol. 9850, pp. 39–56. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45348-4_3

    Chapter  Google Scholar 

  14. van Dongen, B., Carmona, J., Chatain, T., Taymouri, F.: Aligning Modeled and Observed Behavior: A Compromise Between Computation Complexity and Quality. In: Dubois, E., Pohl, K. (eds.) CAiSE 2017. LNCS, vol. 10253, pp. 94–109. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59536-8_7

    Chapter  Google Scholar 

  15. Garcia-Banuelos, L., van Beest, N., Dumas, M., La Rosa, M., Mertens, W.: Complete and interpretable conformance checking of business processes. IEEE Trans. Softw. Eng. 44(3), 262–290 (2018)

    Article  Google Scholar 

  16. Goedertier, S., Martens, D., Vanthienen, J., Baesens, B.: Robust process discovery with artificial negative events. J. Mach. Learn. Res. 10, 1305–1340 (2009)

    MathSciNet  MATH  Google Scholar 

  17. Greco, G., Guzzo, A., Pontieri, L., Saccà, D.: Discovering expressive process models by clustering log traces. IEEE Trans. Knowl. Data Eng. 18(8), 1010–1027 (2006)

    Article  Google Scholar 

  18. Janssenswillen, G., Depaire, B.: Towards confirmatory process discovery: making assertions about the underlying system. Bus. Inf. Syst, Eng (2018)

    Google Scholar 

  19. Janssenswillen, G., Donders, N., Jouck, T., Depaire, B.: A comparative study of existing quality measures for process discovery. Inf. Syst. 50(1), 2:1–2:45 (2017)

    Google Scholar 

  20. Janssenswillen, G., Jouck, T., Creemers, M., Depaire, B.: Measuring the Quality of Models with Respect to the Underlying System: An Empirical Study. In: La Rosa, M., Loos, P., Pastor, O. (eds.) BPM 2016. LNCS, vol. 9850, pp. 73–89. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45348-4_5

    Chapter  Google Scholar 

  21. Kerremans, M.: Gartner Market Guide for Process Mining, Research Note G00353970 (2018). www.gartner.com

  22. Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth Movers’ Stochastic Conformance Checking. In: Hildebrandt, T., van Dongen, B.F., Röglinger, M., Mendling, J. (eds.) BPM 2019. LNBIP, vol. 360, pp. 127–143. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-26643-1_8

    Chapter  Google Scholar 

  23. Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Scalable process discovery and conformance checking. Softw. Syst. Model. 17(2), 599–631 (2018)

    Article  Google Scholar 

  24. Mannhardt, F., de Leoni, M., Reijers, H.A., van der Aalst, W.M.P.: Balanced multi-perspective checking of process conformance. Computing 98(4), 407–437 (2016)

    Article  MathSciNet  Google Scholar 

  25. Muñoz-Gama, J., Carmona, J.: A Fresh Look at Precision in Process Conformance. In: Hull, R., Mendling, J., Tai, S. (eds.) BPM 2010. LNCS, vol. 6336, pp. 211–226. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15618-2_16

    Chapter  Google Scholar 

  26. Polyvyanyy, A., Solti, A., Weidlich, M., Di Ciccio, C., Mendling, J.: Behavioural quotients for precision and recall in process mining. Technical report, University of Melbourne (2018)

    Google Scholar 

  27. Rozinat, A., van der Aalst, W.M.P.: Conformance checking of processes based on monitoring real behavior. Inf. Syst. 33(1), 64–95 (2008)

    Article  Google Scholar 

  28. Rozinat A., de Medeiros A.K.A., Günther, C.W., Weijters, A.J.M.M., van der Aalst, W.M.P.: The need for a process mining evaluation framework in research and practice. In: Castellanos, M., Mendling, J., Weber, B. (eds.) Informal Proceedings of the International Workshop on Business Process Intelligence, BPI 2007, pp. 73–78. QUT, Brisbane (2007)

    Google Scholar 

  29. Syring, A.F., Tax, N., van der Aalst, W.M.P.: Evaluating Conformance Measures in Process Mining using Conformance Propositions (Extended Version). CoRR, arXiv:1909.02393 (2019)

  30. Tax, N., Lu, X., Sidorova, N., Fahland, D., van der Aalst, W.M.P.: The imprecisions of precision measures in process mining. Inf. Process. Lett. 135, 1–8 (2018)

    Article  MathSciNet  Google Scholar 

  31. vanden Broucke, S.K.L.M., De Weerdt, J., Vanthienen, J., Baesens, B.: Determining process model precision and generalization with weighted artificial negative events. IEEE Trans. Knowl. Data Eng. 26(8), 1877–1889 (2014)

    Article  Google Scholar 

  32. De Weerdt, J., De Backer, M., Vanthienen, J., Baesens, B.: A multi-dimensional quality assessment of state-of-the-art process discovery algorithms using real-life event logs. Inf. Syst. 37(7), 654–676 (2012)

    Article  Google Scholar 

  33. De Weerdt, J., De Backer, M., Vanthienen, J., Baesens, B.: A robust f-measure for evaluating discovered process models. In: Chawla, N., King, I., Sperduti, A. (eds.) IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2011, pp. 148–155. IEEE, Paris (2011)

    Chapter  Google Scholar 

  34. Weijters, A.J.M.M., van der Aalst, W.M.P., de Medeiros, A.K.A.: Process Mining with the Heuristics Miner-algorithm. BETA Working Paper Series, WP 166, Eindhoven University of Technology, Eindhoven (2006)

    Google Scholar 

Download references

Acknowledgements

We thank the Alexander von Humboldt (AvH) Stiftung for supporting our research.

Author information

Authors and Affiliations

  1. Process and Data Science (Informatik 9), RWTH Aachen University, 52056, Aachen, Germany

    Anja F. Syring & Wil M. P. van der Aalst

  2. Architecture of Information Systems, Eindhoven University of Technology, Eindhoven, The Netherlands

    Niek Tax & Wil M. P. van der Aalst

  3. Fraunhofer Institute for Applied Information Technology FIT, Sankt Augustin, Germany

    Wil M. P. van der Aalst

Authors
  1. Anja F. Syring
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Niek Tax
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wil M. P. van der Aalst
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wil M. P. van der Aalst .

Editor information

Editors and Affiliations

  1. Newcastle University, Newcastle upon Tyne, UK

    Maciej Koutny

  2. Università degli Studi di Milano, Milan, Italy

    Lucia Pomello

  3. Western Norway University of Applied Sciences, Bergen, Norway

    Lars Michael Kristensen

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer-Verlag GmbH Germany, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Syring, A.F., Tax, N., van der Aalst, W.M.P. (2019). Evaluating Conformance Measures in Process Mining Using Conformance Propositions. In: Koutny, M., Pomello, L., Kristensen, L. (eds) Transactions on Petri Nets and Other Models of Concurrency XIV. Lecture Notes in Computer Science(), vol 11790. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-60651-3_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-60651-3_8

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-60650-6

  • Online ISBN: 978-3-662-60651-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation