Skip to main content
SpringerLink
Log in

Measuring precision of modeled behavior

  • Original Article
  • Published:
Information Systems and e-Business Management Aims and scope Submit manuscript

Abstract

Conformance checking techniques compare observed behavior (i.e., event logs) with modeled behavior for a variety of reasons. For example, discrepancies between a normative process model and recorded behavior may point to fraud or inefficiencies. The resulting diagnostics can be used for auditing and compliance management. Conformance checking can also be used to judge a process model automatically discovered from an event log. Models discovered using different process discovery techniques need to be compared objectively. These examples illustrate just a few of the many use cases for aligning observed and modeled behavior. Thus far, most conformance checking techniques focused on replay fitness, i.e., the ability to reproduce the event log. However, it is easy to construct models that allow for lots of behavior (including the observed behavior) without being precise. In this paper, we propose a method to measure precision of process models, given their event logs by first aligning the logs to the models. This way, the measurement is not sensitive to non-fitting executions and more accurate values can be obtained for non-fitting logs. Furthermore, we introduce several variants of the technique to deal better with incomplete logs and reduce possible bias due to behavioral property of process models. The approach has been implemented in the ProM 6 framework and tested against both artificial and real-life cases. Experiments show that the approach is robust to noise and applicable to handle logs and models of real-life complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

Notes

  1. For the reader not familiar with Petri nets, a Petri net is a bipartite graph that contains two types of nodes: places (circles) and transitions (boxes). A place may contain tokens (black dots), and a transition can fire if its predecessor places contain a token. When fired, a transition removes a token from each input place and adds a token to each successor place.

  2. The distance function can be user-defined, but for simplicity we use a default distance function that assigns unit costs to moves in log/model only.

  3. For the sake of readability, in the figures, we use the label abc as an abuse of notation for referring to the sequence \(\langle a,b,c \rangle.\)

  4. Notice that, for the case of Petri nets with one unique initial and final markings, the set of all reversed complete activity sequences can be generated by simulating the behavior of a net obtained from the original net by reversing its arcs and swapping their initial with final marking.

  5. See http://www.healthcare-analytics-process-mining.org/.

References

  • Adriansyah A, van Dongen B, van der Aalst W (2011a) Conformance checking using cost-based fitness analysis. In: Proceedings of the 2011 IEEE 15th international enterprise distributed object computing conference (EDOC ’11), Helsinki, Finland. IEEE Computer Society, pp 55–64

  • Adriansyah A, Sidorova N, van Dongen B (2011b) Cost-based fitness in conformance checking. In: 11th International conference on application of concurrency to system design (ACSD), pp 57–66

  • Adriansyah A, van Dongen B, van der Aalst W (2013a) Memory-efficient alignment of observed and modeled behavior. Technical report BPM-03-03, BPMcenter.org

  • Adriansyah A, Munoz-Gama J, Carmona J, van Dongen B, van der Aalst W (2013b) Alignment based precision checking. In: Rosa ML, Soffer P (eds) Business process management workshops. Lecture notes in business information processing, vol 132. Springer, Berlin, pp 137–149

  • Banescu S, Zannone N (2011) Measuring privacy compliance with process specifications. In: Proceedings of the 2011 third international workshop on security measurements and metrics (METRISEC ’11). IEEE Computer Society, pp 41–50

  • Buijs J, van Dongen B, van der Aalst W (2012a) On the role of fitness, precision, generalization and simplicity in process discovery. In: Meersman R, Panetto H, Dillon T, Rinderle-Ma S, Dadam P, Zhou X, Pearson S, Ferscha A, Bergamaschi S, Cruz I (eds) On the move to meaningful internet systems (OTM 2012). Lecture notes in computer science, vol 7565. Springer, Berlin, pp 305–322

  • Buijs J, van Dongen B, van der Aalst W (2012b) Towards cross-organizational process mining in collections of process models and their executions. In: Daniel F, Barkaoui K, Dustdar S (eds) Business process management workshops. Lecture notes in business information processing, vol 100. Springer, Berlin, pp 2–13

  • Cook J, Wolf A (1999) Software process validation: quantitatively measuring the correspondence of a process to a model. ACM Trans Softw Eng Methodol (TOSEM) 8:147–176

    Article  Google Scholar 

  • de Weerdt J, de Backer M, Vanthienen J, Baesens B (2011a) A critical evaluation study of model-log metrics in process discovery. In: Muehlen M, Su J (eds) Business process management workshops. Lecture notes in business information processing, vol 66. Springer, Berlin, pp 158–169

  • de Weerdt J, de Backer M, Vanthienen J, Baesens B (2011b) A robust F-measure for evaluating discovered process models. In: IEEE symposium series on computational intelligence. IEEE, pp 148–155

  • Dijkman R, Dumas M, Ouyang C (2008) Semantics and analysis of business process models in BPMN. Inf Softw Technol 50(12):1281–1294

    Article  Google Scholar 

  • Dumas M, Rosa ML, Mendling J, Reijers H (2013) Fundamentals of business process management. Springer, Berlin

    Book  Google Scholar 

  • Gerke K, Cardoso J, Claus A (2009) Measuring the compliance of processes with reference models. In: Proceedings of the confederated international conferences, CoopIS, DOA, IS, and ODBASE 2009 on. On the move to meaningful internet systems: part I (OTM ’09). Springer, Berlin, pp 76–93

  • Greco G, Guzzo A, Pontieri L, Sacca D (2006) Discovering expressive process models by clustering log traces. IEEE Trans Knowl Data Eng 18:1010–1027

    Article  Google Scholar 

  • Hofstede A, van der Aalst W, Adams M, Russell N (2010) Modern business process automation. Springer, Berlin

    Book  Google Scholar 

  • Kristensen L, Schmidt K, Valmari A (2006) Question-guided stubborn set methods for state properties. Formal Methods Syst Des 29(3):215–251

    Article  Google Scholar 

  • Munoz-Gama J, Carmona J (2010) A fresh look at precision in process conformance. In: Proceedings of the 8th international conference on business process management (BPM ’10). Springer, Berlin, pp 211–226

  • Munoz-Gama J, Carmona J (2011) Enhancing precision in process conformance: stability, confidence and severity. In: Proceedings of the IEEE symposium on computational intelligence and data mining (CIDM 2011), April 11–15, Paris, France. IEEE, pp 184–191

  • Munoz-Gama J, Carmona J (2012) A general framework for precision checking. Int J Innov Comput Inf Control (IJICIC) 8(7B):5317–5339

    Google Scholar 

  • Murata T (1989) Petri nets: properties, analysis and applications. Proc IEEE 77(4):541–580

    Article  Google Scholar 

  • Pande P, Neuman R, Cavanagh R (2000) The six sigma way: how GE, Motorola, and other top companies are honing their performance. McGraw-Hill, New York

    Google Scholar 

  • Petkovic M, Prandi D, Zannone N (2011) Purpose control: did you process the data for the intended purpose? In: Jonker W, Petkovic M (eds) Secure data management. Lecture notes in computer science, vol 6933. Springer, Berlin, pp 145–168

  • Porter L, Parker A (1993) Total quality managementthe critical success factors. Total Qual Manag 4(1):13–22

    Article  Google Scholar 

  • Rozinat A, van der Aalst W (2008) Conformance checking of processes based on monitoring real behavior. Inf Syst 33:64–95

    Article  Google Scholar 

  • Rozinat A, de Jong I, Günther C, van der Aalst W (2009) Process Mining applied to the test process of wafer steppers in ASML. IEEE Trans Syst Man Cybern Part C 39:474–479

    Article  Google Scholar 

  • Schmidt K (1999) Stubborn sets for standard properties. In: Proceedings of the 20th international conference on application and theory of petri nets. Springer, London, pp 46–65

  • van der Aalst W (2011a) Business process configuration in the cloud: how to support and analyze multi-tenant processes? In: Zavattaro G, Schreier U, Pautasso C (eds) Proceedings of the 9th IEEE European conference on web services (ECOWS 2011). IEEE Computer Society Press, pp 3–10

  • van der Aalst W (2011b) Process mining: discovery, conformance and enhancement of business processes. Springer, New York

    Book  Google Scholar 

  • van der Aalst W, van Hee K (2004) Workflow management: models, methods, and systems. MIT Press, Cambridge, MA

    Google Scholar 

  • van der Aalst W, Rubin V, Verbeek H, van Dongen B, Kindler E, Gunther C (2010) Process mining: a two-step approach to balance between underfitting and overfitting. Softw Syst Model 9(1):87–111

    Article  Google Scholar 

  • van der Aalst W, Adriansyah A, van Dongen B (2011) Causal nets: a modeling language tailored towards process discovery. In: Proceedings of the 22nd international conference on concurrency theory (CONCUR ’11). Springer, Berlin, pp 28–42

  • van der Aalst W, Adriansyah A, van Dongen B (2012) Replaying history on process models for conformance checking and performance analysis. Wiley Interdiscip Rev Data Min Knowl Discov 2(2):182–192

    Article  Google Scholar 

  • vanden Broucke S, de Weerdt J, Baesens B, Vanthienen J (2012) Improved artificial negative event generation to enhance process event logs. In: Ralyt J, Franch X, Brinkkemper S, Wrycza S (eds) Advanced information systems engineering. Lecture notes in computer science, vol 7328. Springer, Heidelberg, pp 254–269

  • vanden Broucke S, de Weerdt J, Vanthienen J, Baesens B (2013) A comprehensive benchmarking framework (CoBeFra) for conformance analysis between procedural process models and event logs in ProM. In: Proceedings of the IEEE symposium on computational intelligence and data mining (CIDM 2013), part of the IEEE symposium series on computational intelligence 2013 (SSCI 2013)

  • Weidlich M, Polyvyanyy A, Desai N, Mendling J (2010) Process compliance measurement based on behavioural profiles. In: Proceedings of the 22nd international conference on Advanced information systems engineering (CAiSE ’10). Springer, Berlin, pp 499–514

  • Weijters A, van der Aalst W, de Medeiros AA (2006) Process mining with the heuristics miner-algorithm. BETA working paper series, WP 166. Technical report, Eindhoven University of Technology, Eindhoven

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, Eindhoven University of Technology, P.O. Box 513, 5600 MB, Eindhoven, The Netherlands

    A. Adriansyah, B. F. van Dongen & W. M. P. van der Aalst

  2. Universitat Politecnica de Catalunya, Barcelona, Spain

    J. Munoz-Gama & J. Carmona

Authors
  1. A. Adriansyah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. Munoz-Gama
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Carmona
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. B. F. van Dongen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. W. M. P. van der Aalst
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. Adriansyah.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Adriansyah, A., Munoz-Gama, J., Carmona, J. et al. Measuring precision of modeled behavior. Inf Syst E-Bus Manage 13, 37–67 (2015). https://doi.org/10.1007/s10257-014-0234-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10257-014-0234-7

Keywords

Search

Navigation