OTM 2017: On the Move to Meaningful Internet Systems. OTM 2017 Conferences pp 298-305 | Cite as
Ranking-Based Evaluation of Process Model Matching
(Short Paper)
Conference paper
First Online:
Abstract
Process model matching refers to the automatic detection of semantically equivalent or similar activities between two process models. The output of process model matchers is the basis for many advanced process model analysis techniques and, therefore, must be as accurate as possible. Measuring the performance of process model matchers, however, is a difficult task. On the one hand, it is hard to define which correspondences are actually correct. On the other hand, it is challenging to appropriately take the output of matchers into account, because they often produce confidence values between zero and one. In this paper, we propose the first evaluation procedure for process model matchers that addresses both of these challenges. The core idea is to rank both the computed and the desired correspondences based on their confidence values and compare them using the Spearman’s rank correlation coefficient. We perform an in-depth evaluation in which we apply the new evaluation procedure and illustrate how it helps gaining interesting insights.
Keywords
Process model matching Ranking-based evaluation Non-binary gold standardReferences
- 1.Achichi, M., Cheatham, M., Dragisic, Z., Euzenat, J., Faria, D., Ferrara, A., Flouris, G., Fundulaki, I., Harrow, I., Ivanova, V., Jiménez-Ruiz, E., Kuss, E., Lambrix, P., Leopold, H., Li, H., Meilicke, C., Montanelli, S., Pesquita, C., Saveta, T., Shvaiko, P., Splendiani, A., Stuckenschmidt, H., Todorov, K., Trojahn, C., Zamazal, O.: Results of the ontology alignment evaluation initiative 2016. In: CEUR Workshop Proceedings, vol. 1766, pp. 73-129. RWTH (2016)Google Scholar
- 2.Antunes, G., et al.: The process model matching contest 2015. In: 6th International Workshop on Enterprise Modelling and Information Systems Architectures (2015)Google Scholar
- 3.Cayoglu, U., Oberweis, A., Schoknecht, A., Ullrich, M.: Triple-s: A matching approach for Petri nets on syntactic, semantic and structural level. Technical report Karlsruhe Institute of Technology (KIT) (2013)Google Scholar
- 4.Cayoglu, U., et al.: The process model matching contest 2013. In: 4th International Workshop on Process Model Collections: Management and Reuse (PMC-MR 2013) (2013)Google Scholar
- 5.Cheatham, M., Hitzler, P.: Conference v2.0: an uncertain version of the OAEI conference benchmark. In: Mika, P., et al. (eds.) ISWC 2014. LNCS, vol. 8797, pp. 33–48. Springer, Cham (2014). doi: 10.1007/978-3-319-11915-1_3 Google Scholar
- 6.Dumas, M., Rosa, M., Mendling, J., Reijers, H.: Fundamentals of Business Process Management. Springer, Heidelberg (2013)CrossRefGoogle Scholar
- 7.Jin, T., Wang, J., La Rosa, M., Ter Hofstede, A., Wen, L.: Efficient querying of large process model repositories. Comput. Ind. 64(1), 41–49 (2013)CrossRefGoogle Scholar
- 8.Klinkmüller, C., Weber, I., Mendling, J., Leopold, H., Ludwig, A.: Increasing recall of process model matching by improved activity label matching. In: Daniel, F., Wang, J., Weber, B. (eds.) BPM 2013. LNCS, vol. 8094, pp. 211–218. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40176-3_17 CrossRefGoogle Scholar
- 9.Kuss, E., Leopold, H., van der Aa, H., Stuckenschmidt, H., Reijers, H.A.: Probabilistic evaluation of process model matching techniques. In: Comyn-Wattiau, I., Tanaka, K., Song, I.-Y., Yamamoto, S., Saeki, M. (eds.) ER 2016. LNCS, vol. 9974, pp. 279–292. Springer, Cham (2016). doi: 10.1007/978-3-319-46397-1_22 CrossRefGoogle Scholar
- 10.Küster, J.M., Gerth, C., Förster, A., Engels, G.: Detecting and resolving process model differences in the absence of a change log. In: Dumas, M., Reichert, M., Shan, M.-C. (eds.) BPM 2008. LNCS, vol. 5240, pp. 244–260. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-85758-7_19 CrossRefGoogle Scholar
- 11.La Rosa, M., Dumas, M., Uba, R., Dijkman, R.: Business process model merging: an approach to business process consolidation. ACM Trans. Softw. Eng. Methodol. (TOSEM) 22(2), 11 (2013)Google Scholar
- 12.Leopold, H., Niepert, M., Weidlich, M., Mendling, J., Dijkman, R., Stuckenschmidt, H.: Probabilistic optimization of semantic process model matching. In: Barros, A., Gal, A., Kindler, E. (eds.) BPM 2012. LNCS, vol. 7481, pp. 319–334. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-32885-5_25 CrossRefGoogle Scholar
- 13.Mena, E., Kashyap, V., Illarramendi, A., Sheth, A.: Imprecise answers in distributed environments: estimation of information loss for multi-ontology based query processing. Int. J. Coop. Inf. Syst. 9(04), 403–425 (2000)CrossRefGoogle Scholar
- 14.Rahm, E., Bernstein, P.A.: A survey of approaches to automatic schema matching. VLDB J. 10(4), 334–350 (2001)CrossRefMATHGoogle Scholar
- 15.Sagi, T., Gal, A.: Non-binary evaluation for schema matching. In: Atzeni, P., Cheung, D., Ram, S. (eds.) ER 2012. LNCS, vol. 7532, pp. 477–486. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-34002-4_37 CrossRefGoogle Scholar
- 16.Shvaiko, P., Euzenat, J.: Ontology matching: state of the art and future challenges. IEEE Trans. Knowl. Data Eng. 25(1), 158–176 (2013)CrossRefGoogle Scholar
- 17.Spearman, C.: The proof and measurement of association between two things. Am. J. Psychol. 15(1), 72–101 (1904)CrossRefGoogle Scholar
- 18.Weidlich, M., Dijkman, R., Mendling, J.: The ICoP framework: identification of correspondences between process models. In: Pernici, B. (ed.) CAiSE 2010. LNCS, vol. 6051, pp. 483–498. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-13094-6_37 CrossRefGoogle Scholar
- 19.Weidlich, M., Sheetrit, E., Branco, M.C., Gal, A.: Matching business process models using positional passage-based language models. In: Ng, W., Storey, V.C., Trujillo, J.C. (eds.) ER 2013. LNCS, vol. 8217, pp. 130–137. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-41924-9_12 CrossRefGoogle Scholar
Copyright information
© Springer International Publishing AG 2017