Abstract
Despite the widespread availability of process modeling tools, the first version of a process model is often drawn by hand on a piece of paper or whiteboard, especially when several people are involved in its elicitation. Though this has been found to be beneficial for the modeling task itself, it also creates the need to manually convert hand-drawn models afterward, such that they can be further used in a modeling tool. This manual transformation is associated with considerable time and effort and, furthermore, creates undesirable friction in the modeling workflow. In this paper, we alleviate this problem by presenting a technique that can automatically recognize and convert a sketch process model into a digital BPMN model. A key driver and contribution of our work is the creation of a publicly available dataset consisting of 502 manually annotated, hand-drawn BPMN models, covering 25 different BPMN elements. Based on this data set, we have established a neural network-based recognition technique that can reliably recognize and transform hand-drawn BPMN models. Our evaluation shows that our technique considerably outperforms available baselines and, therefore, provides a valuable basis to smoothen the modeling process.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
See also the public repository for the modeling tasks and provided instructions.
- 2.
- 3.
- 4.
Note that an edge is still considered correct if its associated shapes are incorrectly classified.
References
Aagesen, G., Krogstie, J.: Analysis and design of business processes using BPMN. In: Handbook on Business Process Management 1, pp. 213–235. Springer (2010). https://doi.org/10.1007/978-3-642-00416-2_10
Allweyer, T.: BPMN 2.0: Introduction to the standard for business process modeling. BoD-Books on Demand (2016)
Antunes, P., Simões, D., Carriço, L., Pino, J.A.: An end-user approach to business process modeling. J. Netw. Comput. Appl. 36(6), 1466–1479 (2013)
Bartelt, C., Vogel, M., Warnecke, T.: Collaborative creativity: from hand drawn sketches to formal domain specific models and back again. In: MoRoCo@ ECSCW, pp. 25–32 (2013)
Bresler, M., Průša, D., Hlaváč, V.: Recognizing off-line flowcharts by reconstructing strokes and using on-line recognition techniques. In: (ICFHR), pp. 48–53 (2016)
Bresler, M., Průša, D., Hlaváč, V.: Online recognition of sketched arrow-connected diagrams. Int. J. Doc. Anal. Recogn. (IJDAR) 19(3), 253–267 (2016). https://doi.org/10.1007/s10032-016-0269-z
Brown, R., Recker, J., West, S.: Using virtual worlds for collaborative business process modeling. Bus. Process Manage. J. 17(3), 546–564 (2011)
Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A.A.: Albumentations: fast and flexible image augmentations. Information 11(2), 125 (2020)
Cherubini, M., Venolia, G., DeLine, R., Ko, A.J.: Let’s go to the whiteboard: how and why software developers use drawings. In: SIGCHI, pp. 557–566 (2007)
Damm, C.H., Hansen, K.M., Thomsen, M.: Tool support for cooperative object-oriented design: gesture based modelling on an electronic whiteboard. In: SIGCHI, pp. 518–525 (2000)
Doermann, D., Liang, J., Li, H.: Progress in camera-based document image analysis. In: ICDAR, vol. 1, pp. 606–616 (2003)
Dumas, M., Rosa, M.L., Mendling, J., Reijers, H.A.: Fundamentals of Business Process Management. Springer, 2 edn. (2018). https://doi.org/10.1007/978-3-642-33143-5.pdf
Forster, S., Pinggera, J., Weber, B.: Toward an understanding of the collaborative process of process modeling. In: CAiSE Forum, pp. 98–105 (2013)
Gervais, P., Deselaers, T., Aksan, E., Hilliges, O.: The DIDI dataset: Digital Ink Diagram data. arXiv:2002.09303 [cs] (2020)
Hammond, T., Davis, R.: Tahuti: a geometrical sketch recognition system for UML class diagrams. In: ACM SIGGRAPH 2006 Courses, pp. 25-es (2006)
Julca-Aguilar, F., Mouchère, H., Viard-Gaudin, C., Hirata, N.S.T.: A general framework for the recognition of online handwritten graphics. Int. J. Doc. Anal. Recogn. (IJDAR) 23(2), 143–160 (2020). https://doi.org/10.1007/s10032-019-00349-6
Polančič, G., Jagečić, S., Kous, K.: An empirical investigation of the effectiveness of optical recognition of hand-drawn business process elements by applying machine learning. IEEE Access 8, 206118–206131 (2020)
Recker, J., Mendling, J., Hahn, C.: How collaborative technology supports cognitive processes in collaborative process modeling: a capabilities-gains-outcome model. Inf. Syst. 38(8), 1031–1045 (2013)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NeurIPS, pp. 91–99 (2015)
Schäfer, B., Keuper, M., Stuckenschmidt, H.: Arrow R-CNN for handwritten diagram recognition. Int. J. Doc. Analy. Recogn. (IJDAR) (2021)
Schäfer, B., Stuckenschmidt, H.: Arrow R-CNN for flowchart recognition. In: International Conference on Document Analysis and Recognition Workshops (ICDARW) (2019)
Wu, J., Wang, C., Zhang, L., Rui, Y.: Offline sketch parsing via shapeness estimation. In: IJCAI (2015)
Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
Zapp, M., Fettke, P., Loos, P.: Towards a Software Prototype Supporting Automatic Recognition of Sketched Business Process Models. Wirtschaftsinformatik 2017 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Schäfer, B., van der Aa, H., Leopold, H., Stuckenschmidt, H. (2021). Sketch2BPMN: Automatic Recognition of Hand-Drawn BPMN Models. In: La Rosa, M., Sadiq, S., Teniente, E. (eds) Advanced Information Systems Engineering. CAiSE 2021. Lecture Notes in Computer Science(), vol 12751. Springer, Cham. https://doi.org/10.1007/978-3-030-79382-1_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-79382-1_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-79381-4
Online ISBN: 978-3-030-79382-1
eBook Packages: Computer ScienceComputer Science (R0)