Skip to main content

Sketch2BPMN: Automatic Recognition of Hand-Drawn BPMN Models

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12751)

Abstract

Despite the widespread availability of process modeling tools, the first version of a process model is often drawn by hand on a piece of paper or whiteboard, especially when several people are involved in its elicitation. Though this has been found to be beneficial for the modeling task itself, it also creates the need to manually convert hand-drawn models afterward, such that they can be further used in a modeling tool. This manual transformation is associated with considerable time and effort and, furthermore, creates undesirable friction in the modeling workflow. In this paper, we alleviate this problem by presenting a technique that can automatically recognize and convert a sketch process model into a digital BPMN model. A key driver and contribution of our work is the creation of a publicly available dataset consisting of 502 manually annotated, hand-drawn BPMN models, covering 25 different BPMN elements. Based on this data set, we have established a neural network-based recognition technique that can reliably recognize and transform hand-drawn BPMN models. Our evaluation shows that our technique considerably outperforms available baselines and, therefore, provides a valuable basis to smoothen the modeling process.

Keywords

  • Process modeling
  • Sketch recognition
  • Hand-drawn process models

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-79382-1_21
  • Chapter length: 17 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-79382-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

Notes

  1. 1.

    See also the public repository for the modeling tasks and provided instructions.

  2. 2.

    https://github.com/bpmn-io/bpmn-js.

  3. 3.

    https://github.com/dwslab/bpmn-image-annotator.

  4. 4.

    Note that an edge is still considered correct if its associated shapes are incorrectly classified.

References

  1. Aagesen, G., Krogstie, J.: Analysis and design of business processes using BPMN. In: Handbook on Business Process Management 1, pp. 213–235. Springer (2010). https://doi.org/10.1007/978-3-642-00416-2_10

  2. Allweyer, T.: BPMN 2.0: Introduction to the standard for business process modeling. BoD-Books on Demand (2016)

    Google Scholar 

  3. Antunes, P., Simões, D., Carriço, L., Pino, J.A.: An end-user approach to business process modeling. J. Netw. Comput. Appl. 36(6), 1466–1479 (2013)

    CrossRef  Google Scholar 

  4. Bartelt, C., Vogel, M., Warnecke, T.: Collaborative creativity: from hand drawn sketches to formal domain specific models and back again. In: MoRoCo@ ECSCW, pp. 25–32 (2013)

    Google Scholar 

  5. Bresler, M., Průša, D., Hlaváč, V.: Recognizing off-line flowcharts by reconstructing strokes and using on-line recognition techniques. In: (ICFHR), pp. 48–53 (2016)

    Google Scholar 

  6. Bresler, M., Průša, D., Hlaváč, V.: Online recognition of sketched arrow-connected diagrams. Int. J. Doc. Anal. Recogn. (IJDAR) 19(3), 253–267 (2016). https://doi.org/10.1007/s10032-016-0269-z

    CrossRef  Google Scholar 

  7. Brown, R., Recker, J., West, S.: Using virtual worlds for collaborative business process modeling. Bus. Process Manage. J. 17(3), 546–564 (2011)

    Google Scholar 

  8. Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A.A.: Albumentations: fast and flexible image augmentations. Information 11(2), 125 (2020)

    CrossRef  Google Scholar 

  9. Cherubini, M., Venolia, G., DeLine, R., Ko, A.J.: Let’s go to the whiteboard: how and why software developers use drawings. In: SIGCHI, pp. 557–566 (2007)

    Google Scholar 

  10. Damm, C.H., Hansen, K.M., Thomsen, M.: Tool support for cooperative object-oriented design: gesture based modelling on an electronic whiteboard. In: SIGCHI, pp. 518–525 (2000)

    Google Scholar 

  11. Doermann, D., Liang, J., Li, H.: Progress in camera-based document image analysis. In: ICDAR, vol. 1, pp. 606–616 (2003)

    Google Scholar 

  12. Dumas, M., Rosa, M.L., Mendling, J., Reijers, H.A.: Fundamentals of Business Process Management. Springer, 2 edn. (2018). https://doi.org/10.1007/978-3-642-33143-5.pdf

  13. Forster, S., Pinggera, J., Weber, B.: Toward an understanding of the collaborative process of process modeling. In: CAiSE Forum, pp. 98–105 (2013)

    Google Scholar 

  14. Gervais, P., Deselaers, T., Aksan, E., Hilliges, O.: The DIDI dataset: Digital Ink Diagram data. arXiv:2002.09303 [cs] (2020)

  15. Hammond, T., Davis, R.: Tahuti: a geometrical sketch recognition system for UML class diagrams. In: ACM SIGGRAPH 2006 Courses, pp. 25-es (2006)

    Google Scholar 

  16. Julca-Aguilar, F., Mouchère, H., Viard-Gaudin, C., Hirata, N.S.T.: A general framework for the recognition of online handwritten graphics. Int. J. Doc. Anal. Recogn. (IJDAR) 23(2), 143–160 (2020). https://doi.org/10.1007/s10032-019-00349-6

    CrossRef  Google Scholar 

  17. Polančič, G., Jagečić, S., Kous, K.: An empirical investigation of the effectiveness of optical recognition of hand-drawn business process elements by applying machine learning. IEEE Access 8, 206118–206131 (2020)

    Google Scholar 

  18. Recker, J., Mendling, J., Hahn, C.: How collaborative technology supports cognitive processes in collaborative process modeling: a capabilities-gains-outcome model. Inf. Syst. 38(8), 1031–1045 (2013)

    CrossRef  Google Scholar 

  19. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NeurIPS, pp. 91–99 (2015)

    Google Scholar 

  20. Schäfer, B., Keuper, M., Stuckenschmidt, H.: Arrow R-CNN for handwritten diagram recognition. Int. J. Doc. Analy. Recogn. (IJDAR) (2021)

    Google Scholar 

  21. Schäfer, B., Stuckenschmidt, H.: Arrow R-CNN for flowchart recognition. In: International Conference on Document Analysis and Recognition Workshops (ICDARW) (2019)

    Google Scholar 

  22. Wu, J., Wang, C., Zhang, L., Rui, Y.: Offline sketch parsing via shapeness estimation. In: IJCAI (2015)

    Google Scholar 

  23. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)

  24. Zapp, M., Fettke, P., Loos, P.: Towards a Software Prototype Supporting Automatic Recognition of Sketched Business Process Models. Wirtschaftsinformatik 2017 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Bernhard Schäfer .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Schäfer, B., van der Aa, H., Leopold, H., Stuckenschmidt, H. (2021). Sketch2BPMN: Automatic Recognition of Hand-Drawn BPMN Models. In: La Rosa, M., Sadiq, S., Teniente, E. (eds) Advanced Information Systems Engineering. CAiSE 2021. Lecture Notes in Computer Science(), vol 12751. Springer, Cham. https://doi.org/10.1007/978-3-030-79382-1_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-79382-1_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-79381-4

  • Online ISBN: 978-3-030-79382-1

  • eBook Packages: Computer ScienceComputer Science (R0)