Augmented Reality-Based Process Modelling for the Internet of Things with HoloFlows

Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 352)


Workflow technologies can be handy to model, execute and analyse simple processes in Internet of Things (IoT) environments. End-users are enabled to compose processes and thereby automate basic repetitive tasks involving one or more IoT devices. However, the modelling of these IoT workflows currently relies on rather bloated and complex desktop applications, deep knowledge of the underlying process notations and a high level of abstraction, which makes workflow modelling too complicated for end-users. In this work we propose to use augmented reality (AR) to simplify the modelling and configuration of IoT workflows. With our HoloFlows app for smart glasses end-users are able to explore their surrounding IoT environment and model various types of basic processes involving sensors and actuators by simply connecting two or more physical IoT devices via virtual wires. AR technology hereby facilitates the understanding of the physical contexts and relations among the IoT devices and provides a new and more intuitive way of modelling workflows in the cyber-physical world. We demonstrate the HoloFlows app with the help of various IoT workflows from the smart home domain.


Process modelling Augmented reality Internet of Things 



This research has received funding under the grant number 100268299 by the European Social Fund (ESF) and the German Federal State of Saxony. Kudos to our student Henrik Schole for his support regarding the implementation.


  1. 1.
    Avila, L., Bailey, M.: Augment your reality. IEEE Comput. Graph. Appl. 36(1), 6–7 (2016)CrossRefGoogle Scholar
  2. 2.
    Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. Technical reports, Naval Research Lab, Washington, DC (2001)CrossRefGoogle Scholar
  3. 3.
    Bauer, M., et al.: IoT reference model. In: Bassi, A., et al. (eds.) Enabling Things to Talk, pp. 113–162. Springer, Heidelberg (2013). Scholar
  4. 4.
    Brouns, N., Tata, S., Ludwig, H., Asensio, E.S., Grefen, P.: Modeling iot-aware business processes-a state of the art report. arXiv preprint arXiv:1811.00652 (2018)
  5. 5.
    Chang, C., Srirama, S.N., Buyya, R.: Mobile cloud business process management system for the internet of things: a survey. ACM Comput. Surv. 49(4), 70 (2016)CrossRefGoogle Scholar
  6. 6.
    Domingos, D., Martins, F., Cândido, C., Martinho, R.: Internet of Things aware WS-BPEL business processes - context variables and expected exceptions. J. UCS 20(8), 1109–1129 (2014)Google Scholar
  7. 7.
    Friedow, C., Völker, M., Hewelt, M.: Integrating IoT devices into business processes. In: Matulevičius, R., Dijkman, R. (eds.) CAiSE 2018. LNBIP, vol. 316, pp. 265–277. Springer, Cham (2018). Scholar
  8. 8.
    Giner, P., Cetina, C., Fons, J., Pelechano, V.: Developing mobile workflow support in the Internet of Things. IEEE Pervasive Comput. 9(2), 18–26 (2010)CrossRefGoogle Scholar
  9. 9.
    Graja, I., Kallel, S., Guermouche, N., Kacem, A.H.: BPMN4CPS: a BPMN extension for modeling cyber-physical systems. In: 2016 IEEE 25th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), pp. 152–157, June 2016Google Scholar
  10. 10.
    Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.: Internet of Things (IoT): a vision, architectural elements, and future directions. Future Gener. Comput. Syst. 29(7), 1645–1660 (2013)CrossRefGoogle Scholar
  11. 11.
    Huber, S., Seiger, R., Kühnert, A., Theodorou, V., Schlegel, T.: Goal-based semantic queries for dynamic processes in the Internet of Things. Int. J. Semant. Comput. 10(02), 269–293 (2016)CrossRefGoogle Scholar
  12. 12.
    Janiesch, C., et al.: The Internet-of-Things meets business process management: Mutual benefits and challenges. arXiv preprint arXiv:1709.03628 (2017)
  13. 13.
    Kammerer, K., Pryss, R., Sommer, K., Reichert, M.: Towards context-aware process guidance in cyber-physical systems with augmented reality. In: 2018 4th International Workshop on Requirements Engineering for Self-Adaptive, Collaborative, and Cyber Physical Systems (RESACS), pp. 44–51. IEEE (2018)Google Scholar
  14. 14.
    Korzetz, M., Kühn, R., Gohlke, M., Aßmann, U.: HoloFacility: get in touch with machines at trade fairs using holograms. In: Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces, pp. 336–341. ACM (2017)Google Scholar
  15. 15.
    Martins, F., Domingos, D.: Modelling IoT behaviour within BPMN business processes. Procedia Comput. Sci. 121, 1014–1022 (2017)CrossRefGoogle Scholar
  16. 16.
    Mayer, S., Inhelder, N., Verborgh, R., Van de Walle, R.: User-friendly configuration of smart environments. In: 12th IEEE International Conference on Pervasive Computing and Communication (PERCOM), pp. 163–165. IEEE (2014)Google Scholar
  17. 17.
    Mendling, J., Baesens, B., Bernstein, A., Fellmann, M.: Challenges of smart business process management: an introduction to the special issue. Decis. Support Syst. 100, 1–5 (2017)CrossRefGoogle Scholar
  18. 18.
    Meyer, S., Ruppen, A., Magerkurth, C.: Internet of Things-aware process modeling: integrating IoT devices as business process resources. In: Salinesi, C., Norrie, M.C., Pastor, Ó. (eds.) CAiSE 2013. LNCS, vol. 7908, pp. 84–98. Springer, Heidelberg (2013). Scholar
  19. 19.
    OMG: Business Process Model and Notation (BPMN), Version 2.0, January 2011Google Scholar
  20. 20.
    Poppe, E., Brown, R.A., Recker, J.C., Johnson, D.M.: A prototype augmented reality collaborative process modelling tool (2011)Google Scholar
  21. 21.
    Pryss, R., Reichert, M., Bachmeier, A., Albach, J.: BPM to go: supporting business processes in a mobile and sensing world (2015)Google Scholar
  22. 22.
    Seiger, R., Huber, S., Schlegel, T.: Toward an execution system for self-healing workflows in cyber-physical systems. Softw. Syst. Model. 17, 551–572 (2018)CrossRefGoogle Scholar
  23. 23.
    Seiger, R., Keller, C., Niebling, F., Schlegel, T.: Modelling complex and flexible processes for smart cyber-physical environments. J. Comput. Sci. 10, 137–148 (2015)CrossRefGoogle Scholar
  24. 24.
    Seiger, R., Korzetz, M., Gohlke, M., Aßmann, U.: Mixed reality cyber-physical systems control and workflow composition. In: Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia, pp. 495–500. ACM (2017)Google Scholar
  25. 25.
    Sungur, C.T., Spiess, P., Oertel, N., Kopp, O.: Extending BPMN for wireless sensor networks. In: IEEE 15th Conference on Business Informatics, pp. 109–116 (2013)Google Scholar
  26. 26.
    Suri, K., Gaaloul, W., Cuccuru, A., Gerard, S.: Semantic framework for internet of things-aware business process development. In: IEEE 26th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), pp. 214–219. IEEE (2017)Google Scholar
  27. 27.
    Walch, M., Karagiannis, D.: Service-driven enrichment for KbR in the OMiLAB environment. Serviceology for Services. LNCS, vol. 10371, pp. 164–177. Springer, Cham (2017). Scholar
  28. 28.
    Wang, X., Ong, S.K., Nee, A.Y.: A comprehensive survey of augmented reality assembly research. Adv. Manuf. 4(1), 1–22 (2016)CrossRefGoogle Scholar
  29. 29.
    Weiser, M.: The computer for the 21st century. Sci. Am. 265(3), 94–104 (1991)CrossRefGoogle Scholar
  30. 30.
    Yousfi, A., Hewelt, M., Bauer, C., Weske, M.: Towards uBPMN-based patterns for modeling ubiquitous business processes. IEEE Trans. Ind. Inf. 14, 3358–3367 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Software Technology GroupTechnische Universität DresdenDresdenGermany

Personalised recommendations