A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments

  • Felix Mannhardt
  • Riccardo Bovo
  • Manuel Fradinho Oliveira
  • Simon Julier
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11315)


Despite the increasing automation levels in an Industry 4.0 scenario, the tacit knowledge of highly skilled manufacturing workers remains of strategic importance. Retaining this knowledge by formally capturing it is a challenge for industrial organisations. This paper explores research on automatically capturing this knowledge by using methods from activity recognition and process mining on data obtained from sensorised workers and environments. Activity recognition lifts the abstraction level of sensor data to recognizable activities and process mining methods discover models of process executions. We classify the existing work, which largely neglects the possibility of applying process mining, and derive a taxonomy that identifies challenges and research gaps.


Activity recognition Process mining Manufacturing Industrial environment Tacit knowledge Literature overview 



This research has received funding from the European Union’s H2020 research and innovation programme under grant agreement no. 723737 (HUMAN).


  1. 1.
    van der Aalst, W.M.P.: Process Mining - Data Science in Action, 2nd edn. Springer, Heidelberg (2016). Scholar
  2. 2.
    Abdallah, Z.S., Gaber, M.M., Srinivasan, B., Krishnaswamy, S.: Activity recognition with evolving data streams. ACM Comput. Surv. 51(4), 1–36 (2018)CrossRefGoogle Scholar
  3. 3.
    Aehnelt, M., Gutzeit, E., Urban, B.: Using activity recognition for the tracking of assembly processes: challenges and requirements. In: WOAR 2014. Fraunhofer Verlag (2014)Google Scholar
  4. 4.
    Aggarwal, J., Ryoo, M.: Human activity analysis. ACM Comput. Surv. 43(3), 1–43 (2011)CrossRefGoogle Scholar
  5. 5.
    Al-Naser, M., et al.: Hierarchical model for zero-shot activity recognition using wearable sensors. In: ICAART (2), pp. 478–485. SciTePress (2018)Google Scholar
  6. 6.
    Augusto, A., et al.: Automated discovery of process models from event logs: review and benchmark. IEEE Trans. Knowl. Data Eng. (2018)Google Scholar
  7. 7.
    Bader, S., Aehnelt, M.: Tracking assembly processes and providing assistance in smart factories. In: ICAART 2014. SCITEPRESS (2014)Google Scholar
  8. 8.
    Bader, S., Krüger, F., Kirste, T.: Computational causal behaviour models for assisted manufacturing. In: iWOAR 2015. ACM Press (2015)Google Scholar
  9. 9.
    Bannach, D., Kunze, K., Lukowicz, P., Amft, O.: Distributed modular toolbox for multi-modal context recognition. In: Grass, W., Sick, B., Waldschmidt, K. (eds.) ARCS 2006. LNCS, vol. 3894, pp. 99–113. Springer, Heidelberg (2006). Scholar
  10. 10.
    Blanke, U., Schiele, B.: Remember and transfer what you have learned - recognizing composite activities based on activity spotting. In: ISWC 2010. IEEE (2010)Google Scholar
  11. 11.
    Böttcher, S., Scholl, P.M., Laerhoven, K.V.: Detecting process transitions from wearable sensors. In: iWOAR 2017. ACM Press (2017)Google Scholar
  12. 12.
    Feldhorst, S., Masoudenijad, M., ten Hompel, M., Fink, G.A.: Motion classification for analyzing the order picking process using mobile sensors - general concepts, case studies and empirical evaluation. In: ICPRAM 2016, pp. 706–713. SCITEPRESS (2016)Google Scholar
  13. 13.
    Gonella, P., Castellano, M., Riccardi, P., Carbone, R.: Process mining: a database of applications. Technical report, HSPI SpA - Management Consulting (2017)Google Scholar
  14. 14.
    Goto, H., Miura, J., Sugiyama, J.: Human-robot collaborative assembly by on-line human action recognition based on an FSM task model. In: Human-Robot Interaction 2013 Workshop on Collaborative Manipulation (2013)Google Scholar
  15. 15.
    Grzeszick, R., Lenk, J.M., Rueda, F.M., Fink, G.A., Feldhorst, S., ten Hompel, M.: Deep neural network based human activity recognition for the order picking process. In: iWOAR 2017. ACM Press (2017)Google Scholar
  16. 16.
    Janiesch, C., et al.: The Internet-of-Things meets business process management: mutual benefits and challenges (2017). arXiv:1709.03628
  17. 17.
    Knoch, S., Ponpathirkoottam, S., Fettke, P., Loos, P.: Technology-enhanced process elicitation of worker activities in manufacturing. In: Teniente, E., Weidlich, M. (eds.) BPM 2017. LNBIP, vol. 308, pp. 273–284. Springer, Cham (2018). Scholar
  18. 18.
    Lara, O.D., Labrador, M.A.: A survey on human activity recognition using wearable sensors. IEEE Commun. Surv. Tutor. 15(3), 1192–1209 (2013)CrossRefGoogle Scholar
  19. 19.
    Longstaff, B., Reddy, S., Estrin, D.: Improving activity classification for health applications on mobile devices using active and semi-supervised learning. In: ICST 2010. IEEE (2010)Google Scholar
  20. 20.
    Lukowicz, P., et al.: Recognizing workshop activity using body worn microphones and accelerometers. In: Ferscha, A., Mattern, F. (eds.) Pervasive 2004. LNCS, vol. 3001, pp. 18–32. Springer, Heidelberg (2004). Scholar
  21. 21.
    Maekawa, T., Nakai, D., Ohara, K., Namioka, Y.: Toward practical factory activity recognition. In: UbiComp 2016. ACM Press (2016)Google Scholar
  22. 22.
    Mannhardt, F., de Leoni, M., Reijers, H.A., van der Aalst, W.M.P.: Balanced multi-perspective checking of process conformance. Computing 98(4), 407–437 (2016)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Mannhardt, F., Petersen, S.A., de Oliveira, M.F.D.: Privacy challenges for process mining in human-centered industrial environments. In: Intelligent Environments (IE). IEEE Xplore (2018, to appear)Google Scholar
  24. 24.
    Marin-Perianu, M., Lombriser, C., Amft, O., Havinga, P., Tröster, G.: Distributed activity recognition with fuzzy-enabled wireless sensor networks. In: Nikoletseas, S.E., Chlebus, B.S., Johnson, D.B., Krishnamachari, B. (eds.) DCOSS 2008. LNCS, vol. 5067, pp. 296–313. Springer, Heidelberg (2008). Scholar
  25. 25.
    Mörzinger, R., et al.: Tools for semi-automatic monitoring of industrial workflows. In: ARTEMIS 2010. ACM Press (2010)Google Scholar
  26. 26.
    Mura, M.D., Dini, G., Failli, F.: An integrated environment based on augmented reality and sensing device for manual assembly workstations. Procedia CIRP 41, 340–345 (2016)CrossRefGoogle Scholar
  27. 27.
    Ogris, G., Lukowicz, P., Stiefmeier, T., Tröster, G.: Continuous activity recognition in a maintenance scenario: combining motion sensors and ultrasonic hands tracking. Pattern Anal. Appl. 15(1), 87–111 (2011)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Ogris, G., Stiefmeier, T., Lukowicz, P., Troster, G.: Using a complex multi-modal on-body sensor system for activity spotting. In: IWSC 2008. IEEE (2008)Google Scholar
  29. 29.
    Ramamurthy, S.R., Roy, N.: Recent trends in machine learning for human activity recognition-a survey. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 8(4), e1254 (2018)CrossRefGoogle Scholar
  30. 30.
    Raso, R., et al.: Activity monitoring using wearable sensors in manual production processes - an application of CPS for automated ergonomic assessments. In: MKWI 2018. Leuphana Universität Lüneburg (2018)Google Scholar
  31. 31.
    Repta, D., Moisescu, M.A., Sacala, I.S., Stanescu, A.M., Constantin, N.: Generic architecture for process mining in the context of cyber physical systems. Appl. Mech. Mater. 656, 569–577 (2014)CrossRefGoogle Scholar
  32. 32.
    Roitberg, A., Perzylo, A., Somani, N., Giuliani, M., Rickert, M., Knoll, A.: Human activity recognition in the context of industrial human-robot interaction. In: APSIPA 2014. IEEE (2014)Google Scholar
  33. 33.
    Roitberg, A., Somani, N., Perzylo, A., Rickert, M., Knoll, A.: Multimodal human activity recognition for industrial manufacturing processes in robotic workcells. In: ICMI 2015. ACM Press (2015)Google Scholar
  34. 34.
    Schlenoff, C., Kootbally, Z., Pietromartire, A., Franaszek, M., Foufou, S.: Intention recognition in manufacturing applications. Robot. Comput. Integr. Manuf. 33, 29–41 (2015)CrossRefGoogle Scholar
  35. 35.
    Stiefmeier, T., Lombriser, C., Roggen, D., Junker, H., Ogris, G., Troester, G.: Event-based activity tracking in work environments. In: IFAWC 2006, pp. 1–10 (2006)Google Scholar
  36. 36.
    Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tr, G.: Wearable activity tracking in car manufacturing. IEEE Pervasive Comput. 7(2), 42–50 (2008)CrossRefGoogle Scholar
  37. 37.
    Voulodimos, A., et al.: A threefold dataset for activity and workflow recognition in complex industrial environments. IEEE Multimed. 19(3), 42–52 (2012)CrossRefGoogle Scholar
  38. 38.
    Voulodimos, A.S., Kosmopoulos, D.I., Doulamis, N.D., Varvarigou, T.A.: A top-down event-driven approach for concurrent activity recognition. Multimed. Tools Appl. 69(2), 293–311 (2012)CrossRefGoogle Scholar
  39. 39.
    Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recognit. Lett. (2018, in press)Google Scholar
  40. 40.
    Ward, J.A., Lukowicz, P., Troster, G., Starner, T.E.: Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1553–1567 (2006)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.SINTEF DigitalTrondheimNorway
  2. 2.Department of Computer ScienceUCLLondonUK

Personalised recommendations