Human-in-the-Loop Control Processes in Gas Turbine Maintenance

  • Michael BarzEmail author
  • Peter Poller
  • Martin Schneider
  • Sonja Zillner
  • Daniel Sonntag
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10444)


In this applied research paper, we describe an architecture for seamlessly integrating factory workers in industrial cyber-physical production environments. Our human-in-the-loop control process uses novel input techniques and relies on state-of-the-art industry standards. Our architecture allows for real-time processing of semantically annotated data from multiple sources (e.g., machine sensors, user input devices) and real-time analysis of data for anomaly detection and recovery. We use a semantic knowledge base for storing and querying data ( and the Business Process Model and Notation (BPMN) for modelling and controlling the process. We exemplify our industrial solution in the use case of the maintenance of a Siemens gas turbine. We report on this case study and show the advantages of our approach for smart factories. An informal evaluation in the gas turbine maintenance use case shows the utility of automated anomaly detection and handling: workers can fill in paper-based incident reports by using a digital pen; the digitised version is stored in metaphacts and linked to semantic knowledge sources such as process models, structure models, business process models, and user models. Subsequently, automatic maintenance and recovery processes that involve human experts are triggered.


Cyber Physical System (CPS) Human-in-the-loop Industry 4.0 Smart factory Case study Handwriting recognition Gesture recognition Anomaly handling Business Process Model and Notation (BPMN) Anomaly detection Semantic Knowledge Base 



This research was funded in part by the European Institute of Innovation and Technology (EIT) in the CPS for Smart Factories project, see The responsibility for this publication lies with the authors.


  1. 1.
    Alsyouf, I.: The role of maintenance in improving companies’ productivity and profitability. Int. J. Prod. Econ. 105(1), 70–78 (2007). CrossRefGoogle Scholar
  2. 2.
    Barz, M., Poller, P., Sonntag, D.: Evaluating remote and head-worn eye trackers in multi-modal speech-based HRI. In: Mutlu, B., Tscheligi, M., Weiss, A., Young, J.E. (eds.) Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2017, Vienna, Austria, 6–9 March 2017, pp. 79–80. ACM (2017). doi: 10.1145/3029798.3038367
  3. 3.
    Barz, M., Sonntag, D.: Gaze-guided object classification using deep neural networks for attention-based computing. In: Lukowicz, P., Krüger, A., Bulling, A., Lim, Y., Patel, S.N. (eds.) Proceedings of 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp Adjunct 2016, Heidelberg, Germany, 12–16 September 2016, pp. 253–256. ACM (2016). doi: 10.1145/2968219.2971389
  4. 4.
    Compton, M., Barnaghi, P., Bermudez, L., García-Castro, R., Corcho, O., Cox, S., Graybeal, J., Hauswirth, M., Henson, C., Herzog, A., Huang, V., Janowicz, K., Kelsey, W.D., Phuoc, D.L., Lefort, L., Leggieri, M., Neuhaus, H., Nikolov, A., Page, K., Passant, A., Sheth, A., Taylor, K.: The SSN ontology of the W3C semantic sensor network incubator group. Web Semant.: Sci. Serv. Agents World Wide Web 17, 25–32 (2012).
  5. 5.
    Fantini, P., Tavola, G., Taisch, M., Barbosa, J., Leitao, P., Liu, Y., Sayed, M.S., Lohse, N.: Exploring the integration of the human as a flexibility factor in CPS enabled manufacturing environments: Methodology and results. In: IECON 2016–42nd Annual Conference of the IEEE Industrial Electronics Society, pp. 5711–5716, October 2016Google Scholar
  6. 6.
    Gaham, M., Bouzouia, B., Achour, N.: Human-in-the-loop cyber-physical production systems control (HiLCP2sC): a multi-objective interactive framework proposal. In: Borangiu, T., Thomas, A., Trentesaux, D. (eds.) Service Orientation in Holonic and Multi-agent Manufacturing, pp. 315–325. Springer International Publishing, Cham (2015). doi: 10.1007/978-3-319-15159-5_29 Google Scholar
  7. 7.
    Hermann, M., Pentek, T., Otto, B.: Design principles for industrie 4.0 scenarios: a literature review. Technische Universität Dortmund, Dortmund (2015)Google Scholar
  8. 8.
    Kahle, P.: Hot-rolled strip production. EKO Stahl.
  9. 9.
    Klyne, G., Carroll, J.J.: Resource description framework (RDF): concepts and abstract syntax, February 2006.
  10. 10.
    Krötzsch, M., Vrandečić, D., Völkel, M.: Semantic mediaWiki. In: Cruz, I., Decker, S., Allemang, D., Preist, C., Schwabe, D., Mika, P., Uschold, M., Aroyo, L.M. (eds.) ISWC 2006. LNCS, vol. 4273, pp. 935–942. Springer, Heidelberg (2006). doi: 10.1007/11926078_68 CrossRefGoogle Scholar
  11. 11.
    Laurent, S.S., Dumbill, E., Johnston, J.: Programming Web Services with XML-RPC. O’Reilly & Associates Inc, Sebastopol (2001)Google Scholar
  12. 12.
    Lee, J., Bagheri, B., Kao, H.A.: A cyber-physical systems architecture for industry 4.0-based manufacturing systems. Manuf. Lett. 3, 18–23 (2015)CrossRefGoogle Scholar
  13. 13.
    Lu, Y., Morris, K.C., Frechette, S.: Standards landscape and directions for smart manufacturing systems. In: 2015 IEEE International Conference on Automation Science and Engineering (CASE), pp. 998–1005. IEEE (2015)Google Scholar
  14. 14.
    Mayer, S., Hodges, J., Yu, D., Kritzler, M., Michahelles, F.: An open semantic framework for the industrial internet of things. IEEE Intell. Syst. 32(1), 96–101 (2017)CrossRefGoogle Scholar
  15. 15.
    Object Management Group: Business Process Model and Notation (BPMN) Version 2.0. Techical report, Open Management Group (2011).
  16. 16.
    Panfilenko, D., Poller, P., Sonntag, D., Zillner, S., Schneider, M.: BPMN for knowledge acquisition and anomaly handling in CPS for smart factories. In: 2016 IEEE 21st International Conference on Emerging Technologies and Factory Automation (ETFA), pp. 1–4. IEEE, September 2016.
  17. 17.
    Petersen, N., Galkin, M., Lange, C., Lohmann, S., Auer, S.: Monitoring and Automating Factories Using Semantic Models. Springer International Publishing, Cham (2016)CrossRefGoogle Scholar
  18. 18.
    Prange, A., Sonntag, D.: Digital PI-RADS: smartphone sketches for instant knowledge acquisition in prostate cancer detection. In: 29th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2016, Belfast, UK and Dublin, Ireland, 20–24 June 2016, pp. 13–18. IEEE Computer Society (2016). doi: 10.1109/CBMS.2016.23
  19. 19.
    Prud’Hommeaux, E., Seaborne, A.: SPARQL query language for RDF, January 2008.
  20. 20.
    Sonntag, D., Liwicki, M., Weber, M.: Digital pen in mammography patient forms. In: Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI 2011, NY, USA, pp. 303–306 (2011). doi: 10.1145/2070481.2070537
  21. 21.
    Sonntag, D., Romanelli, M.: A multimodal result ontology for integrated semantic web dialogue applications. In: Proceedings of 5th Conference on Language Resources and Evaluation, LREC 2006 (2006).
  22. 22.
    Sonntag, D., Sonnenberg, G., Nesselrath, R., Herzog, G.: Supporting a rapid dialogue system engineering process. In: Proceedings of 1st International Workshop on Spoken Dialogue Systems Technology, IWSDS (2009)Google Scholar
  23. 23.
    Sonntag, D., Weber, M., Cavallaro, A., Hammon, M.: Integrating digital pens in breast imaging for instant knowledge acquisition. AI Mag. 35(1), 26–37 (2014). CrossRefGoogle Scholar
  24. 24.
    Sonntag, D., Zillner, S., van der Smagt, P., Lörincz, A.: Overview of the CPS for smart factories project: deep learning, knowledge acquisition, anomaly detection and intelligent user interfaces. In: Jeschke, S., Brecher, C., Song, H., Rawat, D.B.R. (eds.) Industrial Internet of Things, pp. 487–504. Springer International Publishing, Cham (2017). doi: 10.1007/978-3-319-42559-7_19 CrossRefGoogle Scholar
  25. 25.
    Wang, K.: Intelligent predictive maintenance (IPdM) system-industry 4.0 scenario. WIT Trans. Eng. Sci. 113, 259–268 (2016)Google Scholar
  26. 26.
    Wang, L., Törngren, M., Onori, M.: Current status and advancement of cyber-physical systems in manufacturing. J. Manuf. Syst. 37(Part 2), 517–527 (2015)CrossRefGoogle Scholar
  27. 27.
    Zamfirescu, C.B., Pirvu, B.C., Gorecky, D., Chakravarthy, H.: Human-centred assembly: a case study for an anthropocentric cyber-physical system. Procedia Technol. 15, 90–98 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Michael Barz
    • 1
    Email author
  • Peter Poller
    • 1
  • Martin Schneider
    • 2
  • Sonja Zillner
    • 2
  • Daniel Sonntag
    • 1
  1. 1.German Research Center for Artificial Intelligence (DFKI)SaarbrückenGermany
  2. 2.Corporate Technology Siemens AGMünchenGermany

Personalised recommendations