Skip to main content

Intuitive, VR- and Gesture-based Physical Interaction with Virtual Commissioning Simulation Models


To manage complexity in modern, highly automated production systems, functional simulation for logic validation, known as virtual commissioning, is an important enabler. However, established systems for virtual commissioning are highly specialized software, which are solely used by simulation experts. In this contribution, we present a system for intuitive, user-integrated virtual commissioning. Using virtual reality, we allow for a first person experience of the simulation at runtime. By implementing a stereo-camera-based hand tracking system, we facilitate intuitive manual interaction with the simulation, including physical manipulation. Furthermore, we also integrate common human-machine interfaces, like touch panels, which are directly streamed from the industrial controller, offering the operator comparable interaction experience as with the real system. We conduct a user study quantifying the intuitiveness of interaction for defined tasks of our system compared to a standard simulation setup.


  • Virtual Commissioning
  • Virtual Reality
  • Motion Capturing


  • 1. Wünsch G (2008) Methoden für die virtuelle Inbetriebnahme automatisierter Produktionssysteme. IWB-Forschungsberichte, vol 215. Utz, München

    Google Scholar 

  • 2. Lechler T, Fischer E, Metzner M et al. (2019) Virtual Commissioning – Scientific review and exploratory use cases in advanced production systems. Procedia CIRP 81: 1125–1130.

  • 3. Glogowski P, Lemmerz K, Schulte L et al. (2018) Task-Based Simulation Tool for Human-Robot Collaboration within Assembly Systems. In: Schüppstuhl T, Tracht K, Franke J (eds) Tagungsband des 3. Kongresses Montage Handhabung Industrieroboter, 1. Auflage 2018. Springer Vieweg, Berlin, pp 151–159

    Google Scholar 

  • 4. Lemmerz K, Glogowski P, Hypki A et al. (2018) Functional Integration of a Robotics Software Framework into a Human Simulation System. 50th International Symposium on Robotics (ISR)

    Google Scholar 

  • 5. Giorgio A de, Romero M, Onori M et al. (2017) Human-machine Collaboration in Virtual Reality for Adaptive Production Engineering. Procedia Manufacturing 11: 1279–1287.

  • 6. Kim J, You S, Lee S et al. (2015) Evaluation of Human Robot Collaboration in Masonry Work Using Immersive Virtual Environments. International Conference on Construction Applications of Virtual Reality

    Google Scholar 

  • 7. Matsas E, Vosniakos G-C (2017) Design of a virtual reality training system for human–robot collaboration in manufacturing tasks. Int J Interact Des Manuf 11(2): 139–153.

  • 8. Dahl M, Albo A, Eriksson J et al. (2017) Virtual reality commissioning in production systems preparation. In: 2017 22nd IEEE International Conference on Emerging Technologies and Factory Automation: September 12-15, 2017, Limassol, Cyprus. IEEE, Piscataway, NJ, pp 1–7

    Google Scholar 

  • 9. Metzner M, Bönig J, Blank A et al. (2018) “Human-In-The-Loop”- Virtual Commissioning of Human-Robot Collaboration Systems. In: Schüppstuhl T, Tracht K, Franke J (eds) Tagungsband des 3. Kongresses Montage Handhabung Industrieroboter, 1. Auflage 2018. Springer Vieweg, Berlin, pp 131–138

    Google Scholar 

  • 10. Metzner M, Krieg L, Merhof J et al. (2019) Intuitive Interaction with Virtual Commissioning of Production Systems for Design Validation. Procedia CIRP 84: 892–895.

  • 11. Allmacher C, Dudczig M, Knopp S et al. (2019 - 2019) Virtual Reality for Virtual Commissioning of Automated Guided Vehicles. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, pp 838–839

    Google Scholar 

  • 12. Burduk A (ed) (2018) Intelligent Systems in Production Engineering and Maintenance, 1st edition 2019. Advances in Intelligent Systems and Computing. Springer International Publishing, Cham

    Google Scholar 

  • 13. Brecher C, Özdemir D (2012) Disziplinübergreifende Maschinenentwicklung. Mechatronics Concept Designer - Erprobt im Anwendungsfall. TOOLS-Informationen der Aachener Produktionstechniker 19(2): 2–4

    Google Scholar 

  • 14. Anokhin O, Anderl R (2019) Direct Control of 3D Models Through User Input to Simulate the Behavior of Mechatronic Systems. In: Karwowski W, Trzcielinski S, Mrugalska B et al. (eds) Advances in manufacturing, production management and process control, vol 793. Springer International Publishing; Imprint: Springer, Cham, pp 224–230

    Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Maximilian Metzner .

Editor information

Editors and Affiliations

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and Permissions

Copyright information

© 2020 The Author(s)

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Metzner, M., Krieg, L., Krüger, D., Ködel, T., Franke, J. (2020). Intuitive, VR- and Gesture-based Physical Interaction with Virtual Commissioning Simulation Models. In: Schüppstuhl, T., Tracht, K., Henrich, D. (eds) Annals of Scientific Society for Assembly, Handling and Industrial Robotics. Springer Vieweg, Berlin, Heidelberg.

Download citation