Intuitive, VR-and Gesture-based Physical Interaction with Virtual Commissioning Simulation Models

. To manage complexity in modern, highly automated production systems, functional simulation for logic validation, known as virtual commissioning, is an important enabler. However, established systems for virtual commissioning are highly specialized software, which are solely used by simulation experts. In this contribution, we present a system for intuitive, user-integrated virtual commissioning. Using virtual reality, we allow for a first person experience of the simulation at runtime. By implementing a stereo-camera-based hand tracking system, we facilitate intuitive manual interaction with the simulation, including physical manipulation. Furthermore, we also integrate common human-machine interfaces, like touch panels, which are directly streamed from the industrial controller, offering the operator comparable interaction experience as with the real system. We conduct a user study quantifying the intuitiveness of interaction for defined tasks of our system compared to a standard simulation setup.


Introduction
Increasing complexity in functional control logic of production system drives the need for a thorough testing of these algorithms. Virtual commissioning (VC), a method for testing programmable logic controller (PLC) functionality by connecting it to a simulation emulating the real system, is an important tool in this context [1,2]. However, most production systems still require frequent interaction with human operators for value creation, setup or adaption, maintenance or fault remedying. This human factor is thus critical for a comprehensive simulation of system behavior and function. The active integration of humans into VC environments is hence an important research topic. In the next section, we highlight related work for human integration into simulations and VC of production systems. Based on this, we derive the research gap to be addressed in this contribution. An overview of the abstract solution structure including main components and functions is given before a real implementation is described. Based on a robotic handling use-case incorporating many common automation components, we conduct a user study focusing on the interaction capabilities of our system. We describe and discuss the results of this study before summarizing our research and giving an outlook to future research areas in this field.

Related work
Virtual commissioning is defined as the development and testing of functional logic on a simulation model for later deployment in a real system [1]. For this, a virtual commissioning setup consists of one or more simulation tools and PLCs connected to it. Each PLC can be emulated in separate PLC simulation tool or consist of a hardware PLC connected to the simulation system. For special system components, like industrial robots, the integration of further controllers may be necessary to ensure realistic behavior.
[2] The integration of humans into such simulation systems is still a research focus. This is mainly due to the inherent non-determinism of human behavior, that makes a comprehensive simulation challenging. However, as human interaction with (partially) automated systems is common, it is also a vital part of virtual commissioning. Especially in the field of human robot collaboration (HRC), the simulation of humans has been extensively researched. Some approaches use ergonomics simulation tools to offline program the behavior of a digital human model (DHM) [3,4]. While this method can be used for workstation design and ergonomics improvements, real interaction is hardly implementable.
Other approaches focus on a live integration, using different forms of motion capturing systems (MCS) to map a real human's behavior onto a DHM. De Giorgio et al. [5], Kim et al. [6] and Matsas et al. [7] each implement robot models into a Unity®based environment which is used for VR visualization. These environments are designed as standalone applications for each system to be simulated. Also, system and robot behavior are implemented through scripts in the simulation.
Dahl et al. [8] and Metzner et al. [9] implement VR directly into a virtual commissioning tool for robotic applications, Tecnomatix Process Simulate®, to enable live human integration. Motion capturing is implemented using gaming controllers or fullbody tracking. The simulation tool itself does not support physics or general mechatronics simulation.
Metzner et al. furthermore present an approach to directly map the operator's hands, as the most relevant channel of interaction, into a physics-integrated mechatronics simulation tool [10]. In this approach, however, no VR technology is implemented, decreasing intuitiveness of interaction. Allmacher et al. use a similar approach, implementing the user including his hands into a Unity®-based VR simulation of an automated guided vehicle [11].

Research Gap and Contribution
In the following section, we derive evaluation criteria and benchmark the described approaches to derive a research gap.
Current research in the field of human integration to VC systems mostly focuses on realistic first person visualization of the simulated system. These systems are however mostly standalone applications developed in gaming engines [5][6][7]11]. This not only greatly decreases re-usability of components and functional logic; it also increases implementation efforts and prevents realistic system behavior. As actual logic and motion controllers, like PLCs and robot controllers, are mostly only imitated through customized behavior programming in the gaming application, these systems also do not fulfil the requirements for virtual commissioning. Furthermore, the sole visualization of the system is not sufficient to enable realistic interaction. The realistic mapping of the operator's actions in the simulation is also vital. Some contributions integrate physical gaming controllers from the VR system to the virtual scene, allowing for an abstract interaction [6,8]. Other contributions realize the mapping of the operator's hands through motion capturing systems, finding that this increases intuitiveness of interaction [10,11]. These contributions however either base on gaming setups or lack VR visualization. The research gap is thus the live integration of a human operator, mainly his hands, into a realistic VR visualization of a simulation controlled by at least one PLC through motion capturing. To allow interaction similar as with the real system, grasping and other physical interaction with parts is necessary. Furthermore, the pressing of buttons and the interaction with HMI panels is also important. As virtual commissioning is used to test real control logic, the integration of PLCs and, optionally, robot controllers and other specialized controllers is necessary. This is complemented by the requirement that device behavior in the simulation is realistic and that the control algorithms can thus be transferred to the real system. To allow scalable use for different systems, a compatibility with standard CAD systems is necessary to avoid redundant modeling efforts. As displayed in Table 1, no current system fulfills these requirements. We thus present a system that aims to combine the approaches presented in [5][6][7] with interaction capabilities from [10] while maintaining CAD/PLM compatibility.

System Concept
The system is based on the real-time mapping of a simulation environment to a VR scene while also translating inputs and interaction from the user back into the simulation or PLC. Thus, the user is tracked through a motion capturing system during the simulation. We consider the hands as the predominant interaction channel from human to machine. To allow realistic physical interaction, a simulation of physical behavior, especially rigid body physics, such as gravity and collisions is required. As human-machine interaction is often not only done through physical manipulation of parts, buttons and switches but via HMI panels, a corresponding representation in VR is also necessary. To ensure PLM/VC compatibility, the industrial simulation environment is the center component. Physics simulation and interaction capabilities available in the VR engine are not used, as these effects would only affect the visualization rather than the simulation. Hence, the simulation environment is the single source of truth considering system states as well as object and hands placement.

System Setup
The implementation integrates a Leap Motion Controller® (LM) MCS mounted on an HTC Vive® HMD. The LM uses an active infrared stereo camera for hand recognition and tracking. The HMD is tracked via infrared laser-emitting beacons, which trigger sensors on the HMD and allow pose estimation. For fast movements, an inertial measurement unit is also integrated. The simulation is modelled in the Mechatronics Concept Designer® (MCD), a VC tool in the industrial CAD system Siemens NX®, which also features physics simulation capabilities, capable of covering many mechatronics and automation-related simulation tasks [12][13][14]. The VR rendering of the scene is performed in the Unity® Engine. It is hence open for combination with different simulation tools or frameworks such as ROS. PLCSIM Advanced® is used to emulate a SIMATIC S7® PLC, which is connected to an emulated WinCC HMI in TIA Portal®. Two computers are used for the evaluation: a standard CAD laptop workstation (Workstation 1) running the mechatronics simulation and PLC emulation, and a VRready workstation (Workstation 2) running the VR environment and processing tracking data (Fig. 1). This setup enables the reusability of the concept for multiple simulation experts without the need of individual VR-ready computers, since any computer capable of running the VC can be combined with a VR-ready workstation. The position data of each rigid body in MCD is streamed to the VR scene in Workstation 2 via UDP. The tracking data of the HMD is streamed to Workstation 1 via the same protocol, where a virtual LM sensor is placed accordingly in the simulation. This function is necessary to compensate the kinematic dependency between the HMD and the MCS. The hand tracking data, transferred via a TCP Web Socket connection, is then used to place a virtual hand skeleton model, described in [10], relative to the virtual sensor. The hand links are modelled as collision bodies, allowing for physical interaction with other simulation components. Physical and gesture-based gripping functionalities, described in [10], are implemented. As the skeletal visualization of the hand has proven uncanny to some users in VR, an overlay of a more humanoid hand model in the VR environment is implemented, see Fig. 2. The HMI is streamed into the VR environment through a virtual network connection (VNC) as an animation when the user turns the left palm towards his face (Fig. 3).
The hand tracking data of the right index finger is then used to project a pointer into the WinCC environment, allowing for interaction with the virtual touch display.

Evaluation Use-Case
The use-case for the evaluation is a combined assembly and screw-driving cell. Two components are transported via separate conveyors towards an articulated robot arm. Mechanical baffles ensure a correct positioning of the components. The robot arm assembles the components at a processing position. Via conveyor, the assembly is transported to an automated pick-and-place screw driver, mounted to a portal axis, for final processing. The finished assembly is then transported out of the station on the third conveyor, see Fig. 4. The station is controlled through a PLC, monitoring several laser sensors in the station as well as equipment status signals. The entire station, including sensor and actuators, is modelled in MCD. An HMI is used to set the status of the system and the robot separately, as well as monitor signals and performance. The machine setup was not designed for this use case. An already existing simulation model is used for the validation of our system to prove re-usability of existing models. We choose this setup as it unites common automation components, including conveyors, linear axes, automated tools, an articulated robot arm, fixtures and feeders, and thus highlights the universality of the approach.

User Study
The intuitiveness and functionality of the system is tested through a user study. Our system is benchmarked to a mouse-keyboard (KM) setting. The task of the operator is to activate the station and the robot through the HMI panel. Then, new parts to be processed have to be placed on the two inbound conveyors in specific orientation. The parts are located on pillars next to the station. The goal of the simulation is to determine, whether the designed mechanical baffles enable a robust re-orientation of the parts to allow robotic handling. To this end, an automated generation of part instances in the simulation cannot fully satisfy the requirement to map the operators influence on the system. Furthermore, the logic control functionality as well as robot programs are also to be verified through VC. As the station consists of multiple sensors, different actuator types including a robot arm, and mechanical baffles, it represents many of the aspects commonly modelled in VC. Intuitiveness is rated on an ascending scale from 1 to 5. A total of 30 participants took part in the study. 12 participants had no previous experience with the simulation environment, 27 had no experience with the VR environment and none had used a Leap Motion® controller before. The results of the study are displayed in Fig. 5. The KM setup is rated between 1 and 4 with a mean score of 2, while the proposed VR approach is rated between 3 and 5 with a mean score of 4.5.
It is hence concluded that the proposed concept outperforms the common KM setup in terms of intuitiveness. Furthermore, even though 12 participants had no experience with the simulation tool itself, high ratings are prevalent. This hints at a possible use of the system for training purposes, as it is seemingly well usable also for users with no previous simulation experience. Even for experienced users of the simulation tools with no previous VR experience, the system still achieves better ratings.

Conclusion and Outlook
This contribution describes a method to integrate a human operator into a virtual commissioning system at real time. Using VR and motion capturing, an intuitive interaction with the virtual system is enabled. An implementation of the concept for a robot-based production system based on industrial simulation software is shown. The setup is compared to a KM setup in a user study, indicating a significantly higher usability and hinting at new potentials for application in operator training. In future research, the integration of a full body tracking system will be examined. Furthermore, a field study to assess the potentials of using this system for operator training should be conducted. As the VR engine used in our setup is also compatible with established augmented reality devices, further use-cases basing on this technology, like real-time control of a physical system through manual interaction with a digital representation, can be explored. Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.