Keywords

1 Introduction

Automation is one of the key technologies for enhancing the efficiency of industrial production processes. Due to the trends of increasingly high number of variants and small series, industrial robots need to be programmed more and more frequently. Especially Small and Medium Enterprises (SMEs) have difficulties employing automation economically because of their small batch sizes and the high costs [1]. This puts cutting down the costs of robot programming into the spotlight. Today, the dominating programming method in mass production is hybrid programming [2], a two-stage procedure consisting of an extensive offline programming (OLP) and simulation step, followed by a relatively short online commissioning step using teach-in. New approaches that aim to make robot programming more intuitive and less costly include Programming by Demonstration (PbD) as well as Augmented Reality-assisted programming systemsFootnote 1 [4]. They are not as costly and complex as enterprise-grade OLP software suites, yet more efficient and intuitive than online programming, using the teach pendant. These approaches address production processes which have low to medium complexity and thus do not require extensive simulation-based optimization.

ARRPS have been researched extensively [3, 5,6,7,8]. The basic idea is to provide an AR-based human machine interface (HMI) for robot programming that overlays useful virtual information like robot paths with the real environment and essentially aims to replace online programming via the teach pendant. This allows for faster and more intuitive programming. Typically, path points can be programmed via 3D user input and the resulting robot motion can be previewed based on a basic simulation. It has been shown that efficiency could be increased significantly, when compared to conventional teach-in [5]. However, there are still considerable issues with existing concepts including the unavailability of mature input methods and high hardware costs. The goal of this paper is to present a novel input and tracking system for AR-based robot programming, as a step towards mitigating some of the existing deficits. The contributions include:

  • A flexible and cost-efficient input and tracking system based on the VIVE Lighthouse technology,

  • a modular software platform, released as an open-source software package,Footnote 2

  • a demonstration of the basic feasibility of the system to act as an ARRPS,

  • an examination of the accuracy of the augmentations and the end-to-end accuracy of the path point placement that could be achieved with the presented system.

In Chap. 2 the theoretical background and related research are presented and analyzed. Subsequently, in Chap. 3 the concept for the proposed system is detailed, and in Chap. 4 the implementation is briefly outlined. In Chap. 5 the evaluation is presented and the results are thoroughly discussed. The paper concludes with a summary and an outlook on future work in Chap. 6.

2 Related Work

ARRPS systems [3, 5,6,7,8] try to improve robot programming by employing AR as an intuitive HMI utilized at the production site. The most comprehensive milestone publications are presented subsequently.

Lambrecht et al. [6] present an ARRPS that consists of a tablet and a Kinect to provide gesture input. The components are calibrated to each other using ArUco markers. The accuracy of the gesture recognition component is evaluated to be around 6 mm, the end-to-end accuracy of the system is not examined.

Vogl [5] presents a system for robot programming using an infrared-tracked stylus for path point placement. The robot trajectory is projected onto workpieces with high accuracy using a laser projector. The overall system accuracy is not examined, however, based on the components’ accuracies the overall accuracy is likely under 2 mm.

In their recent work Ong et al. [8] show an ARRPS for welding applications. The system is built with the commercial OptiTrack system as the main tracking system. The input device is a computer mouse with OptiTrack markers attached to it. The system is evaluated with user tests, showing that programming time could be saved and the accuracy necessary for the process could be achieved.

Analyzing the literature, little focus has gone into examining end-to-end system accuracy of the proposed concepts, although this is a crucial criterion determining which range of production processes could be covered. As for the components, the presented input devices mostly have low maturity or few functions. The employed tracking systems tend to be either very costly or rather inaccurate. Lastly, none of the systems are released as open-source software.

The aforementioned deficits are addressed by designing a novel input and tracking system that has low costs, yet reasonable accuracy. The input hardware needs to have a good level of maturity and a rich function set. The system shall be implemented in a modular software platform and released as open-source software, so others can build upon it. The system will be evaluated through a demonstration of basic feasibility. Furthermore, an extensive examination of the system accuracy is going to be conducted.

3 Concept

The basis for the proposed system is the Microsoft HoloLens, a modern Optical See-Through Head Mounted Display. It is combined with the Lighthouse (LH) system as an additional outside-in tracking system which was originally built for VR applications. This enables the use of the VIVE controllers and the so-called VIVE Trackers for object tracking. The LH system hasn’t been previously used as an input and tracking system for ARRPS systems.

This setup enables natural and ergonomic input using established controller hardware. The functional space of the controller is rich, multiple buttons and a trackpad can be used. Also, diverse gesture input can be implemented using a 3D-tracked controller. The used VIVE setup can be considered low-cost when compared to other commercially available tracking systems. The LH technology is robust against environmental conditions, as it uses infrared light. Its mean static accuracy has been determined to be < 3 mm [9], which is a very good performance in a low-cost system. The recommended workspace with two LHs is 3.5 × 3.5 m2 but can generally be arbitrarily chosen. The system can be extended by adding more LHs, yielding flexibility in terms of setup. On the flip side, the mobility of the system is limited because the LHs need to be set up. Placing them on movable tripods, combined with their ability to calibrate themselves automatically, decent mobility of the system is still warranted.

Fig. 1
figure 1

System overview

Full size image

Figure 1 shows the system overview, that is, the main components and their interactions. The communication of all components is carried out via the central server. The HoloLens acts as the main user interface. A distributed software architecture is employed where the HoloLens is the front-end, and the server is the back-end. The LH system complements the internal capabilities of the HoloLens. It acts as the central tracking system, tracking the controller as well as the HoloLens with a Tracker mounted on it. The LH system sends the tracking information, as well as the controller inputs to the server. In order to automatically register the virtual AR scene with objects tracked by the LH tracking system, the HoloLens is tracked externally. The robot interface exchanges robot motion commands as well as the current state of the robot with the server. A Tracker is mounted initially to the robot flange to reference the robot with the LH, it can be removed during usage of the system.

4 Implementation

Figure 2 shows the system architecture. The server is implemented using a modular microservice architecture. It provides essential services to the HoloLens application, namely, the tracking hub service, the robot path service and the registration service. The tracking hub service collects all the information from the LH tracking system. Additionally, the registration service helps with the Lighthouse-HoloLens-registration procedure. The robot path service stores robot paths and controls the robot via the robot interface. Finally, the front-end app on the HoloLens manages the application state and provides the user interface.

Fig. 2
figure 2

Software architecture

Full size image

The server is run in a Docker container for system-independent and robust operation. Internally, the communication is carried out via gRPC. Since the HoloLens does not support gRPC, TCP is used for the communication between the server and the HoloLens. The server is implemented using the Python programming language, whereas the HoloLens front-end is designed using the Unity game engine. The latter is rather basic, as the focus of this work does not lie on the user interface but rather on the tracking system. The robot-specific robot interface is not implemented, instead the data is transferred manually. It could be implemented in the future by using the API of the robot manufacturer or using a common robot interface like ROS (Robot Operating System). The robot used is a KUKA KR 6 R900 sixx, and the interface to the LH system is implemented with SteamVR.

In order for the tracking scheme to work in practice, the Tracker needs to be calibrated with respect to the HoloLens it is mounted on, we call this procedure HoloLens-Lighthouse-registration. It needs to be carried out once by each user.

Figure 3 shows a scheme of the transformations, the transformation \({}^{\mathrm{HoloLens}}{T}_{\mathrm{HoloTracker}}\) is sought-after. In order to determine it, the user is asked to align a real object (with a Tracker mounted onto it) with a virtual copy that he can control through the AR interface. Based on this, correspondences, which are the same points expressed in two different coordinate frames, can be collected. The collected data set can be used to solve for the unknown transformation \({}^{\mathrm{HoloLens}}{T}_{\mathrm{HoloTracker}}\) using Point Set Registration methods [10].

Fig. 3
figure 3

HoloLens-Lighthouse-registration

Full size image

In order to ultimately be able to move the robot flange along user-defined path points, the robot location needs to be known with respect to the LH system. The transformation scheme is shown in Fig. 4a), the goal is to determine the unknown transformation between the robot base and the LH system \({}^{\mathrm{Base}}{T}_{\mathrm{Lighthouse}}\). A procedure called Robot-Lighthouse-referencing is proposed. A Tracker is initially attached to the robot flange, it is then moved to discrete locations according to a predefined robot program (as depicted in Fig. 4b)). At the same time, the LH tracking system records the locations of the Tracker. The resulting correspondences can be used to determine the unknown transformation using Point Set Registration methods. After the referencing is done, the Tracker can be removed. To make this procedure more efficient, a quick changer system for the robot flange can be used. All of the steps in the Robot-Lighthouse-referencing could potentially be automated.

Fig. 4
figure 4

Robot-Lighthouse-referencing: a Transformation scheme, b Robot movement scheme

Full size image

5 Evaluation

To evaluate the system a basic demonstration is conducted in Sect. 5.1. An in-depth examination of the system accuracy is presented in Sect. 5.2

5.1 Demonstration

To evaluate the ARRPS platform, a basic functional demonstration has been conducted. The essential feature set to prove the feasibility are basic tools for robot path creation and editing using the controller.

This feature set has been implemented and shown to work reliably. The creation of a robot path is depicted in Fig. 7. To program a path point, the user places the lower tip of the controller to the desired location, where it gets created after pressing the trigger button. By pressing the trackpad, the motion type can be altered (Point-to-Point or Linear). Multiple points are visually connected with lines, whose color indicates the motion type. Via the Menu button, the last point can be deleted. The Grip button saves the path to the storage.

The result of the programming process is a list of path points stored on the server which could be transferred through the robot interface. However, this transfer has not been implemented. Instead, the robot path is entered manually in the robot teach pendant in order to validate it. Further and more complex functionality could be implemented; however, creating a functionally rich and mature HMI was not the goal of this work.

5.2 Examination of the System Accuracy

Subsequently, the accuracy of the system is evaluated. First, an error analysis is conducted to explain which errors are examined and to understand what components contribute to the overall errors. Afterwards, the augmentation error and the Robot-Lighthouse-referencing are examined separately. Finally, the overall end-to-end accuracy of the path point placement is examined because it is the system’s most relevant performance characteristic.

Error Analysis

First, the errors and transformations of the path point placement are presented. The path point placement error consists of the user error, the LH tracking error, the Robot-Lighthouse-referencing error and the robot error, as shown in Fig. 5. This leads to a deviation between the location intended by the user and the location the robot would actually move to.

Fig. 5
figure 5

Path point placement error

Full size image

Note that the AR display is not playing any role in this, since the path point location is created directly based on the controller location and transformed into robot coordinates. The location displayed in AR is meant only as a visual assistance and represents the approximate path point location. This is a deliberate design choice because it reduces the path point placement error considerably. Other designs could still be implemented with the platform, but should be expected to have a different accuracy.

Fig. 6
figure 6

Augmentation error

Full size image

Even though it’s not used for path point placement, the error of the augmentation is still important because augmentations should be near the correct locations in order to be useful for user information and guidance. As shown in Fig. 6, the augmentation error comprises the tracking error, the HoloLens-Lighthouse-registration error as well as the HoloLens error. The HoloLens error includes all errors that come from the AR display, such as the internal tracking and display error.

Augmentation Error

The augmentation is evaluated with user-generated ground truth data. To acquire it, the user is asked to align a real controller with a virtual one that he can control via the AR system. This is repeated in five different locations across a flat, rectangular workspace of 0.55 m edge length. The LHs are located at the corners of a 4 × 4 m rectangle, at the center of which the workspace is located. The absolute error of the augmentation is measured as (15.6 ± 10.3 mm) translationally and (1.0 ± 1.1°) rotationally. As expected, the accuracy of the augmentations is not very high, which is the reason why they are not used for the actual path point placement.

Accuracy of the Robot-Lighthouse-referencing

The Robot-Lighthouse-referencing is evaluated by adding a set of test points to the Robot-Lighthouse-referencing procedure and evaluating the registration error for this test set, using the transformation that was calculated using the original data set. All points are located within a cuboid of 0.55 × 0.55 × 0.45 m3 volume. The absolute error of the test set is determined as (10.8 ± 3.0 mm). These results are worse than the expectations before the experiments. The observed error is of random nature, as no axis-specific bias could be proven with statistical t-tests. The following steps have been taken to find out if they can reduce the error: Variation of position and number of LHs, covering of metallic surfaces, change of the axis configuration of the robot, as well as change of the rotations of the robot flange. However, the error could not be reduced. More research on this problem is needed.

End-to-end Accuracy of the Path Point Placement

To examine the end-to-end accuracy of the path point placement, the following experiment is conducted. The user places a path point at one corner of a cuboid shaped, 3D printed workpiece that is mounted on a tripod. The placement is repeated 5 times per point, so as to rule out random user error at each location. Subsequently, the tip of a 3D printed tool that holds a needle is moved to the programmed location. This setup is shown in Fig. 7. Both the position of the workpiece and the tool are measured using a Leica LTD 800 laser tracker. That is why the tool and the workpiece are constructed to be able to hold three laser tracker targets each. This is repeated 14 times within a cuboid shaped workspace of approximately 0.6 × 0.6 × 0.25 m3 volume.

Fig. 7
figure 7

(a) Path point placement in front of a workpiece, (b) Path point placement experiment

Full size image

Table 1 shows an overview of the errors that have been measured. The absolute value of the error is (10.7 ± 3.5 mm). There is a bias in the z-axis of 9.3 mm. A possible explanation for the bias is the random error of the Robot-Lighthouse-referencing, which was carried out once previous to the path point placement and thus would appear as a systematic error in the subsequent experiment. Thus, like mentioned before, the referencing procedure should be revised. Overall, a mean absolute error of 10.7 mm is not as good as was expected when designing the system; however, a large bias suggests that by fixing the suspected issue, the achievable accuracy should be substantially higher.

Only a relatively small workspace was tested because the robot used in the experiments limited the workspace of the evaluation procedure. It is expected that the accuracy will be lower in bigger workspaces.

Table 1 Error of the path point placement
Full size table

5.3 Discussion

To conclude, the mean absolute error of 10.7 mm in a relatively small workspace is decent, especially when considering it is a low-cost system, but not as good as was expected when designing the system. On top of that, in real-world application the overall error is expected to increase rather than decrease, due to perturbations and generally less ideal conditions, so robustness needs to be evaluated as well. Because of the determined accuracy only industrial processes such as painting and handling could be programmed using the presented system, since they usually have a compatible tolerance of the tool positioning. The presented stylus-based placement method is ideal for the quick robot program creation of small programs from scratch. However, development of more complex functionality, e.g. the commissioning of a robot program that was planned in a simulation software is also possible. At the same time, the applications need to be limited to medium-size workspaces like 1 m3 which is likely a sensible trade-off between workspace size and accuracy for this particular setup. To sum up, the accuracy in medium-size workspaces will be quite high, if the problem in the referencing can be solved.

6 Conclusion

In this work, a flexible and cost-efficient input and tracking system based on the VIVE technology has been developed. Overall, the system is feasible as a flexible and low-cost ARRPS system. The end-to-end mean absolute error of the path point placement is 10.7 mm. The VIVE controller integration allows for many interesting possibilities when designing the user experience. The software platform can be a starting point for other developers to implement their own ARRPS-related ideas.

In future work, the robot interface could be implemented, e.g. using ROS. The robustness of the Robot-Lighthouse-referencing should be improved, so as to improve overall accuracy. Also, the setup time of the Robot-Lighthouse-referencing should be reduced by fully automating the procedure. Furthermore, the system should be evaluated in terms of usability and ergonomics for the intended use-cases.