Vision based trajectory dynamic compensation system of industrial robot

Abstract

This study aims to enhance the flexibility of robots in the manufacturing process to meet the growing demand for flexible manufacturing systems by constructing a dynamic compensation system. The system enables the end effector of the robot to track the trajectory contour on the horizontal plane, even if there is a deviation between the teaching trajectory and the target trajectory contour. The proposed system employs an innovative mechanism design and algorithm that enables dynamic compensation of the robot’s absolute position in the XY-plane, even when the end effector is obstructed. The system uses an XY-stage and two industrial cameras, with each axis being separately controlled. To detect and compensate for errors in the end effector and target trajectory, the two cameras are aligned with the X- and Y-direction linear stage, respectively. The cameras are fixed with respect to the linear stage, and the end effector is placed on the stage work surface. Unlike traditional stereo cameras, the cameras in this system do not require calibration for their positional relationship. To verify the system’s effectiveness, this study set a group of teaching and target trajectory contours with a maximum deviation of over than 8 mm. Under the compensation of the system, the maximum error between the end effector and the target trajectory contour was 22.8 pixels (1.34 mm), with an average absolute error of 5.4 pixels (0.32 mm). Compared to the maximum error of 8.45 mm before compensation, the error has been reduced by 84 %.

1 Introduction

1.1 Background

Flexible manufacturing is a production method designed to easily adapt to changes in the types of products being manufactured. It can easily adjust to variations in the type and quantity of goods produced, compared to traditional manufacturing methods. Consequently, its adoption has been intensifying in recent years, even though flexible manufacturing systems frequently suggest elevated expenses and dependency on adept operators. Many researchers have attempted to categorize and summarize flexible manufacturing [1,2,3], but still no unified standard exists. For example, in [4], flexible manufacturing was divided into two categories based on the request of flexibility: environmental uncertainty and variability of the products and processes. Sethi et al. [5] organized flexible manufacturing into eleven categories (e.g., process flexibility, product flexibility, routing flexibility, and others [5]). Among them, the flexibility of the system’s components was divided into three categories: machine flexibility, material handling flexibility, and operation flexibility. In this study, we focus on the system’s machine flexibility (i.e., various operations performed without set-up change).

1.2 Prior art

In recent years, the application of industrial robots in the industrial field has become increasingly widespread [6]. For instance, robots are not only used in traditional industrial sectors such as welding, assembly, and material handling, but in recent years, robots have also been used in emerging fields such as 3D printing [7,8,9] and 3D scanning [10]. This is primarily due to the variety of industrial robots and their high degree of freedom, allowing them to meet various application scenarios.

Industrial robots are typically programmed using the teach-in method, where a human manipulates the robot manually with a teaching pendant, and the robot memorizes it [11]. While the operator’s command trajectory may or may not be optimal, when the task performed by the robot is sufficiently specific, the operator can manually adjust its movements to successfully carry out the required task. However, for flexible manufacturing that needs to deal with changes in product types, the teach-in programming and manual modification of the robot’s motion are no longer possible.

Addressing the issue of applying industrial robots to flexible manufacturing, many researchers have proposed their own solutions. The main method is to install various sensors on the robot to meet different processing needs. This structure is also known as a macro/micro manipulator [12].

Djelal et al. [13] proposed a system based on vision and force sensors to address the situation where the robot tracks an unknown constraint surface in the presence of uncertainties in the robot’s kinematics, dynamics, and camera models. The system’s effectiveness was confirmed through simulation. González et al. [14] focus on the application of the vision-force system in robotic machining, providing a more specific application scenario: surface treatment of the workpiece. In this study, the vision sensor is used to provide visual information to the robot operator. A force sensor is included to protect both the workpiece and the robot tool. A similar proposal based on force sensors for implementing flexible manufacturing was also proposed in [15, 16].

In [17,18,19], some researchers proposed a dynamic compensation method and applied it to a human-robot system to improve the flexibility of that system. Huang et al. [18] provide a macro/micro manipulator based on visual feedback to meet the demand for robot manufacturing in a zero-waiting time production line. The application of this manipulator was extended to contour followed by human-robot collaboration in [19]. Similar research on a vision-feedback-based macro/micro manipulator is also presented in [20].

1.3 Research gap

In recent years, the combination of vision and robotics aimed at enhancing the reliability of robots has become a mainstream solution. Lachmayer et al. [21] present an onboard localization system to enhance robot positioning accuracy. Garcia et al. [22] use a fixed camera to detect the relative position of the workpiece and the robot and correct the robot’s errors based on this information. Baek et al. [23] and Zou et al. [24] discuss improvements in robot accuracy within centimeter-level spaces, enhancing robot reliability through a typical eye-in-hand structure. In all these studies, there is a problem of limited directional flexibility. Moreover, in studies [22,23,24], feedback needs to be implemented through the robot, meaning that the final compensation accuracy is still limited by the robot’s performance. Although [21] adopts a macro/micro structure design, the robot is used as the micro part, which means that its compensation accuracy is still constrained by the robot’s performance.

While prior research has made significant advancements in flexible manufacturing systems, especially concerning industrial robots and macro/micro manipulator structures, several gaps persist.

While some researchers in [17,18,19] have proposed dynamic compensation methods, there is limited work on a system that can seamlessly integrate vision and execution modules for broader applications. Many of these systems are either too complex for practical application or lack the necessary robustness to handle unforeseen manufacturing challenges.

Furthermore, the issue of camera occlusion by the end effector, although acknowledged, hasn’t been solved in the current research, especially in scenarios where precise trajectory tracking is critical.

The current research lacks a comprehensive solution that can dynamically compensate for such occlusions while maintaining manufacturing accuracy and having a simple and easy-to-set-up structure.

1.4 Purpose and contribution

Building upon the aforementioned research background, the aim of this study is to establish a more efficient and robust system for flexible manufacturing. This study presents a system designed to dynamically adjust the position of a robot’s end effector, accomplished by installing an additional vision module and execution module on the robot tool flange. To evaluate the system’s effectiveness, it was applied in an early application scenario in flexible manufacturing, specifically for compensating the contour tracking of an unknown target.

Moreover, this system makes real-time compensation of target trajectory possible when an end effector is installed, using customized fixtures.

Furthermore, considering the occlusion of the camera by the end effector, the study proposes a dynamic compensation algorithm based on the camera and XY-stage. This algorithm sets Regions of Interest (ROIs) on both sides of the end effector, identifies the error between the trajectory contour center and the end effector, and separately controls the X and Y axes, achieving dynamic compensation for the end effector.

Compared to traditional methods of calibrating robots to improve reliability [22,23,24], this study adopts a macro/micro structure, installing an additional actuation module on the robot’s hand to perform compensation actions. Since the actuation module operates independently of the robot’s motion, the compensation precision at the robot’s end is not limited by the robot’s performance. At the same time, the vertically arranged two cameras also make the compensation of this system in the 2D plane unrestricted by direction.

Fig. 1

System construction

2 Methodology

2.1 System construction

Based on the early application scenario in flexible manufacturing, a dynamic compensation system consisting of XY-stage and two industrial cameras is proposed in this study, and the system structure is shown in Fig. 1. The hardware components of this system can be divided into the following three parts according to their functions:

• 1. Industrial robotic module: featuring an expansive operational workspace with limited precision in teaching accuracy.

• 2. Vision module: consists of an image processing PC and two cameras rigidly mounted on the linear stage. PC not only calculates the target compensation center, ${P}_{n,\textrm{c}} \in \mathbb {R}^2$, but also determines trajectory state, $State_n \in \mathbb {R}$.

• 3. Execution module: consists of an XY-stage, two motors, and a controller. Using the target value input from the compensation center XY-stage is controlled to compensate for the error between the end effector and the target. Provide a small range of workspace and high absolute positioning accuracy.

The two dimensions of the XY-stage are controlled by the data returned by the industrial camera which is rigidly mounted on the linear stage in the corresponding direction. The vision module and execution module are connected with a custom fixture (see Fig. 2), which is adjustable to modify the camera’s posture to position the robot’s end-effector at the center of the camera’s field of view. Compared to the solution in [19], the cameras in the proposed system do not occupy the space of the end effector, making it possible to install conventional machining tools on the robot, which means that this system is more flexible than the previous system.

Fig. 2

Structural details of the execution module and the vision module

2.2 Vision module

The vision module detects the trajectory contour and relays its state to the execution module. This module consists of two cameras, each oriented in different directions, as shown in Fig. 2. The camera installed on the linear stage in the vision module is named $CAM_{n}$, $(n=1,2)$. In the X-direction, $n\!=\!1$; in the Y-direction, $n\!=\!2$.

Fig. 3

ROIs setting

2.2.1 Identify compensation center

To identify the compensation center, this study proposes a high-speed trajectory contour recognition scheme. This scheme improves the image processing speed by setting ROIs (Regions of Interest) in the camera’s imaging plane, ensuring that the recognition process remains unaffected by the robot’s end effector. $P_{n,\textrm{0}}$ is the imaging position of the end effector on the camera imaging plane as shown in Fig. 3 by a red cross. $P_{n,\textrm{0}}$ is also the coordinate origin (0,0) of the imaging plane for the coordinate system shown by black arrows (Unlike the conventional image coordinate system with the origin at the upper left corner). Two ROIs, $ROI_{n, 1}$ and $ROI_{n, 2}$, were set within each camera’s imaging area. $ROI_{n, i}$ are the blue box area of symmetry about $P_{n,\textrm{0}}$. To make the $ROI_{n,i}$ coverage omnidirectional, $ROI_{n,i}$’s width, $W_{n,i}$, height, $H_{n,i}$, and the distance, $D_{n,i}$, from ${P}_{n,\textrm{c}}$, need to satisfy Eq. (1). If the ROIs do not meet the requirements of Eq. 1, it will result in blind spots when the vision module detects trajectory contour.

$$\begin{aligned} D_{n,i}+W_{n,i} < \frac{H_{n,i}}{2} (n=1,2)(i=1,2) \end{aligned}$$

(1)

Trajectory contours within $ROI_{n, i}$ are identified by inverting the original image and setting appropriate thresholds for grayscale values. The zeroth order moment, $Mom0_{n, i} \in \mathbb {R}^2$, and centroid, $P_{n, i} \in \mathbb {R}^2$, of trajectory contours in $ROI_{n, i}$ are calculated by Eqs. 2 and 3.

$$\begin{aligned} Mom0_{n, i}=\sum _{x_n}\sum _{y_n} I(x_n,y_n) \end{aligned}$$

(2)

$$\begin{aligned} \begin{aligned} P_{n, i}={} & {} \\{} & {} \Biggl (\frac{{\displaystyle \sum _{x_n}\sum _{y_n} x_n I(x_n,y_n)}}{{\displaystyle \sum _{x_n}\sum _{y_n} I(x_n,y_n)}}, \frac{{\displaystyle \sum _{x_n}\sum _{y_n} y_n I(x_n,y_n)}}{{\displaystyle \sum _{x_n}\sum _{y_n} I(x_n,y_n)}}\Biggr ) \end{aligned} \end{aligned}$$

(3)

Here, $(x_n,y_n)$ refers to the row and column index, and $I(x_n,y_n)$ refers to the intensity at that location $(x_n,y_n)$ in $CAM_n$.

To investigate the effectiveness of the proposed system in a simpler way, target compensation center ${P}_{n,\textrm{c}}\in \mathbb {R}$ is calculated as the midpoint of $P_{n,1}$ and $P_{n, 2}$ as shown in Eq. 4.

$$\begin{aligned} {P}_{n,\textrm{c}}= \frac{P_{n,1}+P_{n,2}}{2}\begin{bmatrix} 0&1 \end{bmatrix}^\top \end{aligned}$$

(4)

2.2.2 Determine trajectory state

For the XY-stage, different trajectory angles correspond to different operating states. In the event that the trajectory runs parallel to the X-axis, there’s no need for compensation on the linear stage in the X-direction. Conversely, if the trajectory is tilted at a 45-degree angle relative to the X-axis, both axes of the XY-stage must be involved in the compensation process. Notably, with a constantly shifting trajectory, deciding which axes should participate in the compensation emerges as a complex challenge.

This study proposes an algorithm for the vision module to determine the contour state, $State_n$, and outputs it to the controller of the execution module. The torque of the execution module is influenced by the value of $State_n$ (see Section 2.3). The algorithm acting in the vision module is as follows:

• 1. Set the upper and lower thresholds, ThershMax and ThershMin for the trajectory area in $ROI_{n, i}$.

• 2. Compare whether the area in the $ROI_{n, i}$ is within the threshold, return ${result}_{n, i} = 1$ if it is fully within the range. In other cases, return ${result}_{n, i} = 0$.

• 3. When the areas within $ROI_{n, 1}$ and $ROI_{n, 2}$ meet the requirement, i.e., ${result}_{n, 1}\ \& \& \ {result}_{n, 2}\!=\!1$, return $State_{n}=1$ and stage involved in the compensation. In other cases, return $State_{n}=0$.

The above algorithm is shown in pseudo-code form in Algorithm 1.

Algorithm 1

Determine trajectory state.

2.3 Execution module

The execution module is comprised of an XY-stage and two motors to drive the XY-stage. The proposed execution module is controlled by the feedback information, ${P}_{n,\textrm{c}}$ and $State_n$, from the vision module. As previously demonstrated, since the cameras in the vision module are rigidly mounted to the linear stages on the XY-stage, any movement of the XY-stage induces a shift in the camera’s coordinate system, resulting in feedback. When $State_n=1 (n=1,2)$, the execution module was controlled by a Proportional-Integral (PI) control method. The PI control law is as follows:

$$\begin{aligned} \tau _n=-K_p \cdot {P}_{n,\textrm{c}} -K_i \cdot \sum {P}_{n,\textrm{c}} \end{aligned}$$

(5)

where $K_p$ and $K_i$ are coefficients for proportional and integral terms, respectively. $\tau _n \in \mathbb {R}$ is the control reference under PI control, used to provide torque for the execution module.

The execution module has the characteristics of a small working space and high absolute positioning accuracy, while the industrial robot has a low teaching accuracy. To compensate for errors from the robot in all directions, the execution module should maintain an ideal position that allows movement in all directions. This means the XY-stage should avoid reaching actuator saturation, ensuring unrestricted motion in any direction. Resolving the conflict between execution module accuracy and actuator saturation becomes a specific problem. Based on the fact that the target position is not unique in the target trajectory tracking task, under the premise of $State_1 \Vert State_2=1$, a balancing coefficient $C_{n,f}$ is introduced into the system. $C_{n,f}$ is used to adjust the weight of two stages in the tracking process to avoid actuator saturation. When the position of the linear stage exceeds the boundary $B_{w}$, the weight of this linear stage in the system is reduced. The values of balancing coefficient $C_{n,f}$ are shown in Eq. 6.

$$ \begin{aligned} C_{n,f}= {\left\{ \begin{array}{ll} C &{} \text {if } \hat{q}_n> B_{w} \ \& \ \hat{q}_n\cdot \tau _n >0 \\ 1 &{} \text {if } otherwise\\ \end{array}\right. } \end{aligned}$$

(6)

$\hat{q}_n$ is the command stage position value for the X-linear stage (when $n=1$) and Y-linear stage (when $n=2$). The C is a constant between 1 and 0. The control reference ${\tau _n}^{'}$ after the introduction of $C_{n,f}$ is shown in Eq. 7

$$\begin{aligned} {\tau _n}^{'}=C_{n,f} \cdot {\tau _n} \end{aligned}$$

(7)

When $State_n=0$, it means that there are stages that are not involved in the compensation at this time, and $P_{n,c}$ is manually set to 0 to mask any potential interference caused by the vision module. To increase the robustness of the system, when $State_n=0$ and this state lasts for 0.5 s, the stage will be moved toward the home position using Proportional (P) control. The control reference in this situation is $\tau _n^0$ as shown in Eq. 8.

$$\begin{aligned} \tau _n^0= {\left\{ \begin{array}{ll} -V \cdot q_n &{} \text {if } \vert q_n \vert > B_{0}\\ 0 &{} \text {if } otherwise\\ \end{array}\right. } \end{aligned}$$

(8)

The coefficient V is a constant greater than 0. $q_n$ represents the current position value of the stage obtained through the controller. $B_{0}$ is the Boundary value. When $\vert q_n\vert < B_{0}$, the stage will be considered close to the home position, and the stage will remain at the current position.

2.4 Summary

As the input of the entire system, the information from two camera views is transmitted to an image processing PC for further processing. The ${P}_{n,\textrm{c}}$ obtained through image processing and the $State_{n}$ computed by Algorithm 1 is outputted by the image processing PC to the controller of the execution module. The controller utilizes the input of ${P}_{n,\textrm{c}}$ and $State_{n}$ to determine the control reference value and drive the execution module motion accordingly. As stated in Section 2.1, the vision module and execution module are connected by a custom fixture, such that the compensation effect on the robot end-effector resulting from the execution module motion is instantaneously fed back to the camera’s field of view, thereby completing the feedback process.

Table 1 Specifications of the motors of the execution module

3 Experimental verification of the system’s feasibility

In this study, an UR3e robotic system, manufactured by Universal Robots Co., Ltd., served as the industrial robotic module, offering an extensive working area. Figure 2 shows the details of the execution module and the vision module. In Fig. 2, a customized XY-stage is derived by motors, featuring a 1,000 Hz frequency response and 20 mm total stroke for each axis, functioning as an execution module. The specifications for the motors of the execution module were estimated by an accelerometer and are shown in Table 1. This XY-stage is made of aluminum for mass optimization and dynamics. The vision module utilizes two optical inspection cameras MQ013MG-ON produced by XIMEA Corp., capable of operating in grayscale mode within the ROI, achieving 500 FPS with a resolution of 640 $\times$ 512. Cameras were positioned to the X-direction and Y-direction work planes, respectively. Figure 4 shows the Experiment setup. A customized pen, serving as a manipulation instrument, is fixed to the XY-stage’s work plane. By adjusting the fixture, the cameras’ field of view centers have been made to coincide with the pen tip.

Fig. 4

Experiment setup: execution module and vision module installed at robot’s end effector

This study places special emphasis on defining the concept of “real-time” within the system. “Real-time” is characterized by the stage’s rapid response following the camera’s detection of the trajectory contour. The stage adjusts according to the target’s position to maintain alignment between the robot’s end and the trajectory contour. To quantify this real-time performance, the focus is primarily on the “latency” indicator of the system. In this context, “latency” is the time interval from the camera’s detection of the trajectory contour to the commencement of the stage’s movement.

Referring to previous research results, in [18], an XY-stage compensation system equipped with a 1000 Hz camera, the system’s latency was measured at 2–3 ms. In this study, the camera operates at a frequency of 500 FPS. Based on this, the latency of the compensation system discussed in this paper is approximately 3–4 ms. This estimate is based on the direct relationship between camera frequency and system processing speed.

As shown in Fig. 4, an irregular trajectory with a width of 2 mm is printed on paper. This paper is situated on the robot’s work plane. In Fig. 5, the black contour represents the target trajectory, while the blue contour denotes the teaching trajectory. Considering the total stroke of the XY-stage, the maximum deviation between the teaching and target trajectories remains under 10 mm (i.e., the target trajectory is constrained within the annular region enclosed by dotted lines shown in Fig. 5).

Fig. 5

Teaching and target trajectories contours in robot base coordinate system (Z direction value is -30 mm)

The choice of resolution and field of view in the vision module is closely related to the desired precision. In this study, the width of the trajectory is 2 mm. Ideally, during the compensation process, the pen tip should be within the trajectory contour. Therefore, the expected average error should be less than 1 mm, which is half the width of the trajectory contour. For this purpose, in the vision module, the minimum precision requirement is that the physical width corresponding to 1 pixel should be less than 1 mm. Additionally, to meet the requirements of trajectory recognition, the camera’s field of view should include both sides of the pen tip.

The camera chosen in the study has a pixel-to-physical width ratio of 1 pixel is 0.06 mm. The field of view range is shown in Fig. 6, which meets the performance requirements for the vision module.

Furthermore, the frame rate of the vision module also affects the overall system’s response speed, thereby impacting the system’s compensation accuracy. In the experiment setup, the robot’s motion speed is 40 mm/s, and to ensure the robot’s end always remains within the trajectory contour, the absolute average error should not exceed 1 mm, so the system’s maximum allowable latency is 25 ms. In this study, the frame rate of the vision module is 500FPS, with the system’s maximum delay being 4 ms.

Considering the application of this system in industrial scenarios, such as the glue application process in automotive manufacturing, where the robot’s end position moves quickly but the requirement for compensation precision is low, the resolution can be relatively reduced, but the frame rate can be further increased to reduce the system’s latency. In scenarios like welding, where the robot’s end moves slowly but high compensation precision is needed, the camera’s resolution should be increased, and the frame rate can be reduced to decrease the time consumed in image processing.

Fig. 6

Image processing result

3.1 Experimental schemes for verifying the compensation system effect

In order to verify the compensation effect of the compensation system, experiment schemes with two steps are proposed. To enable the robot end effector to move along the teaching trajectory, a straightforward teaching method was used in the experimental schemes. In the robot base coordinate system, the robot was controlled to perform a circular motion in the XY-plane with a radius of 100 mm centered at (-150 mm, -330 mm) while maintaining a height of -30 mm. In the proposed schemes, the robot starts at the position shown in Fig. 4 and follows the teaching trajectory shown in Fig. 5 as the command value. The robot will execute a clockwise movement in the XY-plane until it returns to its starting position and comes to a stop. This process will be referred to as a “cycle” in the following sections. The proposed schemes are as follows:

• 1. Without compensation, the robot completes one cycle at a speed of 40 mm/s.

• 2. With compensation, the robot completes one cycle at a speed of 40 mm/s.

Steps 1 and 2 are intended to indicate the compensation effect of the system.

3.2 System compensation effect evaluation

To identify the compensation center, the size, and location of ROIs were adjusted based on the shape and volume of the customized pen. The parameters defining the ROIs are shown in Table 2 ($n=1,2$, $i=1,2$). The parameters defining the control reference are shown in Table 3.

Table 2 Parameters for defining control reference

Table 3 Parameters for defining ROIs size and location

Image processing results calculated by Algorithm 1 at the moment of 14.1 s after the system startup are shown in Fig. 6. Figure 6a shows the image in the X-direction camera field of view. Three green circles in the field of view, from left to right, correspond to the positions of $P_{1,1}$, ${P}_{1,\textrm{c}}$ and $P_{1,2}$ after image processing. Figure 6b shows an image in the Y-direction camera field of view. Similarly, the three green circles from left to right in the field of view corresponding to the positions $P_{2,1}$, ${P}_{2,\textrm{c}}$ and $P_{2,2}$ after image processing.

Fig. 7

Snapshots of trajectory contour tracking process

Figure 7 shows the continuous images during one cycle of trajectory contour tracking. In the snapshot of Fig. 7, two linear trajectory contour regions (Fig. 7b, d) where only a single camera is involved in compensation are selected, as well as four trajectory contour turning regions (Fig. 7a, c, e, f) where both cameras are simultaneously involved in compensation. The tracking errors during the system moving in these four regions are also represented in the cyan-colored area of Fig. 10.

In one cycle, the motion of the execution module is depicted in Fig. 8. The red line represents the motion values of the X-direction stage, while the blue line represents the motion values of the Y-direction stage. Figure 8 also illustrates how the control reference controls the stages under certain conditions.

When Algorithm 1 determines that the stage does not require compensation ($State_n=0$), the stage will move toward the home position until the distance between the current position and the home position is less than 0.5mm. This process is shown within the dashed ellipse in Fig. 8.

As the stage’s motion values approach the exceed the boundary value, the torque applied to the stage will decrease (refer to Eq. 7), resulting in a reduction in motion speed, as shown in the Partial view of Fig. 8.

Fig. 8

Motion of the execution module at an endpoint speed of 40 mm/s

Fig. 9

Compare the tracking errors in the image before and after compensation system activation at an endpoint speed of 40 mm/s

Fig. 10

Compare the absolute distance in image before and after compensation system activation at an endpoint speed of 40 mm/s

Based on the results of image processing, the tracking errors of the system are shown in Fig. 9, where the green and orange dot lines respectively represent the uncompensated X-direction center ${P}_{1,\textrm{c}}$ and Y-direction center ${P}_{2,\textrm{c}}$. In Fig. 9, ${P}_{n,\textrm{c}}$ suddenly jump to 0 at certain moments. This occurs because the trajectory contour does not meet the constraint conditions of Algorithm 1. In such cases, the system assigns the value 0 to ${P}_{n,\textrm{c}}$. At this point, the system cannot perform compensation activities. The red and blue lines represent ${P}_{1,\textrm{c}}$ and ${P}_{2,\textrm{c}}$ during compensation. Compared to the values in the uncompensated state, the error is significantly reduced, indicating the effectiveness of the compensation in the proposed system.

To assess the compensation effect of the proposed system more intuitively, Fig. 10 shows the absolute distance between the center of the trajectory contour and $P_{n,\textrm{0}}$ before and after compensation, denoted as ${P}_\textrm{c} \in \mathbb {R}$. ${P}_\textrm{c}$ is calculated using Eq. 9. In Fig. 10, the red dotted line represents the variation of ${P}_\textrm{c}$ before compensation. Before compensation, the maximum error recognizable by the vision module is 121 pixels, and there is a significant discontinuous region, indicating that the trajectory contour does not meet the constraint conditions of the Algorithm 1, such as extending beyond the ROIs. In contrast, the blue line represents the variation of ${P}_\textrm{c}$ after compensation.

$$\begin{aligned} {P}_\textrm{c}=\sqrt{\begin{Vmatrix}{P}_{1,\textrm{c}}-P_{1,\textrm{0}}\end{Vmatrix}^2+\begin{Vmatrix}{P}_{2,\textrm{c}}-P_{2,\textrm{0}}\end{Vmatrix}^2}\end{aligned}$$

(9)

In Fig. 10, there are four regions (cyan rectangular areas) where the value of ${P}_\textrm{c}$ changes rapidly. These regions correspond to the points where the robot undergoes trajectory contour inflection. Before this moment, the execution module transitions from being compensated by only one linear stage. In these regions, both axes of the execution module are involved in the compensation simultaneously. It is evident that compared to the regions where only a single linear stage is involved in the compensation, the compensation control is more complex in the cyan areas. However, these regions do not exhibit higher tracking errors.

The absolute tracking errors ($\mu _{|x|}$,$\mu _{|y|}$) and average tracking errors ($\mu _{x}$,$\mu _{y}$) in the X- and Y- directions for the Fig. 9 are presented in Table 4. Furthermore, the maximum value and average value of ${P}_\textrm{c}$ are also presented in Table 4. The maximum value of ${P}_\textrm{c}$ after compensation is 22.8 pixels (1.34mm), with an average absolute value of 5.4 pixels (0.32mm). The values in Table 4 further validates the effectiveness of this proposed system.

Table 4 Tracking errors before and after compensation system activation

4 Comparative evaluation

To discuss the effectiveness of the $C_{n,f}$ in this study and the effect of contour recognition on compensation, two comparative experiments were introduced. The experimental setup is shown in Fig. 11. In the comparative experiments, the trajectory contour in Fig. 5 was rotated by -33 degrees along the Z-axis of the robot’s base coordinate system (following the right-hand rule). The rotated trajectory contour is better for comparing comparison of the system’s performance with and without $C_{n,f}$.

Fig. 11

Setup of the comparative experiments: The target trajectory was rotated by -33 degrees

4.1 Comparative experiment 1: verification of the effectiveness of $C_{n,f}$

In comparative experiment 1, to evaluate the contribution of $C_{n,f}$ on the stage’s travel, the boundary value $B_w$ was reduced to 5 mm. The rest of the experimental parameters are the same as those in Chapter 3. The specific experimental schemes are as follows:

• 1. With speed limitation ($C_{n,f}$=5 mm), the robot completes one cycle at a speed of 40 mm/s.

• 2. Without speed limitation ($C_{n,f}$=Undefined), the robot completes one cycle at a speed of 40 mm/s

In Step 2, since the $C_{n,f}$ value was undefined, the movement speed of the stage will not be affected by the travel distance or direction of movement.

Fig. 12

The impact of $C_{n,f}$ on the stage’s trave

Figure 12 shows the travel values of the stage in a cycle for Steps 1 and 2. In Fig. 12, the solid line represents the stage’s travel when $C_{n,f}$ is set. The dotted line represents the stage’s travel without setting $C_{n,f}$. The two datasets show a significant difference in the light yellow area. With $C_{n,f}$ set, the travel in the X-direction of the stage within the light yellow area is controlled to be within 6.8 mm. In contrast, when $C_{n,f}$ was undefined, the stage reached its maximum travel in X-direction.

Fig. 13

The impact of $C_{n,f}$ on the system tracking error

Figure 13 shows the tracking errors in the two steps. Figure 13a represents the error in the image when $C_{n,f}$ is set, and Fig. 13b represents the error in the image when $C_{n,f}$ is undefined. Although $C_{n,f}$ limits the movement speed of the stage, the errors in the two steps in Fig. 13 are basically consistent. This indicates that the introduction of $C_{n,f}$ does not lead to greater tracking errors.

Table 5 Impact of $C_{n,f}$ on tracking errors

Comparative experiment 1 elucidates the significance of $C_{n,f}$ in managing the stage’s travel. The data suggests that through the appropriate setting of $C_{n,f}$, one can achieve more precise control over the stage’s travel without augmenting the tracking error, as evidenced in Table 5. This observation is further corroborated by Figs. 12 and 13. The implementation of $C_{n,f}$ ensures the XY-stage does not approach its travel limits. Consequently, the execution module remains poised to counteract errors from any direction, bolstering the overall robustness of the system.

Table 6 Parameters for defining control reference in comparative experiment 2

Fig. 14

Image processing result in comparative experiment 2

4.2 Comparative experiment 2: effect of contour recognition on compensation

In Section 2.2.1, to address the issue of the end effector obscuring the trajectory contour, a scheme was proposed to recognize the areas on both sides of the end effector. The compensation results based on this scheme are shown in Fig. 9. As can be seen from Fig. 9, there are still errors in the compensation process of this system. To investigate whether the recognition scheme caused the errors, comparative experiment 2 was introduced.

Fig. 15

System tracking error when the two ROIs overlap

In comparative experiment 2, the robot’s end effector (pen) was removed. By adjusting the parameters of the defining control reference to the values shown in Table 6, $ROI_{n, 1}$ and $ROI_{n, 2}$ were made to coincide. From Fig. 3, it can be inferred that when $ROI_{n, 1}$ and $ROI_{n, 2}$ overlap, the three points $P_{n,1}$, $P_{n,2}$, and $P_{n,c}$ will satisfy Eq. (10). That is, the values of the Y-direction of $P_{n,1}$ and $P_{n,2}$ will be equal to $P_{n,c}$, and $P_{n,c}$ will be directly located on the contour of the target trajectory. This eliminates the potential impact that the separated ROIs might have on the system’s tracking error when $P_{n,c}$ does not fully lie on the contour. The adjusted ROIs are shown in Fig. 14.

$$\begin{aligned} {P}_{n,\textrm{c}}=P_{n,1}\left[ 0 \quad 1 \right] ^\top =P_{n,2}\left[ 0 \quad 1 \right] ^\top \end{aligned}$$

(10)

After adjusting ROIs, the system undergoes one cycle in the same scheme as Step 2 in comparative experiment 1, while also recording the tracking errors in the process, as shown in Fig. 15. At the same time, the data from Fig. 13a is represented in Fig. 15 with orange and green lines. By comparing the datasets in Fig. 15, it is evident that the errors produced in both experiments are similar, even though the set ROIs are different in the two experiments. This indicates that the tracking errors in this system are caused by the control part of the stage and are not related to the contour recognition scheme. Furthermore, the absolute errors ($\mu _{|x|}$,$\mu _{|y|}$) and average errors ($\mu _{x}$,$\mu _{y}$) in the X- and Y- directions for the two datasets are presented in Table 7. The errors in Table 7 also support the above conclusion.

Table 7 Tracking errors under different ROIs settings

Fig. 16

Optimized teaching trajectory in robot base coordinate system (Z direction value is -30 mm)

4.3 Comparative experiment 3: accuracy of generic compensation method

Nowadays, offline compensation is widely used in the field of robotics, especially in applications where high precision and stability are required [25,26,27]. This method for enhancing the reliability of robots involves using external sensors or devices to compensate for their positional errors through the robot controller. This section introduces a comparative experiment to evaluate the effectiveness of traditional compensation methods. In comparative experiment 3, the robot used an optimized teaching trajectory that is closer to the trajectory contour (see Fig. 16). This trajectory was constructed by manually teaching the robot to match the contour of the trajectory, recording 80 teaching points. The manual teaching process aimed to simulate the robot’s motion state after compensation through the controller.

When the robot operated along the optimized teaching trajectory shown in Fig. 16, the proposed vision module measured the error of the pen tip relative to the trajectory contour. The error of the pen tip relative to the trajectory contour, when the robot followed this optimized trajectory, as shown in Fig. 17. In Fig. 17, the green link indicates the error of the pen tip to the trajectory contour in the X-direction. And yellow link indicates the error in the Y-direction. The maximum error of the pen tip relative to the trajectory contour was 68.9 pixels (4.05 mm) in the X-direction and 75.8 pixels (4.45 mm) in the Y-direction.

Fig. 17

The absolute distance in the image under optimized teaching trajectory at an endpoint speed of 40 mm/s

Despite a 47 % reduction in maximum error compared to the circular teaching trajectory, the error was still significant compared to the approach proposed in this study. This difference is due to errors in the robot’s internal geometric model, which means that compensation relying solely on the robot controller is limited by its repeatability and absolute positioning accuracy. This result highlights the significant advantage in compensation accuracy of the macro/micro structure proposed in this study, compared to traditional methods that rely solely on the robot’s own compensation.

5 Limitation

Based on the mechanical structure of the compensation system utilized in this study, the system is limited to compensation within a two-dimensional plane. To expand the system’s application range, future research would discuss implementation strategies for tracking the contours of targets in three-dimensional space.

Moreover, the compensation range of the system is confined to a square region with a total stroke of 20 mm in the two-dimensional plane, and any offsets beyond this region cannot be compensated. In terms of the visual component, the external parameters of the camera were not calibrated. Although the experimental results indicate that even without considering the angle between the camera imaging plane and the trajectory plane, it is still possible to compensate for the robot’s end-effector offset based on the small size of the recognition area, enhancing the accuracy of the dynamic compensation requires further calibration of the camera.

6 Conclusion

Within the domain of flexible manufacturing, robotic technology was vastly employed in operations necessitating mechanical flexibility, attributed to their high DoF and user-friendly interface. Over the past years, many studies have attached cameras, accelerometers, and other sensors to industrial robots to enable them to respond to uncertainties during the manufacturing process.

This study introduced a system designed to dynamically compensate for the position of a robot’s end effector by adding an extra vision module and execution module to the robot’s hand. To evaluate the system, it was used in an early application scenario of flexible manufacturing, specifically compensating for the contour tracing of an unknown target. This system achieved real-time compensation for the target trajectory, even when the end effector obstructed the trajectory. The maximum error during the entire compensation process was 22.8 pixels (1.34mm), with an average absolute error of 5.4 pixels (0.32mm). Compared to the maximum error of 8.45mm between the target trajectory contour and the teaching trajectory contour, the error has been reduced by 84%.