SHARIDEAS: a smart collaborative assembly platform based on augmented reality supporting assembly intention recognition

Abstract

With the development of augmented reality supporting manual assembly collaboration, it is particularly important to explore the transformation from traditional “human-machine” cooperation mode to smart “human-machine” cooperation mode. Early studies have shown that user cues (i.e., head, gesture, eye) and scene cues (i.e., objects, tools, space) are intuitive and highly expressive for traditional AR collaborative mode. However, how to integrate these cues into an assembly system, reasonably infer an operator’s work intention, and then give an appropriate rendering scheme is one of the problems in collaborative assembly. This paper describes a AR collaborative assembly platform: SHARIDEAS. It uses a generalized grey correlation method to integrate user cues and scene cues. The results of data fusion can provide appropriate and intuitive assembly guidance for local workers. A formal user study is to explore the usability and feasibility of SHAREDEAS in a manual assembly task. The experimental data show that SHAREDEAS is more conducive than traditional one to improve the efficiency of human-machine cooperation. Finally, some conclusions of SHARIDEAS are given and the future research direction has prospected.

Appendix

Definition of Generalized Grey Relation

If the behavior factor of SHARIDEAS is x₀ and x₀ is affected by multiple factors x_i (i=1, 2,…, n), the method of expressing x_i's influence on x₀ by the grey correlation degree of factor x_i and factor x₀ is called generalized grey correlation. Based on this concept, we regard the above visual cues as a grey system, and the evaluation of the fusion process is determined by various indicators.

Taking the fusion process of visual cues as a research object, the system characteristic behavior sequence of operator Intention Evaluation Index x₀ is as follows:

$$ {X}_0=\left\{{x}_0(1),{x}_0(2),\dots, {x}_0(k),\dots, {x}_0(n)\right\} $$

(1)

In Formula 1, x₀(k) is the kth evaluation data of a behavior factor in the process of intention perception.

According to the cognitive requirements of various visual cues, the quality of intention cognition depends on the fusion of various behavioral factors. On this basis, the grey relational mathematical model is constructed.

Assuming that there are m factors influencing intention calculation, the effective date of relevant visual cues are extracted by sensors (e.g., industrial camera), and the behavioral index sequence of the ith influencing factor x_i is as follows:

$$ {X}_i=\left\{{x}_i(1),{x}_i(2),\dots, {x}_i(k),\dots, {x}_i(n)\right\} $$

(2)

In formula 2, i is the number of the ith influencing factor(i = 1, 2,…, m); k is the serial number of the data (k = 1, 2,…, n); x₁(k)is the kth data collected by influencing factor x_i.

From Formula 1 and 2, the corresponding forms of system characteristic behavior sequencex₀ and comparative data sequence x_i are deduced, then:

$$ {\displaystyle \begin{array}{l}{X}_0=\left\{{x}_0(1),{x}_0(2),\dots, {x}_0(k),\dots, {x}_0(n)\right\}\\ {}\kern3em \vdots \\ {}{X}_i=\left\{{x}_i(1),{x}_i(2),\dots, {x}_i(k),\dots, {x}_i(n)\right\}\\ {}\kern3em \vdots \\ {}{X}_m=\left\{{x}_m(1),{x}_m(2),\dots, {x}_m(k),\dots, {x}_m(n)\right\}\end{array}} $$

(3)

Fusion parameter calculation of visual cues

In collaborative assembly, visual cues can be divided into user cues (UC) and environment cues (EC). User cues include gesture cues, head gaze cues, and eye gaze cues, while environment cues mainly include spatial orientation cues, tool cues, and assembly objects.

The spatial relationship between the components of the second stage reducer is complex. According to the requirements of its working conditions and technical parameters, such as the assembly of the upper and lower box, the center distance of the box, the dividing circle of the gear shaft, the airtightness requirements of the internal environment, the vibration resistance, and impact resistance, etc., 10 evaluation coefficients of visual cues are determined.

Due to the different numerical units of the two types of visual cues, SHARIDEAS integrates the collected visual cues into a multi-dimensional evaluation. Its evaluation results are divided into 10 grades: when the evaluation coefficient is 8–10, it is an excellent factor; when the evaluation coefficient is 6–7, it is a good factor; when the evaluation coefficient is 4–5, it is determined as a general factor; when the evaluation coefficient is 1–3, it is a general factor, that is, a secondary factor. Therefore, according to the current intention of user, the 10 indexes will be sorted adaptively. Thus, the system characteristic behavior sequence X₀ composed of 10 evaluation coefficients is obtained. In the process of collaborative assembly, SHARIDEAS calculates the corresponding comparative data sequence X_i according to the real-time data.

Then, the interval value is processed according to following formulas.

$$ {X}_iB=\left\{{x}_i(1)b,{x}_i(2)b,\dots, {x}_i(k)b,\dots, {x}_i(n)b\right\} $$

(4)

$$ {x}_i(k)b=\frac{x_i(k)-\min {x}_i(k)}{\max {x}_i(k)-\min {x}_i(k)},\kern0.5em k=1,2,\dots, n $$

(5)

On this basis, the data series ${X}_0^{\prime }$ and ${X}_i^{\prime }$ after interval value processing are obtained. The data are further processed according to the following formula:

$$ {\Delta}_i(k)=\left|{x}_0^{\prime }(k)-{x}_i^{\prime }(k)\right| $$

(6)

$$ {\Delta}_i=\left|{\Delta}_i(1),{\Delta}_i(2),\dots, {\Delta}_i(n)\right|i=0,1,\dots, m $$

(7)

For each group of sequences, the maximum and minimum values are obtained.

$$ M=\underset{i}{\mathit{\max}}\underset{k}{\max }{\Delta}_i(k) $$

(8)

$$ m=\underset{i}{\mathit{\min}}\underset{k}{\mathit{\min}}{\Delta}_i(k) $$

(9)

If the resolution coefficient ζ∈ (0,1) is defined, then,

$$ \gamma \left({x}_0(k),{x}_i(k)\right)=\frac{\underset{i}{\min}\underset{k}{\min}\left|{x}_0(k)-{x}_i(k)\right|+\zeta \underset{i}{\max}\underset{k}{\max}\left|{x}_0(k)-{x}_i(k)\right|}{\left|{x}_0(k)-{x}_i(k)\right|+\zeta \underset{i}{\max}\underset{k}{\max}\left|{x}_0(k)-{x}_i(k)\right|} $$

(10)

$$ \gamma \left({X}_0,{X}_i\right)=\frac{1}{n}\sum \limits_{k=1}^n\gamma \left({x}_0(k),{x}_i(k)\right) $$

(11)

According to formula (8) ~ Eq. (11), SHAREIDEAS performs the following operations:

$$ {\gamma}_{0i}(k)=\frac{m+\zeta M}{\Delta_i(k)-\zeta M} $$

(12)

$$ {\beta}_{0i}(k)=\left\{{\gamma}_{0i}(1),{\gamma}_{0i}(2),\dots, {\gamma}_{0i}(k),\dots, {\gamma}_{0i}(n)\right\} $$

(13)

$$ {\gamma}_{0i}=\frac{1}{n}\sum \limits_{k=1}^n{\gamma}_{0i}(k) $$

(14)

γ_0i’s data itself represents the intention tendency of the clue. For example, in the input cues, the evaluation parameters of the “twist” gesture range from 0.67 to 0.82. When γ_0i’s data is 0.75, this cue dominates the user’s intention.