Immersive Commodity Telepresence with the AVATRINA Robot Avatar

Abstract

Immersive robotic avatars have the potential to aid and replace humans in a variety of applications such as telemedicine and search-and-rescue operations, reducing the need for travel and the risk to people working in dangerous environments. Many challenges, such as kinematic differences between people and robots, reduced perceptual feedback, and communication latency, currently limit how well robot avatars can achieve full immersion. This paper presents AVATRINA, a teleoperated robot designed to address some of these concerns and maximize the operator’s capabilities while using a commodity light-weight human–machine interface. Team AVATRINA took 4th place at the recent $10 million ANA Avatar XPRIZE competition, which required contestants to design avatar systems that could be controlled by novice operators to complete various manipulation, navigation, and social interaction tasks. This paper details the components of AVATRINA and the design process that contributed to our success at the competition. We highlight a novel study on one of these components, namely the effects of baseline-interpupillary distance matching and head mobility for immersive stereo vision and hand-eye coordination.

Appendices

A Elbow Heuristic Details

Fig. 27

Concept of the bias configuration heuristic. Based on the commanded motion of the end effector, the elbow pose is controlled to appear more human-like and reduce the rate of tracking failures due to joint limits and self-collision. (Color figure online)

Fig. 28

A sample of the performance of the facial reconstruction pipeline on one of the author’s under varied facial expressions, with the original image on the left paired with the network’s output on the right

Let the 3D rotation and translation vectors $r = \begin{bmatrix}r_x&r_y&r_z\end{bmatrix}^T$ and $t = \begin{bmatrix}t_x&t_y&t_z\end{bmatrix}^T$ represent the commanded SE(3) transform $T_{ee}$ of a hand. The shoulder angle heuristic for the left and right arms are given by:

$$\begin{aligned} \begin{aligned} q_{\text {shoulder, right}}&= -(\max (k_{\text {angle}} (r_y + r_z), 0) \\&\qquad + k_z(t_z - z_0))\\ q_{\text {shoulder, left}}&= +(\max (k_{\text {angle}} (r_y - r_z), 0) \\&\qquad + k_z(t_z - z_0))\\ \end{aligned} \end{aligned}$$

(15)

where $k_{\text {angle}}, k_z$, and $z_0$ are positive tunable constants. Figure 27 shows the motion of the elbow when the operator turns their wrist inwards: positive z axis rotation corresponds to a negative $q_{\text {shoulder}}$ for the right arm, which forces the elbow outwards. Turning the wrist downwards and lifting the hand upwards also force the elbow to turn outwards.

B Qualitative Evaluation of Facial Reconstruction Pipeline

Figure 28 illustrates the performance of the proposed facial reconstruction pipeline. Examples b) through f) show that under relatively tame facial expressions common during communication, the network’s output is mostly plausible and can convey some of the expressions of the operator to their remote counterparts, despite having trouble with conveying precise mouth movements or proper rendering of the operator’s teeth (unseen in the source image). Examples f) through p) show some of the failure cases of this pipeline: Facial expressions far outside of its training distribution, such as i) and j), as well as any expressions that involve parts of the face not captured by facial landmarks (such as the tongue or cheeks), like m) through p) result in outputs that don’t necessarily capture the operator’s intent. Further, gaze direction is entirely dependent on the neutrally blinking video recording, which can sometimes result in awkward staring or stargazing, such as in example f).