Perceptually salient haptic rendering for enhancing kinesthetic perception in virtual environments

Abstract

Kinesthetic or dynamic touch involves the use of muscle sensitivity to perceive mechanical properties of objects that are gripped in the hand and wielded in space. Many previous studies with real objects have investigated the mechanical properties that underlie human haptic perception. Few virtual environments, however, have systematically incorporated the relevant mechanical parameters underlying kinesthetic perception. In this study, the ability of a haptic device to render kinesthetic information regarding the inertial properties of virtual objects was tested. Results suggest that users were able to perceive length of rendered virtual objects via the haptic device. Further, users can be trained using the haptic device to increase sensitivity to specific mechanical parameters (like inertia) that are perceptually salient in perceiving properties of rendered objects. The primary implication of this finding is that rendering kinesthetic parameters and employing feedback in a systematic manner may increase the realism of virtual environments and also improve haptic perception.

Appendices

Appendix A: derivation of rod dynamics

The dynamical equations for the motion of a handheld rod were derived by defining two frames of reference; a static inertial ($i$) frame and a body ($b$) frame which moves with the moving rod. The rotation from $i$- to the $b$-frame is defined by the rotation angles $\theta$ and $\varphi$, with the sequence of rotation being rotation about the $y_i$-axis using the $\theta$ angle first, followed by rotation about the $x_b$-axis using the $\varphi$ angle. The rotation matrix, $C_i^b ,$ from the inertial to the body frame is

$$\begin{aligned} C_i^b =\left[ {{\begin{array}{l@{\quad }l@{\quad }l} 1&{} 0&{} 0\\ 0&{} {c_\varphi }&{} {s_\varphi }\\ 0&{} {-s_\varphi }&{} {c_\varphi }\\ \end{array}}} \right] \left[ {{\begin{array}{l@{\quad }l@{\quad }l} {c_\theta }&{} 0&{} {-s_\theta }\\ 0&{} 1&{} 0\\ {s_\theta }&{} 0&{} {c_\theta }\\ \end{array} }} \right] =\left[ {{\begin{array}{l@{\quad }l@{\quad }l} {c_\theta }&{} 0&{} {-s_\theta }\\ {s_\varphi s_\theta }&{} {c_\varphi }&{} {s_\varphi c_\theta }\\ {c_\varphi s_\theta }&{} {-s_\varphi }&{} {c_\varphi c_\theta }\\ \end{array} }} \right] , \end{aligned}$$

where $c( \theta )=\cos ( \theta )$ and $s( \theta )=\sin ( \theta ).$ Using Newton–Euler equations for dynamic equation formation, the total torque applied on the virtual rod is the sum of the gravitational torque and torque applied by the user; $M_{\textit{total}} =M_{\textit{gravity}} + M_{\textit{applied}}$.

On the left hand side of the moments equation, the total torque consists of two sub moments; torque due to angular acceleration and torque due to translation of the bottom of the rod. The angular momentum, $H^b,$ in the body frame is defined as $H^b=\varvec{I}w_{ib}^b$, where I is the diagonalized inertia tensor,

$$\begin{aligned} \varvec{I}=\left[ {{\begin{array}{l@{\quad }l@{\quad }l} {I_{xx} }&{} 0&{} 0\\ 0&{} {I_{yy} }&{} 0\\ 0&{} 0&{} {I_{zz} }\\ \end{array} }} \right] , \end{aligned}$$

with $I_{xx} =I_{yy}$ because the rods are cylindrical. $w_{ib}^b$ is the angular velocity of the body with respect to the inertial frame, expressed in the body frame;

$$\begin{aligned} w_{ib}^b =\left[ {{\begin{array}{c} p\\ q\\ r\\ \end{array} }} \right] . \end{aligned}$$

Since the rod rotates only about the $x_b$-and $y_b$-axis, the $z_b$-component of $w_{ib}^b$ is zero $(r=0)$. The moment due to angular acceleration $M^i_{acc}$ in the inertial frame is obtained by differentiating the angular momentum

$$\begin{aligned} M_{acc}^i&= \frac{d}{dt}H^i=\frac{d}{dt}( {C_b^i H^b})=\left( {\frac{d}{dt}C_b^i }\right) H^b\\&\quad +C_b^i \left( {\frac{d}{dt}H^b}\right) =C_b^i \Omega _{ib}^b H^b+C_b^i \left( {\frac{d}{dt}H^b}\right) , \end{aligned}$$

where $\Omega _{ib}^b$ is the skew symmetric matrix of the vector $w_{ib}^b$. Transforming the total moment with respect to the body frame yields

$$\begin{aligned} M_{acc}^b&= C_i^b M_{acc}^i = \frac{d}{dt}H^b+\Omega _{ib}^b H^b\\ M_{{acc}}^{b}&= \left[ {\begin{array}{c} {I_{{xx}} \dot{p}} \\ {I_{{yy}} \dot{q}} \\ {I_{{zz}} \dot{r}} \\ \end{array} } \right] + \left[ {\begin{array}{c} { - rqI_{{yy}} + qrI_{{zz}} } \\ {prI_{{xx}} - prI_{{zz}} } \\ { - pqI_{{xx}} + pqI_{{yy}} } \\ \end{array} } \right] \\&= \left[ {\begin{array}{c} {I_{{xx}} \dot{p} - qr(I_{{yy}} - I_{{zz}} )} \\ {I_{{yy}} \dot{q} - pr(I_{{zz}} - I_{{xx}} )} \\ {I_{{zz}} \dot{r} - pq(I_{{xx}} - I_{{yy}} )} \\ \end{array} } \right] . \end{aligned}$$

Since $r=0, \dot{r} = 0$ and $I_{xx} =I_{yy}$, moment due to angular acceleration with respect to the body frame is given by

$$\begin{aligned} M_{{acc}}^{b} = \left[ {\begin{array}{c} {I_{{xx}} \dot{p}} \\ {I_{{yy}} \dot{q}} \\ 0 \\ \end{array} } \right] . \end{aligned}$$

Moment due to translation of the bottom of the rod causes the moments $M_{tr}^b$

$$\begin{aligned}&M_{tr}^b =-r_{AG}^b \times F_A^b =r_{GA}^b \times F_A^b\\&F_A^b =m\left( {C_i^b \left[ {{\begin{array}{c} {\ddot{x}^i}\\ {\ddot{y}^i}\\ {\ddot{z}^i}\\ \end{array} }} \right] +\dot{\omega }_{ib}^b \times r_{AG}^b +\omega _{ib}^b \times ( {\omega _{ib}^b \times r_{AG}^b })}\right) \\&M_{tr}^b =m\left[ {{\begin{array}{l@{\quad }l@{\quad }l} 0&{} {-\frac{l}{2}}&{} 0\\ {\frac{l}{2}}&{} 0&{} 0\\ 0&{} 0&{} 0\\ \end{array} }} \right] \left( \left[ {{\begin{array}{l@{\quad }l@{\quad }l} {c_\theta }&{} 0&{} {-s_\theta }\\ {s_\varphi s_\theta }&{} {c_\varphi }&{} {s_\varphi c_\theta }\\ {c_\varphi s_\theta }&{} {-s_\varphi }&{} {c_\varphi c_\theta }\\ \end{array} }} \right] \left[ {{\begin{array}{c} {\ddot{x}^i}\\ {\ddot{y}^i}\\ {\ddot{z}^i}\\ \end{array} }} \right] \right. \nonumber \\&\qquad \left. +\left[ {{\begin{array}{l@{\quad }l@{\quad }l} 0&{} {-\dot{r}}&{} {\dot{q}}\\ {\dot{r}}&{} 0&{} {-\dot{p}}\\ {-\dot{q}}&{} {\dot{p}}&{} 0\\ \end{array} }} \right] \left[ {{\begin{array}{c} 0\\ 0\\ {-\frac{l}{2}}\\ \end{array} }} \right] \right. \nonumber \\&\qquad \left. +\left[ {{\begin{array}{l@{\quad }l@{\quad }l} 0&{} {-r}&{} q\\ r&{} 0&{} {-p}\\ {-q}&{} p&{} 0\\ \end{array} }} \right] ^2\left[ {{\begin{array}{c} 0\\ 0\\ {-\frac{l}{2}}\\ \end{array} }} \right] \right) \\&\quad =\frac{ml}{2}\left[ {{\begin{array}{c} {-s_\varphi s_\theta \ddot{x}^i-c_\varphi \ddot{y}^i-s_\varphi c_\theta \ddot{z}^i}\\ {c_\theta \ddot{x}^i-s_\theta \ddot{z}^i}\\ 0\\ \end{array} }} \right] +\frac{ml^2}{4}\left[ {{\begin{array}{c} {-\dot{p}}\\ {-\dot{q}}\\ 0\\ \end{array} }} \right] . \end{aligned}$$

The next moment to be considered is torque due to gravity. Assuming that the gravity is transmitted to the lower end of the rod along the $z_b$-axis in the body frame, the $z_b$-component of the gravity term causes a force $F_g^b$ given by

$$\begin{aligned} F_g^b =C_i^b \left[ {{\begin{array}{c} 0\\ 0\\ {mg}\\ \end{array} }} \right] =\left[ {{\begin{array}{c} {-s_\theta }\\ {s_\varphi c_\theta }\\ {c_\varphi c_\theta }\\ \end{array} }} \right] mg, \end{aligned}$$

where $m$ is mass of the rod. The gravity term causes the moment, $M_g^b$, defined by $M_g^b =-r_{GA}^b \times F_g^b$. Using $r_{GA}^b =[0\,\, 0\frac{l}{2}]^T$ (where $l$ is the length of the rod) and $F_g^b$,

$$\begin{aligned} M_g^b =-\left[ {{\begin{array}{l@{\quad }l@{\quad }l} 0&{} {-\frac{l}{2}}&{} 0\\ {\frac{l}{2}}&{} 0&{} 0\\ 0&{} 0&{} 0\\ \end{array} }} \right] \left[ {{\begin{array}{c} {-s_\theta }\\ {s_\varphi c_\theta }\\ {c_\varphi c_\theta }\\ \end{array} }} \right] mg=\left[ {{\begin{array}{c} {s_\varphi c_\theta }\\ {s_\theta }\\ 0\\ \end{array} }} \right] \frac{lmg}{2}. \end{aligned}$$

The external applied moment of the hand is defined as $M_T^b$. Using Newton–Euler balance equations, $M_{acc} = M_{gravity} + M_{tr} + M_{applied}$, the equilibrium of the body about the $x_b$- and $y_b$-axis results in the following equations

$$\begin{aligned}&\left( {I_{{xx}} + m\frac{{l^{2} }}{4}} \right) \dot{p} = s_{\varphi } c_{\theta } \frac{{lmg}}{2} \\&\quad + ( { - s_{\varphi } s_{\theta } \ddot{x}^{i} - c_{\varphi } \ddot{y}^{i} - s_{\varphi } c_{\theta } \ddot{z}^{i} })\frac{{ml}}{2} + M_{{T_{x} }}^{b} \end{aligned}$$

$$\begin{aligned}&\left( {I_{{yy}} + m\frac{{l^{2} }}{4}} \right) \dot{q} = s_{\theta } \frac{{lmg}}{2} \\&\quad +\, (c_{\theta } \ddot{x}^{i} - s_{\theta } \ddot{z}^{i} )\frac{{ml}}{2} + M_{{T_{y} }}^{b}. \end{aligned}$$

Since the angular rates of the rod can be expressed as the time derivatives of Euler angles using

$$\begin{aligned} \left[ {\begin{array}{c} p \\ q \\ r \\ \end{array} } \right] = \left[ {\begin{array}{l@{\quad }l@{\quad }l} 1 &{} 0 &{} { - s_{\theta } } \\ 0 &{} {c_{\varphi } } &{} {s_{\varphi } c_{\theta } } \\ 0 &{} { - s_{\varphi } } &{} {c_{\varphi } c_{\theta } } \\ \end{array} } \right] \left[ {\begin{array}{c} {\dot{\varphi }} \\ {\dot{\theta }} \\ {\dot{\Psi }} \\ \end{array} } \right] , \end{aligned}$$

torque balance equations about the $x$ and $y$ axis are

$$\begin{aligned}&\left( l_{xx}+\frac{{ml^2}}{4}\right) \left( \ddot{\varphi }-\frac{{s_\varphi }}{c^2_\theta c_\varphi }{\theta ^{^{\!\!\!\cdot }}}^{2}\right) \approx s_{\varphi } c_{\theta } \frac{{lmg}}{2}\\&\quad + ( { - s_{\varphi } s_{\theta } \ddot{x}^{i} - c_{\varphi } \ddot{y}^{i} - s_{\varphi } c_{\theta } \ddot{z}^{i} })\frac{{ml}}{2} + M_{{T_{x} }}^{b}\\&\left( l_{yy}+\frac{{ml^2}}{4}\right) \left( \frac{1}{c_\varphi }{\ddot{\theta }}+\frac{s_\varphi }{c^{2}_\varphi }\dot{\varphi }\dot{\theta }\right) \approx s_{\theta } \frac{{lmg}}{2}\\&+ (c_{\theta } \ddot{x}^{i} - s_{\theta } \ddot{z}^{i} )\frac{{ml}}{2} + M_{{T_{y} }}^{b}. \end{aligned}$$

The vector $[-M_{T_x }^b -M_{T_y }^b 0]$ defines the output response torque and is applied to the 5 DOF haptic device.

Appendix B: table of terms related to the dynamic-touch approach to haptic perception

Dynamic touch	This term refers to perceiving properties of rigid objects by holding or wielding them
Kinesthetic	Typically, this form of haptic perception refers to motion-based touch interaction, in contrast with tactile (skin-based) feedback
Parameter	In this work, a parameter may mean a lower order (e.g., mass) or higher order (e.g., inertia/first moment) mechanical quantity
Inertia	Inertia is defined as the resistance of the object to angular acceleration. The inertia tensor, I, describes the spatial distribution of the object’s mass and its resistance to rotational accelerations in three dimensions. For a rigid object rotating about a fixed point of rotation, I is a constant and as a time-independent quantity, I is an “invariant” mechanical quantity describing the mass distribution of the rotated object. The eigenvalues of I (or principal moments of inertia, $I_{1}, I_{2}$, and $I_{3}$, where $I_{1} \ge I_{2} \ge I_{3})$ describe the resistances to rotations about the respective directions of the eigenvectors (or principal axes of inertia, e$_{1}$, e$_{2}$, and e$_{3}$, where e$_{1}$ is the axis of maximum resistance and e$_{3}$ is the axis of minimal resistance) [8]
Invariant	The mechanical quantity of an object that is not time varying or stimulus-dependent
Specifying variables	Those mechanical variables that are directly related to a particular object property
Attunement	The process of “honing in” on the specifying variable for a particular object property
Calibration	The process of scaling the specifying variable accurately based on visual or haptic feedback
Perceptual learning	In this work, we hypothesize that the dual processes of attunement and calibration are used in improving perception through feedback
Fidelity of rendering	The degree or quality of realism that the device or environment is capable of rendering