1 Introduction

During the last two decades, different efforts have been made to facilitate remote interaction between a human operator and a robot. In: so-called teleoperated robotic systems. These systems basically consist of a slave system (robot), which interacts with a given environment by performing a specific task, and a master system, remotely managed by a human operator. The use of these systems can help a person to carry out and complete complex or dangerous tasks in environments that require great precision and accuracy.

The current health crisis conditions caused by the COVID-19 pandemic provide a scenario where robotics can solve new problems. There were already numerous examples of the use of robots to help detect and care for infected populations and prevent further disease transmission [1]. Robots are also enabling health care professionals to work remotely, avoiding exposure to the virus and eliminating the need to use expensive personal protective equipment. In the industry, collaborative robots are allowing companies to continue working in safe and isolated environments, thus reducing the impact on productivity [2].

However, for these remote-controlled robotic systems to be effective, the human operator must feel as if they were physically present in that environment. This condition is commonly known as telepresence. To make the conditions on the human operator's side as close as possible to the conditions at the remote location, certain requirements must be met. On the operator's side, there must be a manipulator or joystick that allows movements to be performed in a similar way to those to be performed by the device on the remote side. There must also be an interface that provides visual information and feedback about what is happening on the remote side.

At the remote location, the slave system must incorporate three main elements: (a) different sensors that sense the environment and the objects in it, (b) actuators that interact with these objects, and (c) a communications network that transmits data bi-directionally. In most teleoperation systems, the information from the remote system is mainly audiovisual, effected through the use of cameras located in the remote working environment. However, audiovisual information alone may not be enough and different studies [3] have concluded that the use of tactile feedback improves interaction with remotely operated robotic platforms. Especially so. In: industrial applications, such as tasks involving contact, like surface finishing operations [4] in which the architecture of the teleoperated system must be adapted.

Besides, it is important to improve the interaction between the human operator and the robot from the user's point of view, such that these teleoperated robotic systems will be easy to use by occasional and non-expert users. Another benefit of easy-to-use systems is that they also reduce the learning and training curve.

However, research on the assessment of the usability of teleoperated robotic systems is still scant. This study addresses this lack by evaluating the utility of haptic feedback in a teleoperated system through an empirical investigation, as a preliminary step to the development and validation of a usability evaluation framework.

The contributions of this study are: (1) the approach of an experimental model to evaluate the usability of a teleoperated robotic system; (2) a quantitative evaluation of the system via carrying out empirical tests of a teleoperated system composed of a collaborative robot (cobot) and a haptic device; and (3) a qualitative evaluation of the system based on questionnaires completed by non-expert.

The outline of this article is as follows: Sect. 2 summarizes related work; Sect. 3 describes the system used in the study, the graphical user interface and the setting of the experiment; Sect. 4 presents the experimental procedures, the details of the participants in the experiment, the task performed, and the variables and parameters used in the study; Sect. 5 details the main results of the empirical analysis and provides answers to the research questions. The implications of the results are discussed in Sect. 6; finally, Sect. 7 summarizes the main conclusions drawn from this study.

2 Related work

A teleoperation system is composed of a master device, a slave device and a communication channel. This allows a human operator to mechanically manipulate objects locally, duplicating conditions similar to those at the remote location [5]. This is why these systems are used to perform tasks in dangerous locations [6, 7], rescue operations [8, 9], control of unmanned aerial vehicles [10, 11], control of space exploration vehicles [12, 13], telemedicine [14, 15], and telesurgery [16,17,18], among other applications.

When force sensors are added on the slave side, they can transmit the reaction forces generated when performing the task on the master device and the teleoperation system is said to be controlled bilaterally [5]. This is known as haptic feedback, which the human operator perceives through touch and kinetic stimuli.

Nevertheless, the use of this feedback negatively influences the stability of the system [19, 20]. This instability remains a major challenge for researchers.

Several researchers have proposed a wide variety of solutions to guarantee the transparency and stability of teleoperated systems, designing different bilateral controllers. For instance, some studies propose using passivity as the main instrument to provide a stable teleoperation [21].

A different approach to achieving stability in a teleoperated system is to replace haptic feedback on the master side with other sensory stimuli, such as vibrotactile [22], auditory and/or visual feedback [23]. Massimino called this technique sensory substitution [24]. An example of the application of this technique is the system created by McMahan [25] for the da Vinci Surgical System by Intuitive Surgical. In:c. This surgical robot allows the surgeon to feel the vibrations of the instrument used in the slave device in real time without destabilizing the teleoperation loop.

By using this sensory substitution technique, and since the human operator does not receive any force feedback, teleoperation systems are inherently stable. However, these systems show an inferior performance to that achieved with haptic feedback [26].

The use of haptic feedback in a teleoperated system also has the problem of transparency. When the force feedback received by the human operator is attenuated, the teleoperating system does not have stability problems, but it then suffers from a lack of transparency. Ideally, a balance should be achieved between the stability and the transparency of a teleoperation system.

Several researchers have tried to achieve this goal. Lawrence [20] defined transparency as the relationship between impedances which are transmitted and those found in the environment. His design objective was to keep this relationship ratio close to 1 in order to make the human operator feel that he or she is interacting directly with the remote objects in the most real way possible. Chen and colleagues [27] developed a novel waveform control design for the bilateral teleoperation system, where the human operator can choose two separate parameters for different transparency requirements. With this design, stability is guaranteed, and the transparency performances of both position tracking and force feedback are improved.

Monfaredi and colleagues proposed another way to increase the transparency of bilateral teleoperation [28]. The solution proposed is based on the idea of using a passivity controller to monitor the energies flowing to and from the teleoperator.

More recently, Srikar [29] proposed a robot design based on the isotropy to achieve a stable and transparent behavior of the teleoperating system. Another significant contribution of this research is that it eliminates the need for two signals from the traditional four-channel teleoperating architecture, thus reducing the complexity of the system.

Notwithstanding the aforementioned issues, prior research shows that haptic feedback plays an important role in improving the performance of teleoperation systems regarding task completion time [30, 31], accuracy [32, 33] and maximum and average force exerted [34, 35]. In addition, the use of haptic feedback as a complement to visual and auditory information increases the human operator’s sense of being present in the remote environment [36]. For all these reasons. In:cluding haptic feedback in a teleoperating system is desirable in many of today’s telerobotic applications.

There is another important aspect to consider when using a teleoperation system. This parameter is usability. According to the ISO 9241-11 standard [37], usability is defined as the degree to which a product can be used by certain users to achieve their specific objectives effectively, efficiently and satisfactorily in a given context of use. Seffah and colleagues [38] proposed a consolidated usability model to provide a comprehensive approach to usability assessment. They define ten factors of usability: efficiency, effectiveness, productivity, satisfaction, learning ability, safety, confidence, accessibility, universality and utility.

However, not all these elements may necessarily be important for all systems. Thus. In: the context of human–robot interaction, it is important to define usability attributes that are related to the user experience in interacting with the system, the time required to program a task for the robot and the ability to create work configurations for the system [39]. In the case of haptic systems, the five factors considered the most important are: efficiency, effectiveness, satisfaction, learning ability, and safety [40]. Khan. In: his study on usability in haptic systems [41], performed a literature review to ascertain the guidelines related to the different assessment methods and haptic devices used across various domains. The results of the research suggest the need to create new methods of assessing the usability of these devices to be able to use them in any domain [42].

The integration of haptic feedback in teleoperated systems, besides the problems of stability and transparency, makes the evaluation of usability in this type of systems more difficult because the function of the feedback is contingent on the type of application or task to be performed. It is also different depending on the type of haptic device used, the mode of kinematics (direct or inverse) and the preferences of the human operator [43].

There are certain studies that make questionnaire-based assessment of the usability of their own teleoperated systems, such as, NASA’s TLX for workload measurement [44]. In the research [45], the authors show that haptic feedback improves the performance of operator teleoperation by reducing workload.

In [46], the authors evaluated the effect of haptic feedback on the operation of a teleoperated unmanned ground vehicle, based on physical stability, task performance, and operator control effort.

In [47], the researchers conducted a study in which 20 participants used a teleoperation system with haptic feedback to perform two teleoperated surgical tasks; their results suggest that haptic feedback makes users more aware of what they are doing in their environment; respondents also stated that this feedback makes the interaction feel more natural and improves task performance.

Tonel Lima and colleagues [48] presented the development of a haptic teleoperation system for an industrial robotic manipulator; in their study, users performed a simulation test and then, answered a questionnaire about the usefulness of the system and their acceptance of force feedback; the results reflect high satisfaction levels among participants, who declared that the use of haptic feedback increases the perception of the remote environment and the objects manipulated in it.

Given the need for a methodology to evaluate the usability of any teleoperated robotic system with haptic feedback in different domains. In: this study we use a usability evaluation method validated through experimental research. This will allow us. In: the future, to establish a common framework and validate a methodology to evaluate and improve the usability of teleoperated robotic systems among non-expert users. The following section describes in more detail the setting of the research study.

3 Materials and methods

The study uses empirical (experimental) research to compare the usability of a teleoperated robotic system with and without haptic feedback. The first step in performing an empirical investigation is to formulate a working hypothesis [49]. Based on the results of related research presented in the previous section, this study proposes the following research hypothesis: for non-expert users, haptic feedback improves usability in robotic teleoperated tasks.

To test this hypothesis, we designed an experiment that consisted in performing a simple task using a haptic device as a master device. The device allowed including force feedback in the control of the arm of a robot acting as a slave device.

The next step was to pose the following research questions:

  • How does the integration of haptic feedback affect the performance of a teleoperated robotic system?

  • How does this integration affect the human operator’s perception of usability?

  • Do variations in gender, age and level of education significantly affect task performance and workload?

The next phase of an empirical investigation is the phase of deduction. In this phase, we designed an experiment to test the hypothesis. To design our experiment, we relied on Ju’s study [50]. In: which the teleoperation system is a SensAble®Omni haptic device, configured as a master device providing haptic feedback, and the 7-DOF (degrees of freedom) robotic arm of the Baxter® robot as a slave device.

The preparation of the experiment was carried out in several stages. In the first, the robot was programmed to use its right arm in the manipulation task. The haptic device was also programmed to perform the control from the master side. Then, the two devices, which communicate through a local Ethernet network, were synchronized.

Next, an experiment was prepared in which the participants performed a simple task that would serve to validate the working hypothesis of our study. The working environment was designed in such a way that the user could not see the robot or its real environment, since the task to be executed is performed remotely.

In the following stage, the task to be performed by the study participants was designed, as well as the questionnaires used to quantitatively collect their opinions. The parameters that would later be used to perform a qualitative analysis of performance were also chosen.

Finally, the experiment was conducted by the participants selected for the study and the results obtained were analyzed using different methods of qualitative and quantitative analysis.

Figure 1 visually represents the different stages of the experiment.

Fig. 1
figure 1

Stages of the empirical investigation

Full size image

The following is as a more detailed description of the devices used, the graphical user interface and the layout of all the parts of the environment where the experiment was carried out.

3.1 System description

As pointed out in the previous section, a teleoperated system must have a device that acts as a master and a slave device. In this study, we need the master device to be able to send the human operator's movements and commands to the slave device. However, at the same time, this device has to provide the operator with some kinesthetic information, i.e., it has to reflect the forces or vibrations detected on the remote side. In the context of the experiment, haptic devices function perfectly as a master device. In this particular case, we used a Geomagic Touch™ haptic device, called Phantom Omni.

The Geomagic Touch™ is a six DOF (degrees of freedom) device, which has three drives associated with the armature that provides the translation of movements (Cartesian coordinates X, Y and Z) and three other non-activated DOFs associated with the gimbal that monitors the orientation (tilt, roll and yaw).

We connected the device to a workstation with Ubuntu 14.04.4 LTS (Trusty Tahr) with ROS Indigo. To communicate between the haptics and ROS, we used the phantom-omni package developed by Suarez-Ruiz [51]. However, the authors developed this package for ROS Hydro, so we adapted it to our distribution (Indigo) and developed a new program in the Python programming language to control the master device.

As a slave device, we used a Baxter robot. This is an industrial robot produced by Rethink Robotics to enable collaboration and improved interaction between humans and robots. It consists of two arms of 7DOF, which provide a kinematic redundancy to improve the manipulation of objects. Each of them has seven rotating joints (R) of a single DOF, which allow reaching any position easily. The robot has a series of elastic actuators in each joint. In:corporating a complete position and force sensor in each of them. There is also a camera on each hand together with an infrared range sensor (4–40 cm), a camera on the top of the robot's display that is positioned according to the location of the head, and range sensors integrated into the top of the robot. However, for our experiment we only used one of the robot’s arms (right arm). We also decided to add two more cameras in the remote environment: one on the torso and another near the worktable (see Fig. 2). This way the user has a better view of the environment from various points of view.

Fig. 2
figure 2

Position of the cameras on the slave side

Full size image

The robot is connected to the same workstation as the master device and to communicate with it, it uses the ROS package baxter pykdl from [52] which supports Indigo.

3.2 Device programming

The first step in using the Geomagic Touch™ haptic device as a master device is to calculate the forward kinematics model and the Denavit–Hartenberg (DH) parameters [53].

The forward kinematics consist of determining the position and orientation of the end effector of our robot with respect to a system of coordinates that is taken as reference. To do this, it is necessary to describe the relationship between the joint angles of the serial manipulator and the position and orientation of its end effector. To describe this relationship, we used a standard notation called Denavit–Hartenberg. In this convention, we used four parameters: link length (ai), link twist (αi), link offset (di), and joint angle (θi).

The kinematic model of master device and the DH parameters are described in Fig. 3.

Fig. 3
figure 3

Kinematic model and DH parameters of master device

Full size image

We also applied a method to calculate the forward kinematics model and the DH parameters of the Baxter robot [54] in this work, as shown in Fig. 4.

Fig. 4
figure 4

Kinematic model and DH parameters of slave device

Full size image

It is also necessary to correctly translate the physical motion of the haptic device to the Baxter robot. For this, we used the following transformation matrix:

$$^{i - 1} A_{i} = \left[ {\begin{array}{*{20}c} {\cos \theta_{i} } & { - \sin \theta_{i} \cos \alpha_{i} } & {\sin \theta_{i} \sin \alpha i_{{}} } & {\alpha_{i} \cos \theta_{i} } \\ {\sin \theta_{{}} } & {\cos \theta_{i} \cos \alpha_{i} } & { - \cos \theta_{i} \sin \alpha_{i} } & {\alpha_{i} \sin \theta_{i} } \\ 0 & {\sin \alpha_{i} } & {\cos \alpha_{i} } & {d_{i} } \\ 0 & 0 & 0 & 1 \\ \end{array} } \right]$$
(1)

Using Eq. (1) and the DH parameter data calculated for each device, we calculated the homogeneous transformation matrices through matrix multiplication as follows:

$${}_{{}}^{0} A_{n} = {}_{{}}^{0} A_{1} \cdot{}_{{}}^{1} A_{2} \cdot\cdot\cdot{}_{{}}^{n - 1} A_{n}$$
(2)

When it came to synchronizing the movements of both devices, we encountered several difficulties. The first problem was that the workspaces are different. The working space of the haptic device is 160 mm by 120 mm by 70 mm. In the case of the right arm of the robot, the working space is 1305 mm by 1000 mm by 1430 mm.

It is fundamental to know if a certain place is reachable by the slave device. To solve this problem, it is necessary to map the motion paths of the master device in a workspace accessible to the slave device manipulator. Several mapping methods have been developed [55,56,57], but for our work, we adopted the method developed by Ju [50]. This method is quite easy to apply and is based on the Monte Carlo numerical random sampling method to generate the limits of the workspace by simply using forward kinematics [58].

In this study, we described the mapping process in the following equation:

$$\left[ {\begin{array}{*{20}c} {x_{s} } \\ {y_{s} } \\ {z_{s} } \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {\cos \delta } & { - \sin \delta } & 0 \\ {\sin \delta } & {\cos \delta } & 0 \\ 0 & 0 & 1 \\ \end{array} } \right] \times \left( {\left[ {\begin{array}{*{20}c} {S_{x} } & 0 & 0 \\ 0 & {S_{y} } & 0 \\ 0 & 0 & {S_{z} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {x_{m} } \\ {y_{m} } \\ {z_{m} } \\ \end{array} } \right] + \left[ {\begin{array}{*{20}c} {T_{x} } \\ {T_{y} } \\ {T_{z} } \\ \end{array} } \right]} \right)$$
(3)

where [xs ys zs]T and [xm ym zm]T are the Cartesian coordinates of the end effectors of Baxter and Geomagic Touch, respectively, δ is the revolution angle about the Z-axis of the Baxter base frame and [Sx Sy Sz]T and [Tx Ty Tz]T are the scaling factors and translations about the X, Y and Z axis.

In addition, the coordinate system is different, so we had to transform it to coordinate the movements of master–slave. In Fig. 5, these differences can be seen.

Fig. 5
figure 5

Coordinate system of master device (Geomagic Touch) and slave device (Baxter robot)

Full size image

Therefore, we need to modify the Cartesian coordinates of the master device according to the equation:

$$A_{0}{\prime} = R_{z} \left( {\frac{\pi }{2}} \right)R_{x} \left( {\frac{\pi }{2}} \right)A_{0} R_{y} \left( {\frac{\pi }{2}} \right)R_{z} \left( {\frac{\pi }{2}} \right)\left[ {\begin{array}{*{20}c} 1 & 0 & 0 \\ 0 & { - 1} & 0 \\ 0 & 0 & { - 1} \\ \end{array} } \right]$$
(4)

where A0 represents the transform matrix of the haptic device and A’0 the corresponding modified matrix.

In this way, we continuously calculated the forward kinematics transformation from the base to the final effector of the Baxter robot arm.

The control algorithm used is the same as that proposed by Ju [50] and is based on the teleoperative position–position control algorithm, which was the first control method implemented in haptic teleoperation schemes and is easy to implement in our system.

3.3 Graphical user interface description

There are many ways to control a robot remotely [59,60,61], although a widespread interaction approach is to control it using a graphical user interface, GUI. In addition, teleoperation user interfaces should provide features that can increase the operator's situational awareness, reduce their workload and improve their performance.

The proper design and the implementation of the user interface are the main factors that affect the performance of a human operator when using a teleoperated system, and an important part of the usability problems generally come from inadequate GUI designs [62]. Therefore, we consider it important to utilize a user-centered design methodology to develop intuitive interfaces.

To begin with this design phase, one of the first issues to consider when designing the GUI was the information that would be presented in it. We consider the visual information to be one of the most important factors in teleoperation as it allows the human operator to intuitively understand the slave environment from the master side of the system.

However, for this visual feedback to be effective, it is necessary to send the information from the slave side to the master and its transformation into images in real time, without delay. To achieve this, we use the ROS package under the ROS Kinetic distribution that makes use of the Video4Linux2 (V4L2) video streaming layer developed to capture video frames from the robot's cameras in real time. The settings used to control the streaming parameters of the ROS node include frame rate, frame resolution, image format, or pixel format, and path to the camera, or URL.

The video signal can be sent from a webcam placed on the remote site. Nevertheless, a single camera may not provide a complete picture of what is happening in the remote environment. In our study, we considered it necessary to show the operator the images of at least two cameras.

The other option is to create a virtual model of the remote environment and show it to the human operator at the master location. However, this solution can be difficult to implement, mainly because it is necessary to know not only the position of the robot effectors but also the rest of the objects in the remote environment at all times.

The best design choice is to design a GUI that combines both options. Therefore, we decided to visualize the real and the virtual environment on two different screens. To create the virtual model, we used the Gazebo Software (version 9.0), since the software provides a model of the Baxter robot.

Another important aspect in the design of the graphic interface is how to evaluate its usability. In most cases, the effectiveness and usefulness of the graphical interfaces of remote operating systems could only be demonstrated in a specific context and with highly trained users [63]. However, this requires long and difficult training of the motor and perceptive skills of non-expert users to achieve acceptable performance when using the teleoperation system.

For our study, we used an interface designed, evaluated, and improved in a previous study [64]. In that previous study, a first prototype of the GUI was designed and then, evaluated following the Ergonomic Guidelines for Interface Design (GEDIS) methodology [65]. A video describing the environment in which the tests were performed can be seen in the following repository [66]. This working environment is the same as the one used in this study.

As a result of the evaluation of that prototype following the GEDIS methodology, a final version of the GUI was designed (see Fig. 6), which is the one used to perform the experiment. We also decided to integrate the virtual environment into the GUI, eliminating the second screen and focusing the user's attention on a single view.

Fig. 6
figure 6

Graphical user interface used in the experiment

Full size image

4 Experimental method

Once the master and slave devices were programmed and the GUI’s design and evaluation were completed, we proceeded to design the experiment for our study.

Next, we detail how we tested the research hypothesis and how we answered the questions raised in the same section. To do so, the section is structured into several subsections as suggested by the APA [67].

4.1 Participants

We decided to use Convenience Sampling (also known as Haphazard Sampling or Accidental Sampling). The reason for this decision was that members of the target population met certain practical criteria, such as accessibility, availability at a given time, and willingness to participate in the study [68].

All participants were undergraduate or postgraduate students in different engineering degree programs at the University of León. In total, 44 students participated in the study (see Table 1 for details). Of the participants, 27 were men and 17 were women. The age of the participants ranged from 19 to 36 years (see Fig. 7).

Table 1 Participants’ details
Full size table
Fig. 7
figure 7

Age and gender distribution of the participants in the experiment

Full size image

As discussed above, none of the participants had previous experience with the use of a teleoperated system or in the use of haptic devices.

All participants filled out and signed a consent form in accordance with Organic Law 3/2018 of 5 December on the Protection of Personal Data and Guarantee of Digital Rights. No personal data or any other sensitive information were collected. Participation in the experiment did not involve the use of biological samples of human origin, animal experimentation, biological agents or genetically modified organisms. Therefore, we did not need any feedback from the Ethics Committee of University of León to conduct our study.

4.2 Experimental setup

Once the participants had been chosen, we prepared the environment in which the experiment would take place.

In this setting, the user was seated on the master side, where he/she operated the haptic device. The participant was able to view the GUI on a screen in front of him/her. When performing the experiment, the user was not able to see what was happening on the slave side. For this reason, the two working environments were separated by a panel (see Fig. 8).

Fig. 8
figure 8

Age and gender distribution of the participants in the experiment

Full size image

The environment was isolated to avoid distractions of the participant. The observers of the experiment were located in another room watching through a glass window. In: addition to a video camera recording the development of the experiment in real time. These observers did not intervene until the participant had finished the test or a problem occurred, and the user requested their presence.

4.3 Experimental task

Following the guidelines of the ISO 9241–940 standard on the evaluation of tactile/haptic interactions, we chose a multi-step pick-and-place task, a simple task for non-expert users.

The task in the experiment was similar to those performed in the game of checkers; that is, the user picks up a checker piece that is in one position, moves it and places it in another position.

The first problem encountered was the way to pick up the chip, which is round, with the grippers on the robot's arm. To solve this problem, we designed parts which can be gripped from the inside rather than from the outside, allowing for easier grasping. We also designed special grippers for the Baxter arm (see Fig. 9). Both parts were printed with a 3D printer with plastic filament (PLA) and in different colors.

Fig. 9
figure 9

Checkers used in the experiment and grippers of the Baxter robot arm prepared for printing on a BQ Witbox 3D printer

Full size image

The task was performed in two phases: a) first, a piece located in a specific place had to be picked up and moved to another position on the table, marked with a blue circle; b) then, starting from the initial position of the arm, the piece had to be gripped again and moved from this intermediate position to a final position, marked with a green circle. These marks were placed on the table at non-aligned locations and different distances, to prevent the user from automating the moving of the piece.

The task was considered to have been performed successfully when the piece was located within the marked limits. In:dicated on the table with concentric colored circles. These marks were 20% larger than the size of the piece (see Fig. 10).

Fig. 10
figure 10

Marks for the different chip positions on the table

Full size image

4.4 Experimental design

Before starting the test, it was necessary to check that the two devices (master and slave) were synchronized and that there were no significant errors in their operation. For this purpose, a developer and expert user performed the test and the position data of both devices (haptic and robot) were recorded. The results were then analyzed graphically and compared. Figure 11 shows that there was only a small deviation in the position of both devices when moving too fast. This deviation was corrected by the control system of the slave device itself.

Fig. 11
figure 11

Graphical representation of the position of the master and slave devices and comparison of both. The deflection that occurs in the case of too fast a displacement can be seen enlarged

Full size image

After this initial check, the participants were informed about the experiment and were requested to sign the consent form to proceed.

Next, each participant was taken to the environment where the experiment was to be conducted to see what the robot looked like and to check the different positions in which he had to move the part object.

Then, the participants were accompanied to the master part, separated from the previous one by a partition wall. On this side, the user had the haptic device and a monitor displaying the graphical user interface.

No training time was programmed although we did allow the user to manipulate the haptic device in the virtual simulator to learn the limits of movement of the device and the speed with which it transmits to the slave device.

Participants had to perform the task twice, once with visual feedback only and once with haptic feedback added. In the latter case, the haptic feedback allowed the user to appreciate the weight of the parts as he grasped them and moved them to their final position through the use of the haptic device. They were also able to sense the collisions of the robot arm with the table.

In order to minimize the systematic error or bias created by user experience, i.e., that the second test would be performed in less time than the first, given the test experience, we decided to randomize the order of the tests.

After the completion of each phase of the experiment, we gave each participant a survey sheet, the NASA Task Load Index Survey [69], to measure the workload. At the end of the two phases of the experiment, we asked the users to fill out a questionnaire to measure perceived usefulness and perceived ease of use based on the Technology Acceptance Model (TAM) [70]. In: order to obtain a measure of the degree to which a person believes that using the system with haptic feedback takes less effort to perform an assigned task than if only visual feedback is used.

This test procedure was applied to each participant. The following information was collected:

  • User efficiency, measuring the error rate when performing the task. The measurement of error rates included three variables: faults, drops and collisions. We considered that a fault had occurred when the user failed to place the piece within the established limits of the position on the table. A drop occurred when the user dropped the part while moving it from one position to another. Finally, a collision occurred when the robot arm collided with the table;

  • User effectiveness, measuring the time to complete the task;

  • Mental workload, according to NASA-TLX scores;

  • Perceived usefulness;

  • Perceived ease of use.

The analysis of all these variables provides a complete measure of the usability of the teleoperated system when a haptic device is used to control it.

5 Results

In Sect. 3, we proposed a hypothesis for the study and the following three research questions:

  1. 1.

    How does the integration of haptic feedback affect the performance of a teleoperated robotic system?

  2. 2.

    How does this integration affect the human operator's perception of usability?

  3. 3.

    Do variations in gender, age and level of education significantly affect task performance and workload?

5.1. Research question 1.

To answer the first research question, we analyzed user efficiency, user effectiveness and mental workload, and compared the results obtained with and without haptic feedback (represented in the tables by ON and OFF, respectively).

We began by analyzing the user's effectiveness in completing the task. From Table 2, it can be seen that the mean time to perform the test with only visual feedback is 219.74 s, with a standard deviation of 108.21 s. On the other hand, the mean time when haptic feedback is added is 215.53 s, with a standard deviation of 104.51 s.

Table 2 Time to complete the task
Full size table

Analyzing these data, we observed that although the task execution time is reduced when haptic feedback is added, this decrease is very small. If we do an analysis of variance (ANOVA), the F-ratio value is 0.89035 and the p value is 0.347996. This means that there is no statistically significant difference at p < 0.05.

When analyzed individually, 50% of users did perform the task in less time with haptic feedback, with an average difference of 7.99 s.

We then analyzed user efficiency by measuring the errors made by the user in performing the task (Table 3). Next, we compared the values of each of the errors when the task is performed with only visual feedback with those obtained by adding haptic feedback.

Table 3 Errors made in completing the task
Full size table

As indicated in Table 3, the average of errors with visual feedback only is 1.82 faults, with a standard deviation of 2.28 faults. On the other hand, the average errors when haptic feedback is added is 1.43 faults, with a standard deviation of 2.55 faults. Although the number of failures when executing the task is reduced when haptic feedback is added, this decrease is very small. The ANOVA returns an F-ratio value of 0.59033 and a p value of 0.446491. This means that there is no statistically significant difference at p < 0.05.

If we analyze the data individually, 72.7% of users made fewer mistakes when performing the task with haptic feedback, with an average difference of 3 faults.

Next, we analyzed the drops that users experienced when performing the task. The average number of drops with visual feedback is 3.32, with a standard deviation of 2.67 drops. However, when haptic feedback is added, the mean is 2.57 with a standard deviation of 1.57 drops.

Again, the difference is small (F-ratio value of 3.34407 and p value of 0.074389), and therefore, the differences are not statistically significant.

If we analyze the data individually, 70.45% of users committed fewer drops with haptic feedback, with an average difference of 2 drops.

Finally, we analyzed collisions. As shown in Table 3, the collisions when performing the test with only visual feedback is 0.32, with a standard deviation of 0.64 collisions. On the other hand, the collisions when haptic feedback is added are 0.23, with a standard deviation of 0.42 collisions.

This means that, although the number of collisions when executing the task is reduced when haptic feedback is added, this decrease is very small. If we do an analysis of variance (ANOVA), an F-ratio value is 0.60993 and a p value is 0.439093. This means that there is no statistically significant difference at p < 0.05.

An individual analysis shows that 84.09% of users experienced fewer collisions with haptic feedback, although the average difference is very small (practically zero).

Mental workload was measured using NASA-TLX scores. This test is designed to measure the amount of mental effort and concentration a person must exert in order to complete a task. The test analyzes six different variables: mental demand, physical demands, time demands, effort, frustration level and performance.

The first of the variables (mental demand) measures how much mental and perceptual activity was necessary to perform the task. Physical demand analyzes how much physical activity was needed to complete the task and determines whether the task was easy or difficult, relaxed or tired. The third variable, related to temporal demand, aims to measure how long the user felt pressure when performing the task. In terms of effort, it measures how hard one has had to work to complete the task successfully. The level of frustration allows us to know to what extent the participant has felt insecure, discouraged, tense or worried when performing the task. The last variable, performance, measures to what extent the participant in the experiment believes he has succeeded in the objectives set by the researcher and what is the degree of satisfaction with the level of execution of the task.

The test is applied in two phases. In the first phase, we obtain the importance (weight) that each participant gives to each of the six dimensions. For the collection of the necessary data, we used the binary comparisons procedure. We established the 15 possible binary comparisons among the six dimensions, and the participant had to choose, from each pair, the one he or she perceives as the greatest source of burden. For each dimension, we obtained a weight that is given by the number of times it was selected in the binary comparisons. This weight can vary between 0 (the dimension was not selected in any of the comparisons) and 5 (the dimension was selected in all the comparisons in which it appeared).

Table 4 shows the results of the two tests, with visual feedback only and adding haptic feedback.

Table 4 NASA-TLX converted scores
Full size table

From the data presented in Table 4, ‘physical demands’ comes up as the variable with the higher weight. The variables corresponding to mental demands, effort and performance are the next most weighted. The level of frustration has a lower weight than the previous ones. Finally, since we did not set a time limit in our experiment, the variable corresponding to time demand has no specific weight.

Next, we calculated the converted score for each participant and computed the median. A graphical representation of these values can be seen in Fig. 12. Analyzing these results and comparing them to when we added haptic feedback, we can see that the mental demands are the same in both cases. The physical demands are higher in the case where the user is using haptic feedback, although there is no statistically significant difference. Despite this, we observe that the variable corresponding to effort is lower when haptic feedback was added, although there is no statistically significant difference.

Fig. 12
figure 12

Graphical representation of the NASA-TLX converted scores

Full size image

As for the level of frustration, the variable is higher in the case of having only visual feedback during the experiment. For this variable, we can observe that if we do an analysis of variance (ANOVA), the F-ratio value is 5.44476; the p value is 0.024372. The difference is significant at p < 0.05.

Finally, we observe that the variable corresponding to performance is higher in the case of using haptic feedback to complete the task. For this variable, we can observe that if we do an analysis of variance (ANOVA), the F-ratio value is 16.58123; the p value is < 0.000196. The difference is significant at p < 0.05.

If we analyze the scores globally, we have to calculate the weighted average score. To do this, we multiply the converted score of each variable by its weight. We then add these values and make the weighted average. Thus, we observe that. In: the case of the test with visual feedback only, the mean mental workload is 49.33. However, when we add haptic feedback, this value increases to 53.33. This means that the user perceives a higher mental workload when using both types of feedback (visual and haptic), although the difference between the scores in the two cases is not significantly different.

Table 5 shows the results of the two tests.

Table 5 NASA-TLX weighted scores
Full size table

6 Research question 2

To answer the second research question, we analyzed the perceived usefulness and perceived ease of use of the system. The items used to measure perceived usefulness were the following.

  • Item 01: Performing this experiment has helped me to understand haptic devices;

  • Item 02: The use of the haptic device would improve task performance in virtual environments;

  • Item 03: Using the haptic device with feedback would allow me to perform tasks in virtual environments faster;

  • Item 04: Using the haptic device with feedback seems to me to be more appropriate when creating my own tasks in virtual environments in a simpler way;

  • Item 05: Using this device with feedback would limit me in performing certain teleoperation tasks in virtual environments;

  • Item 06: How much experience do you have in the use of similar systems?

Users had to make a choice on a 5-item Likert-type scale from strongly disagree to strongly agree.

If we analyze the overall results obtained to evaluate the perceived usefulness, we can say that. In: general, participants perceive that the use of haptic feedback improves performance when performing the task. Thus, we can say that 59% of the participants agree or strongly agree, while 25% disagree and 16% neither agree nor disagree.

Figure 13 shows a graphical representation of the results.

Fig. 13
figure 13

Graphical representation of the perceived usefulness in the TAM test

Full size image

It is remarkable to note that 82% of the users think that conducting the experiment has helped them understand haptic devices (Item 1). This is very interesting given that 93% of the participants in our study have little to no experience in using these devices (Item 6).

It is also interesting to note that 93% of the users believe that the use of the haptic device could improve performance when performing the task (Item 2), but only 68% believe that they could complete the task in less time with haptic feedback activated (Item 3).

Finally, we observed that 70% of the participants think that the use of the haptic device with feedback is more appropriate when performing tasks in virtual environments. In addition, 32% of the users consider that activating haptic feedback in this type of device could limit the performance of certain teleoperation tasks. However, 43% of the participants think the opposite, with the rest (25%) neither agreeing nor disagreeing.

The items used to measure perceived ease of use include the following.

  • Item 07: Learning to use the haptic device to remotely control the robot arm has been easy for me;

  • Item 08: My interaction with the haptic device has been clear and understandable;

  • Item 09: I find the haptic device flexible when interacting with it;

  • Item 10: It would be easier for me to use the haptic device in a real teleoperation environment when feedback is enabled;

  • Item 11: I find the haptic device easy to use in teleoperation environments.

If we analyze the overall results obtained to evaluate the perceived ease of use, we can say that. In: general, the participants perceive that the use of the haptic device does not involve any extra effort. Thus, we can say that 65% of the participants agree or strongly agree, while 10% disagree and 25% neither agree nor disagree.

Figure 14 shows a graphical representation of the results.

Fig. 14
figure 14

Graphical representation of the perceived ease of use in the TAM test

Full size image

We now analyze the results for each of the items in more detail. In the case of the first question (Item 7), we observe that only 7% of the participants surveyed consider that it was difficult to learn to use the haptic device. On the other hand, 68% of the participants thought that learning to use the device was not difficult, the remaining 25% being undecided.

If we analyze user interaction with the device (Item 8), we can see that 68% of users consider this interaction to have been clear and understandable, compared to 5% who think otherwise. Furthermore, 55% of the participants in the experiment consider that the device is flexible when using it in the execution of the task (Item 9). However, 15% of the users consider the flexibility of the device not acceptable, leaving the remaining 32% with no opinion.

Regarding the use of haptic feedback, 11% of the users interviewed consider that adding this feedback does not facilitate the use of the haptic device in a real teleoperation environment (Item 10). 64% of the participants do consider it positive and the remaining 25% are undecided.

7 Research question 3

To answer the last research question, we had to analyze the gender, age, and level of education of participants. As we saw in paragraph 4.1. In: the distribution of participants by gender, there are 27 men and 17 women. This is due to the fact that historically, there has been a gender bias in the sciences and engineering fields, leading to a lack of female representation.

In terms of age, the majority of participants are in the 20–23 age group. This does not allow us to analyze whether age influences performance and mental load when executing the task. The same happens with the level of education, being most of the participants, undergraduate students. As a result, we only analyze the variables by separating the corresponding data by gender. The data for the first two variables can be found in Table 6.

Table 6 Time and errors made in completing the task differentiating by gender of the participant
Full size table

First, we analyzed the data corresponding to the execution time of the task. We observed that the time when performing the task with only visual feedback (OFF) is higher for females (254.52 s) than for males (197.84 s). The same happens when we add haptic feedback (ON). The mean task execution time for females is 226.59 s, while for males it is 209.12 s.

To see if this difference is significant, we use the Mann–Whitney U test, since it seems more appropriate as the number of values is different for women (17) than for men (27). In the first case, with visual feedback only, the results are: the value of U is 146, the z-score is −2.00058, and the p value is 0.0455, which means that the difference is significant at p < 0.05.

When we add haptic feedback, the value of U is 210, the z-score is −0.45796, and the p value is 0.64552, which means that the difference is not significant at p < 0.05.

Let us now analyze the errors. We start by analyzing the failures made when performing the test. We observe that the average number of faults committed is higher in the case of females in both cases (OFF and ON). Although, when performing the Mann–Whitney U test, the difference is only significant in the first case, where the results are: the value of U is 125, the z-score is −2.50675, and the p value is 0.01208.

Regarding the drops that occur when performing the test, we observe that the average is again higher in the case of females in the two tests (OFF and ON). However, when applying the Mann–Whitney U test. In: neither of the two cases is this difference significant.

Finally, we analyze the collisions that occur when performing the task, observing that the mean is again higher in the case of women in the two tests (OFF and ON). However, as in the case of drops, when applying the Mann–Whitney U test. In: neither of the two cases is this difference significant.

Next, we compare the results of the NASA-TLX test, differentiating the gender of the participant. See the results in Table 7.

Table 7 NASA-TLX scores differentiating by gender of the participant
Full size table

We use the same weights assigned to each variable, since there is no gender difference. We then compute the converted score for each participant and calculate the median and weighted score.

We analyze each of the six variables of the NASA-TLX test individually, comparing the results by gender. The first variable, mental demands, is the same when there is only visual feedback (OFF). When we add haptic feedback (ON), this variable increased in the case of women and did not vary in the case of men. We can see a graphical representation of these values in Fig. 15.

Fig. 15
figure 15

Graphical representation of the NASA-TLX converted scores differentiating the gender of the participant

Full size image

In the case of the second variable, physical demands, we did observe differences between genders. To test whether this difference is significant, we used the Mann–Whitney U test, as before. We began by analyzing the results when there is only visual feedback (OFF). In this case, the variable is lower for females. But according to the test results, this difference is not significant (p = 0.14457) at p < 0.05.

When we add haptic feedback (ON), the weighted score increases for females and decreases for males. In this case, the test results are; U 125.5, the z-score is –2.39096, and the p value is 0.00842. This means that the difference is significant at p < 0.05.

As expressed at the beginning, temporal demands are not considered, as there is no time limit for performing the task.

For the variable corresponding to effort, we observed that when there is only visual feedback (OFF), the results are the same for men and women. In the case of visual and haptic feedback (ON), there is no change for women. For men, the weighted score decreases, although the difference is not significant according to the Mann–Whitney U test results.

Let us now analyze the variable corresponding to the level of frustration. In this case, the values are exactly the same, both in the case of visual feedback only (OFF) and in the case of adding haptic feedback (ON). Therefore. In: this variable there is no difference between genders.

Finally, we analyze the results of the last variable, performance. Again, there are no differences between genders for the two tests.

If we analyze the scores globally, we have to calculate the weighted average score. To do this, we multiply the converted score of each variable by its weight. We then add these values and calculate the weighted average. Thus, we observe that. In: the case of the test with visual feedback only (OFF), the mean mental load is 42.67 for females and 49.33 for males. The results of the Mann–Whitney U test are: U = 151, z-score = 1.76882, and the p value is 0.03836. This means that the difference is significant at p < 0.05.

However, when we add haptic feedback (ON), the value for the case of females increases to 60.67, whereas for males it decreases to 41.33. The results of the Mann–Whitney U test are; U = 121, the z-score =  − 2.50074, and the p value is 0.00621. This means that the difference is significant at p < 0.05.

8 Discussion

The study examines the aspects involved in the assessment of usability of haptic feedback when used in teleoperated robotic systems. The initial research hypothesis was that, from the point of view of a non-expert user, haptic feedback improves usability in teleoperated tasks in which a robot is remotely controlled. For a complete evaluation of usability, we used three of the elements proposed by Khan [42] (efficiency, effectiveness, and satisfaction) and added the mental workload from [47, 48]. The results of the analysis of all these variables corroborate the initial hypothesis, although in some cases they do not have a statistically significant impact.

To further confirm the research hypothesis, we formulated three research questions. In the first, we tried to find out how the use of haptic devices affects the interaction of a teleoperated robotic system. To answer this question, we analyzed the efficiency, effectiveness and mental workload involved in using the system. Overall, we observed better results when force feedback was added, but there were no statistically significant differences, except for the measurement of the performance variable in the NASA-TLX test. The results seem to confirm Weber's quantitative review of empirical studies [17] investigating the effects of haptic feedback in teleoperation systems.

For each of these variables individually, we observe the following: to measure efficiency we collected the errors made by the user when performing the task. We considered three types of errors for the task posed in the experiment: failures, crashes and collisions. The results show that the error rate is reduced when haptic feedback is added. Only 24.25% of the users who participated in the study made more errors with haptic feedback activated. However, statistically there is no difference between the two tests, as in [71]. We believe this is because the number of errors made by the study participants is very low in both cases. Considering that the users had no experience in using this type of system, this low error rate is unexpected. The reason may be that the task that was assigned is very simple.

To evaluate the effectiveness, we measured the time the user took to complete the task. The results indicate that participants completed the task faster if haptic feedback is added. However, as in the previous case, this difference is not statistically significant, since the mean difference is 77.99 s over a maximum value of 511.34 s. In this case, we believe that the reason is that the task posed is too simple and that the time taken to complete it is not very long (approximate mean value of 206 s in both cases). Nevertheless, considering that the users had never used a teleoperated system before, we believe that this improvement is very positive.

Finally, with regards to mental workload, the results indicate that the user perceives a greater mental workload when using haptic feedback to perform the experiment than when not using haptic feedback, although the difference between the scores in the two cases was not significantly different. We believe this is due to the participants' lack of experience in using haptic devices and in completing teleoperation tasks, as demonstrated by Pervez [72], which shows that learning from demonstrations can reduce the human mental workload when learning repetitive teleoperation tasks.

In the second research question, we asked whether participants' perception of usability increases when we add haptic feedback to visual feedback. To answer this question, we observed perceived usefulness and perceived ease of use. The results indicate that the usability perceived by the human operator when we add haptic feedback to the system improves both perceived usefulness and ease of use. These results are in line with Radi and Nitsch [71].

Two results are noteworthy in this case: the high degree of satisfaction in the use of the teleoperated system with haptic feedback, and the other ease of learning. Considering that participants had no experience in the use of these devices or in the use of teleoperated robotic systems, these results seem very positive to us.

With the last research question, we tried to determine whether the gender of the participants in our study influences performance and mental workload. The results indicate that the gender of the user performing the task does influence both task performance and workload, as there are statistically significant differences. We believe that this result is due to gender differences in spatial skills (including visuospatial abilities such as spatial orientation and spatial visualization) and motor skills, as demonstrated in several studies [74–77].

9 Conclusion

The use of teleoperated robotic systems enables users to perform and complete tasks in hazardous environments. It also facilitates working in safer and more isolated environments, which is very important in the current pandemic situation. But to make such systems easy to use by casual and non-expert users, it is necessary for those users to have the feeling of physically being in the remote environment. The use of force feedback provided by a haptic device on the master side of the teleoperated system can help with that telepresence condition. This haptic feedback is viewed as a complement to the visual and auditory feedback provided by the graphical user interface (GUI).

However. In: attempting to evaluate the usability of haptic feedback integration in teleoperated robotic systems, we encounter a major problem. This is because there are a large number of applications of these systems in fields as diverse as medical care and surgery, aerospace and military industry, etc. To this, we must add that the tasks to be performed in each of these fields are also very large and range from very simple and repetitive, such as the manipulation of objects or tools, to much more complex tasks, such as those performed in a surgical operation or in the disarming of explosive devices.

This lack of homogeneity means that there is little common framework for this usability evaluation.

Our study is intended as a preliminary step to develop and validate a usability evaluation framework for teleoperated robotic systems with haptic feedback.

In this study, we have chosen to do an empirical investigation in which we start from an initial hypothesis, design an experiment to test this hypothesis and analyze the results obtained. This hypothesis states that the integration of haptic feedback in teleoperated robotic systems improves the usability of the system.

We have chosen several usability factors used by other authors to evaluate haptic systems and have added other variables such as mental workload. We have also added a questionnaire to ascertain the degree of acceptance by non-expert users participating in the experiment.

Analyzing the results obtained, we can conclude that the experiment carried out with this number of people and with the execution of a simple task were not conclusive, although some metrics indicate that the integration of force feedback could be valuable.

Although this improvement is not statistically significant, the feeling of presence of the human operator is. Considering that the users who participated in the study had no previous experience and no learning period, these results are very positive.

In future work, we intend to repeat the experiment, designing a more complex task using both arms of the collaborative robot. We also plan to use different haptic devices to compare with the results obtained in this study. This will allow us to develop a common usability evaluation framework and validate it.