1 Introduction

Many existing approaches designed to be used in rehabilitation settings are based on technologies to provide what is known as serious games. These solutions assist the routine work carried out by physiotherapists by allowing patients to perform their exercises in a more effective, enjoyable, and comfortable way. Although they are useful for specific injuries, some technologies lack: 1) the capacity to design custom postures; 2) an intuitive environment for the creation of rehabilitation exercises; and 3) flexibility, since patients must go to specific places to be directly monitored by physiotherapists.

In recent years, several interaction devices have emerged that allow the user’s movement to be captured. This new form of interaction has marked a new era in the way users communicate with and ultimately control computers. With this approach, namely ‘movement-based interaction’, the body replaces or enriches the interaction carried out with a mouse, keyboard, or touch screen (among others). By using motion sensing input devices such as Microsoft Kinect, it is possible to connect the patients’ movements with a 3D virtual environment.

As a result, physiotherapists are now able to design custom postures for the rehabilitation process. These postures are designed with one (or more) pathologies in mind, and they can be reused for several patients. The creation and execution of postures is carried out via a highly intuitive environment. Ultimately, postures can be combined for the creation of exercises, providing a guide while they are being performed. These exercises can be performed locally (e.g. in hospitals) or remotely (e.g. at the patient’s home), thanks to the use of the Web as the platform for developing and deploying this solution. This provides a great advantage and freedom both for the physiotherapists and the patients. On the one hand, physiotherapists can create personalized exercises and follow up the rehabilitation process of their patients at any time and any moment, as the 3D editor has been implemented as a web-based tool. On the other hand, patients can perform the exercises developed by their physiotherapist at any time at home without having to commute to the rehabilitation center or hospital, making rehabilitation easier to follow.

The system has been developed following a user-centered design methodology. An initial evaluation checked whether the virtual editor could be used correctly. Additionally, the final version of the system was evaluated with a usability test. These evaluations involved a wide range of users and demographics, such as teenagers, adults, older adults, and professional therapists.

The rest of the article is organized in the following sections. Section 2 describes the related work, highlighting the novel aspects of the system proposed. Section 3 describes the design of postures using the virtual 3D environment. Section 4 presents a discussion of the evaluation of the system. The last section contains the conclusions and lines of future work.

2 Related work

Current technological ecosystems offer possibilities for improving conditions or solving problems that could never have been imagined previously. In particular, new interaction mechanisms allow computer systems to adapt the interaction to the users’ needs and conditions. These systems use virtual body representations in a realistic virtual avatar or an abstract one, since it has been probed as a good solution [9].

In the context of this paper, video cameras are the devices considered as the option to investigate and obtain results from new interaction experiences. Currently, these devices offer solutions in different environments [15]. Evidently, there is a need to explore and to discover not only the benefits of video cameras but also their limitations. This knowledge will assist the development of new systems, exploiting the device’s capabilities in those contexts where its use makes sense [5].

One of the main applications of video cameras in research is that of movement-based interaction. Many research fields have found opportunities for improving this interaction. For example, there are works related to sports in which it is used to train specific abilities, such as in karate [22]. However, one of the fields that has drawn the most attention is healthcare, which is the focus of the system presented in this paper.

Movement-based interaction represents an attractive way to facilitate the rehabilitation process. There has been a marked evolution in movement-based systems from the specialist’s point of view, with the emergence of serious games that use movement-based interaction, which opened up the possibility of using such innovative methods to support rehabilitation. The ‘gaming’ concept provides the possibility of helping people in their rehabilitation in a more entertaining way. Many contributions have consisted in a specialist choosing specific games for a given rehabilitation process [8] [16] [18]. The main limitation is that the customization of exercises is not considered [20], which represents a problem for specific patients. A solution appeared after the development of systems that offered specific exercises for specific lesions [10] [14]. These systems took a step forward in rehabilitation technology but focused only on specific lesions and were not useful for others.

New solutions appeared in which the physiotherapists take part in the development of the exercises offered by the system [2]. However, the physiotherapists are limited and cannot request new exercises that are tailored to the patients’ needs. One solution is the possibility of allowing the physiotherapists to generate specific exercises through the generation of XML files. In this case, the medical staff require a training program but, in any case, the system can be useful for a large number of patients.

As a natural development, a system allowing the physiotherapist to generate new exercises without having to edit files is presented in [21]. The physiotherapist can create a new exercise by demonstrating the desired exercise ten times. Apart from requiring a video camera, five motion sensors need to be worn by the patients.

The related works mentioned above represent essential steps in improving conditions in rehabilitation processes. Taking a further step forward, this paper presents a system that enhances the freedom of the specialist physiotherapist when generating and editing rehabilitation exercises and therapies. The physiotherapist does not need to be assisted by developers or to repeat exercises a specific number of times. The system contains a 3D environment in which the medical staff can easily generate postures and define exercises based on the repetition of postures. There is no need to work with technical staff to edit files or to contact and work with the developers. The interaction between the specialist and the system is by means of the Kinect sensor, using their voice and movement.

With regard to our previous experience in the field of rehabilitation technologies and movement-based interaction, we can highlight the following works: [6] [12] [7] [11]. Unlike these earlier works, in this article we present a system aimed for physiotherapists to design personalized rehabilitation exercises, via a virtual exercise editor, that can then be performed and monitored by the patients at home thanks to the application previously developed for this purpose [11].

3 3D virtual exercise editor

Traditionally, the postures that a patient has to adopt were designed on a piece of paper. It is possible to use storyboards or simple diagrams, firstly so that the physiotherapist can study what exercises the patient should do, and secondly so that the patient has a guide to follow. The designing of the exercises may seem straightforward, but it does involve certain complex matters such as the direction in which the movements in the exercise should be performed.

As can be seen in Fig. 1a, the physiotherapist can design an exercise that is based on two (or more) postures: an initial one and a final one. In order to help the patient to understand the transition between postures, it is possible to include arrows, as shown in Fig. 1b. This helps to determine how to perform the exercise. However, in the case shown in this figure, three different possibilities can be observed when it comes to raising one’s arms from the initial position to the final one. Thus, as this case illustrates, the main problem that arises is that the desired transition for reaching the final posture from the initial one may not be obvious, there existing more than one possible movement between the two postures.The system we propose here allows physiotherapists to design postures, as well as add transitions between them. This is carried out by means of a Web-based 3D editor which provides them with a series of tools for manipulating the skeleton in three-dimensional space, thus enabling them to find appropriate postures. The system incorporates numerous tools that facilitate this task such as change view, undo/redo, copy/paste postures, delete postures or transitions, etc.

Fig. 1
figure 1

Designing postures and transitions. a) Initial and final postures. b) Transitions between postures

Full size image

The postures can be designed in two different and complementary ways and adapted and personalized for the patient and their progress at any time. The physiotherapist can easily switch between these two modes, which are: (1) manually; and (2) via the direct capture of the movement of the physiotherapist’s body. In manual mode, the physiotherapist can manipulate the skeleton in the virtual environment to configure the desired posture. They can shift it along the three axes or move its joints. When in body movement capture mode, the physiotherapist can use their own body to capture the posture they wish to record. This capture is carried out by the movement-based interaction device Kinect v2. The multimodality of our system then allows the use of voice to control the application, in the same way that it can be controlled by a mouse. This is one of the novel features which distinguishes it from other similar works, namely that it consists in a multimodal system based on voice recognition and movement-based interaction. Once two or more postures have been created, transitions between them can be generated so that the movement of the body is exactly what the physiotherapist considers appropriate for the rehabilitation process. A succession of postures, with the corresponding transitions between them, make up a complete exercise which the physiotherapist designs for a specific rehabilitation. Furthermore, a set of exercises can be grouped together into what is called an exercise routine. The patient receives these routines and will have to perform the 3D exercises that the physiotherapist has created for them on a personalized basis. The patient will be monitored in real time in order to check whether they are carrying out the exercises correctly or not. In addition, the application guides the patient in the process of performing the exercises and corrects them in case of error [11]. The monitoring, guidance and error warnings are based on the capture of the position and movement of the patient’s body by means of Kinect v2. The patient’s posture, as well as the transitions between postures, is constantly compared against what has been designed by the physiotherapist for a given exercise to check that they match.

Lastly, we should point out that it is a highly customizable application. The exercise that the patient has to perform has been designed exclusively for that patient in order to resolve their particular problem. The physiotherapist can have a series of basic postures stored on the system and then use them as a starting point, but then the physiotherapist adapts them to meet the needs of a given patient. Furthermore, the physiotherapist can then continue to adjust them in accordance with the data returned by the system monitoring the patient’s progress. The physio can decide to change the difficulty thresholds or the times for the correct detection of a posture at each one of the joints or even in the exercise as a whole. If many errors are detected in the movement of the patient’s arm, for example, the physio can vary the detection settings for that arm in such a way that the system is better adapted to promote the patient’s rehabilitation.

Figure 2 shows the relationship between the main modules of the system. The 3D Virtual Exercise Editor Module gives the physio access to the editor, which includes all the tools necessary for the design of postures, exercises and exercise routines in the two modes described above. In this article we describe the design of postures, in the context of exercises and exercise routines, as well as in relation to the rest of the system. The Rehabilitation Module continually monitors and guides the patient while they are performing the exercises [11]. The patient sees their movements in real time and receives feedback, which is both visual and auditory, about the next steps to perform. The Statistics Module gathers a series of statistics about the performance of the exercise routines and allows the physiotherapist to see the patients’ progress. The server, which is termed a WebSocket Server, controls the Microsoft Kinect device via its SDK, and has access to skeleton detection and the devices’ voice control. It also centralizes all types of information about patients, exercises, and statistics.

Fig. 2
figure 2

Main modules that make up the system

Full size image

Next, we describe some of the features of the virtual environment, the skeleton, and its manipulation.

3.1 Editing and manipulation

Figure 3 shows a partial view of the virtual editor, which allows the physio to design personalized postures for the patient undergoing rehabilitation. The workspace shows a skeleton on which the joints that the physio can manipulate stand out. This skeleton is in a plain room, without distractions, in which the physio can include marks. The dimensions of the workspace fit with the workspace allowed for the performance of movement in Kinect v2.

Fig. 3
figure 3

Partial view of the 3D Editor

Full size image

Our application includes the new features of the Kinect for Windows v2, which have improved the system in different aspects.

WebGL (Web Graphics Library) technology was chosen to implement the 3D editor. WebGL is a JavaScript API for rendering interactive 3D graphics and 2D graphics within any compatible Web browser and without the use of any plug-in. As WebGL is a technology designed to work directly with the GPU, it is difficult to codify in comparison with other more accessible web standards. To solve this problem, several JavaScript libraries have been developed, such as C3DL, SceneJS, Three.js and X3DOM. In our system, the Three.js library was selected because it has the greatest number of users. This is due to its lightweight library and low level of complexity in comparison with the original WebGL specification.

The virtual environment has different toolbars to facilitate the design of rehabilitation postures. These toolbars provide a number of tools to speed up the design process, such as undo or redo movements, the selection of a particular perspective or an already defined posture so as to start designing from that position. The different toolbars are described below.

In order to facilitate posture design, a range of defined postures are offered to the physiotherapist: basic and favorite postures. Then, when the physiotherapist chooses one of these postures, it will be loaded and shown in the virtual editor where the physiotherapist will be able to create the desired posture with the available tools. Thus, the design process will be more efficient and more precise. As shown in Fig. 3-a, some basic postures are automatically provided by the system. In this way, the physiotherapist can immediately start designing, taking the basic posture that best fits the posture they require. Figure 3-c shows a list of several favorite postures. Once the physiotherapist creates a posture, it can appear as a favorite if desired. Favorite postures can be used in the same way as the basic ones.

The edit toolbar (Fig. 3-b) allows the physiotherapist to access the design tools, such as undo and redo to provide a back and forward navigation between the movements, in order to achieve a desired posture. The capture from the Kinect tool activates the movement-based design by directly using the device. After this capture, the posture can be adjusted more accurately by means of the editor. The physiotherapist can choose the most appropriate view for the movement that they want the skeleton to make. The front and both side views of the environment allow the therapist to see the skeleton in the common perspectives, thus facilitating posture design. However, the physiotherapist can also rotate the camera in three dimensions to observe the skeleton from every possible angle. To do this, the physiotherapist has to click and rotate the 3D scene. Figure 3-d shows the x, y and z axes of the virtual 3D space. They are highly useful when moving the different joints of the skeleton together. For instance, if the controller of any axis is moved, it can be seen how the skeleton moves along this axis. The posture editor axis lock system indicates the axes in which the skeleton can move by highlighting the enabled axes in green. The axis lock system will be described in the next section.

In order to represent the postures that the physio needs to design for their patients’ rehabilitation exercises, a representation of the human figure that is based on a series of points is used. The points, which correspond to the human body’s main joints, are joined by lines, thus forming a skeleton which the physio can manipulate by means of these points. Kinect v2 allows the identification of 25 points, as shown in Fig. 4 [1]. Our system is not intended for the rehabilitation of more delicate movements, such as those of fingers or hands, so not all the joints detected are necessary, although the skeleton we have designed is based on the one shown in Fig. 4 by reducing the number of joints.

Fig. 4
figure 4

Joints detected by Microsoft Kinect v2. Image from [1]

Full size image

The physics of the skeleton and interaction with it are not trivial. The algorithm that enables interaction with the skeleton must take into account the relationships between the joints. The movements made by one joint may affect others, and if this is the case, this should happen in a natural way in order to avoid movements and postures of the skeleton which are unnatural. The possible movements and their effects on other joints have been carefully studied, with the additional aim of avoiding impossible postures.

Furthermore, the manipulation of the skeleton in a three-dimensional environment via control devices that work in two dimensions creates certain problems. Depending on the view in which the skeleton is moving in the process of designing a posture, moving a joint could have one effect or another. Perhaps moving a joint, apparently along the x-axis for example, has led to an unintended displacement along another axis, of which one only becomes aware when changing the view. Obviously, this is a problem which does not arise if the posture is generated by means of interaction based on movement, capturing the rehabilitator’s body movements. But when using a mouse, which is intended for 2D, this problem can arise. In order to solve it, our system locks one of the axes so that the joints only move within the pair of most logical axes which the physio sees in the view that they are working in. If the physio rotates the view and passes a certain threshold, the pair of active axes changes in accordance with what seems most logical when considering what the physio is looking at in that view. This completely solved the problem, allowing natural interaction with the skeleton and removing the type of errors in posture movements that are unexpected by the physio.

For example, if the user is in the front view mode, then the movement is blocked in the z axis, so that no wrong movements can be made and the actual movement that the user wants to achieve is reflected. Conversely, if the editor is in a customized view which does not match any of the predefined ones, then the movement of the skeleton’s joints along the x axis or the z axis will not be permitted to ensure that unwanted strange movements are not performed. In this way, we ensure that no joint will move about an axis if it is not enabled in this view, ensuring a 3D design through 2D devices.

The skeleton can be manipulated and moved in different ways in the virtual environment of the posture editor. By using drag and drop it is possible to move the skeleton as a whole around the environment, provided it stays within the margins that can be detected by the movement detection device, namely Kinect v2, which matches the patient’s posture with the posture that has been designed. From the axes shown in Fig. 3-d it is also possible to move the skeleton in any of the three dimensions. By using drag and drop on the skeleton’s joints, the physio can move the skeleton how they wish in order to design a posture. They can start from a collection of basic postures with a single click, which will cause the skeleton to adopt the basic posture selected, which the physio can then adapt (Fig. 5).

Fig. 5
figure 5

Manipulation of the skeleton in manual mode

Full size image

The second way of designing postures that the editor offers involves the physio using their own body to generate the posture by means of movement-based interaction. In this case, the interaction with the application changes. Instead of using the mouse, voice control is used, as it is assumed that if the physio is using their body, they will be standing away from the computer to interact with the Kinect camera, and in these circumstances the best way of carrying out the actions is by means of voice commands.

These commands allow the user to select, save or discard postures. They also allow the physiotherapist to move around postures and transitions.

The voice commands that can be used in the application are the following:

  • “START” to activate exercise recording.

  • “PHOTO” to capture the current position, just to freeze the posture and check if it is the desired one.

  • “POSTURE” to save the current position and create a new one.

  • “SAVE” to save the status of the editor: posture or photo.

  • “DISCARD” to skip and continue with motion detection.

  • “LEFT” to activate the left perspective.

  • “RIGHT” to activate the right perspective.

  • “CENTER” to activate the frontal perspective.

  • “NEXT” to move to the next posture or transition.

  • “PREVIOUS” to move to the previous posture or transition.

  • “GO TO...” followed by the posture number to edit such a posture.

  • “TRANSITION...” followed by the letter of the transition to edit that transition.

  • “OK SIVIRE” to stop voice detection.

For example, to create an exercise to raise the left arm, (1) the physio stands up and stands in front of the sensor, checking that the virtual skeleton responds to their movements properly. (2) Then, they say “START” out loud to activate the exercise recording. (3) Then, they perform the initial posture “arms close to the body” and say “POSTURE”. This position is automatically saved and a new one is created. It is recommended to use the “POSTURE” command when the user is sure of the posture performed, otherwise it is preferable to use the “PHOTO” command to freeze the image, review it and then use the “SAVE” command to validate it. In any case, the user can use the “DISCARD” command at any time to delete the posture. (4) Then, they perform the posture “arms crossed” and say “POSTURE”. This posture is automatically saved and a new one is created. (5) Later, they perform the final posture “arms crossed” and say “PHOTO”. They approach the screen to check that the posture is correct and say “SAVE”. (6) Finally, they say the command “OK SIVIRE” to stop recording the exercise.

In any case, the physio can switch between the different modes at any point. For example, the physio could start with a posture captured with their own body by means of movement-based interaction, and then adjust the skeleton’s joints manually. Later, when the physio has started to receive feedback about the exercises performed by the patient, they can also adjust the postures to ensure that the rehabilitation process is appropriate for the patient’s progress.

Another important aspect that should be highlighted is that even movement-based interaction can take place over the Web, so that the physio only has to connect to the application via the browser, from anywhere and whenever they wish, in order to either design postures or monitor the patient.

This interaction between the user and the system through the Kinect camera, with movement and voice-based interaction, is a highly important aspect of this system since it allows the physiotherapist to design postures in a natural way. However, providing a web-based system, which is what we have chosen, is not easy because the Kinect SDK is aimed at desktop applications. The solution adopted is based on the use of WebSockets, a new and emerging technology within HTML5. This technology provides a bidirectional communication channel with the use of just a TCP socket. Implementing this technology makes it possible to provide bidirectional communication between the Microsoft Kinect device and the web application in the following way:

  • The Web application acts as a client that will be responsible for starting the connection with the server.

  • The server, referred to as a WebSocket Server, has to be specifically implemented. This server will control the Microsoft Kinect device with its SDK. Therefore, it can access the skeleton tracker and the device’s voice control. In addition, it is in charge of checking whether the posture is right or wrong.

  • There is an exchange of messages between the WebSocket Server and the client. Each time a skeleton is detected by Kinect, the information is sent via a message in JSON format by the server to the client. The same process is carried out when it detects a voice command. Finally, when a posture is checked, the result is sent to the Web application.

In order to make the application meet the requirements and to use the Web as a platform, the coordinate system employed is different from the one used by Kinect v2. An internal automatic transformation process fluently translates one into the other so that the application and the device can communicate with each other.

The aim of the system is to create and design personalized postures. Figure 6 shows the basic flow to define a posture. This diagram outlines the basic steps followed in the design of postures to clarify the process.

Fig. 6
figure 6

Basic flow to create a posture

Full size image

3.2 Estimation and use

Once the posture has been designed by the physiotherapist, it is time to check whether the designed posture correctly synchronizes with the Kinect coordinate system. Then, the patient will be able to try this posture and the system will evaluate whether it is correct or not. To do this, a posture checker, whose interface is partially shown in Fig. 7, was implemented.

Fig. 7
figure 7

Postures checking

Full size image

This checker shows a list of several previously designed postures for the patient, highlighting the currently selected posture. Furthermore, it shows the virtual room and the patient’s skeleton with its immediate movements. Once, the patient wants to verify their posture, s/he provides a command to the system through their voice, saying only “Check”. The performance of the posture will be checked by the system, which then communicates the result visually and audibly.

To carry out the process discussed above, the patient is constantly monitored by the Kinect device. Therefore, whenever Kinect detects the coordinates of a skeleton, these are displayed in the virtual environment, allowing the patient to watch their movements in real time. To verify the postures, the customization selected by the physiotherapist is taken into account. In this way, the system uses the error threshold defined in the difficulty level. Furthermore, the system obtains the joints that have been considered in the checking process.

The system guides the patient during the performance of the postures. Therefore, all the postures to be performed are shown as well as the current one. Moreover, the system shows a 3D environment in which the patient’s movement is reflected through the Kinect device. Furthermore, when the posture has been checked, the system informs the patient both visually and audibly. The visual form will display a message and update the scoreboard. The audible form can be configured as a simple confirmation or error sound, or otherwise as a full voice message.

3.3 Exercise design

The aim of this article is to show the design of rehabilitation postures in a three-dimensional virtual environment that can also make use of movement-based interaction. This process is performed within the framework of a study in which these postures are used to create complete exercises tables.

In order to create these exercises, the postures are linked together by means of transitions, which, as mentioned above, generate movements between postures in an appropriate way, and in accordance with the physio’s design. Each posture is thus one step in the performance of the exercise which the patient has to perform in their rehabilitation session. Figure 8 shows the organization of postures for the edition of exercises, in which we can also see the preconditions and postconditions that allow the following postures to be reached.

Fig. 8
figure 8

Postures editing

Full size image

A group of exercises makes up an exercise routine. The patients receive the exercise routines from the physiotherapists and can perform them at the rehabilitation center or at home. The movement-based system monitors the movements of the patients at all times in order to guide them through their exercises, indicating what the next step is, how they have to perform it, and whether any error has been detected in the performance of a posture or transition. All of this is recorded, thus providing valuable information to the physiotherapists, allowing them to study the patient’s progress and to adjust the postures and variables (times, difficulty thresholds, etc.) according to their progress.

Figure 9 shows the basic flow followed to define an exercise from postures. This diagram outlines the basic steps in the design of exercises to clarify the process.

Fig. 9
figure 9

Basic flow to create an exercise

Full size image

The proposed editor enables the physiotherapist to generate exercises that the patient can be performing in a matter of minutes, and these exercises can be adapted in accordance with the patient’s progress equally quickly. Previous prototypes for generating specific exercises [7], without the editor, or specific applications for personalized rehabilitation exercises can require around four months of development with two specialized software developers, so also from this perspective we can appreciate the advantages of the proposed system. So, we have progressed from needing expert software developers and a lengthy development time for each exercise, to only needing one professional, but in the field of rehabilitation, to swiftly generate highly personalized exercises.

4 System evaluation

An evaluation of the system was conducted to determine whether the 3D editor was a valid tool for physiotherapists to design and create personalized set of exercises that could be performed afterwards by the patients at home. It is important to note that the other module developed to guide and monitor patients while performing the exercises at home [11] was successfully assessed by five patients who performed exercises created by the physiotherapists with the 3D editor. The assessment results of the module developed for patients were quite good, getting a score of 85.8 out of 100 in the System Usability Scale (SUS) [3]. Our main goal at this point is to assess the efficacy of the 3D editor proposed for physiotherapists to create personalized rehabilitation exercises to be performed by the patients, as well as the acceptance of this system by the specialists in the field. After running the test, we analyzed the outcomes of the evaluation to improve the system and fix the usability problems discovered. It is important to point out that the version of the system described here reflects most of the improvements that this preliminary evaluation led to.

The key aspects considered during the assessment were the following:

  • Functionality. The system provides all the necessary features to successfully meet the objectives for which it was designed.

  • Usefulness. Physiotherapists believe that the system is useful to support rehabilitation therapies.

  • Ease of use. The system has a simple and intuitive interface and is easy to use.

4.1 Method

In this section, we present the evaluation method we used, including the description of participants, the context in which the evaluation was carried out, the tasks to be performed, as well as the place, and the tools used by the evaluators. We also describe the process and the usability metrics applied. Finally, we present the outcomes of the evaluation. We have taken as a reference the international standard ISO/IEC 25062 [13] to assess the Quality in Use of the system, together with the widely used System Usability Scale (SUS) [3]. This test is made up of ten questions related to the system usage, which have to answered indicating the degree of agreement or disagreement in a 5-point Likert scale.

4.1.1 Participants

The evaluation was carried out with five physiotherapists specialized in sports rehabilitation often performed by young people. According to Jakob Nielsen, even five participants are sufficient to detect 85% of the usability problems in a system [19]. Although it may seem a small number of participants, as the main goal was to assess the acceptance of the system by specialists in the field, we designed the evaluation as a cognitive walkthrough, where it is more important to have representative participants than to recruit a many people with similar skills repeating the same tasks. Finally, at the end of the evaluation, the psychotherapists provided important feedback, as experts in the field, in relevant aspects observed during the evaluation and completed the SUS (System Usability Scale) questionnaire to measure the users’ satisfaction.

4.1.2 Context of use

The assessment was performed in the premises of a research institute, in a 7 × 4-meter room in which a 24-in. screen connected to a Kinect device was installed at a height of one meter (Fig. 10).

Fig. 10
figure 10

Evaluation setting

Full size image

Firstly, the physiotherapists interacted with the virtual 3D editor to design a set of exercises from their own daily work notes and perform certain tasks as described in the next section. Then, using the posture checker, they performed each of these postures to test whether the designed postures were properly synchronized with Kinect or not. The evaluation was performed individually, with just one participant at a time. The evaluator was in charge of writing down the comments provided by the participants, promoting the Thinking Aloud technique to gather all types of suggestions from the physiotherapists, as well as helping them in the case of technical problems. We used a stopwatch to calculate the time spent to perform each task, Finally, all the participants completed the SUS questionnaire at the end of the evaluation.

4.1.3 Experimental design

The steps followed in this process are:

  • Prior explanation: Before starting the evaluation, we gave a complete explanation to the participants about the application, explaining all its features, where to find the tool panels via which the tasks can be carried out, etc.

  • Performance of tasks: The users were given a list of the tasks they had to perform. The “retrospective think-aloud” technique was used so that the users could express their opinion about their interaction with the application.

  • Perceived satisfaction questionnaire: After finishing the tasks with the application, each participant was asked to fill in the SUS questionnaire. The purpose of the questionnaire is to obtain further information and opinions, as well as to discover any problems that have gone undetected.

The tasks performed by the physiotherapists during the evaluation were the following:

  • Task 1: Create a set of exercises by designing 6 different postures and 3 transitions. Previsualize the exercise and save it in the system

  • Task 2: Edit a patient. Modify their phone number. Add a diagnosis and assign the exercise previously created to this patient

4.1.4 Usability metrics

As usability metrics used in the evaluation, we applied the following:

  • Effectiveness

    • Completion rate. Percentage of tasks completed with (assisted completion rate) and without (unassisted completion rate) the help of the person in charge of the evaluation.

    • Errors. Actions that do not lead to completing the task or provoke undesired effects.

    • Assistance. The number of times that the participant asks for help while performing the tasks.

  • Efficiency

    • Task time. Amount of time, in minutes and seconds, needed for a user to complete a task.

  • Satisfaction. This metric was measured with the System Usability Scale Test (SUS test) that all participants completed at the end of the assessment session. The questionnaire, as explained before, consists of ten statements that assess various aspects related to the usability of the application [3].

4.2 Assessment results and discussion

The outcomes of the assessment performed provided us with important feedback which allowed us to improve the system. We mainly detected some usability issues and other weaknesses that are reported in this section.

The results of the evaluation with regard to the verification of the postures designed were most useful. All the postures that were designed by means of the editor were later appropriately verified when they were performed in a correct or an incorrect way. Thus, we may assume that after these exhaustive and extensive tests, the postures designed by the physiotherapist through the 3D Virtual editor are well synchronized with the Kinect’s coordination system and are appropriate for physiotherapists. As a matter of fact, they were enthusiastic about the system and its potential uses.

Next, we show the results of the metrics considered in the evaluation. Table 1 shows the outcomes of Task 1 and Table 2 shows the statistical summary of these data.

Table 1 Results of Task 1
Full size table
Table 2 Statistical data for Task 1
Full size table

The assessment results corresponding to Task 2 are shown in Table 3 and Table 4.

Table 3 Results of Task 2
Full size table
Table 4 Statistical data for Task 2
Full size table

At the end of the assessment, we asked the participants to perform a test to assess the acceptance of the system by using the System Usability Scale (SUS) [3]. The results of this test are shown in Fig. 11. SUS scores range from 0 to 100, and we achieved a mean score of 70 out of 100, which indicates that the usability of the system is reasonably good (based on the accustomed average benchmark of 68 out of 100 and good benchmark of 80 out of 100 [4] [17]. Based on these results, we can state that the system was approved of and accepted by the professional physiotherapists who participated in the system evaluation. All in all, they considered the system proposed as a good tool to be used in physical rehabilitation therapies.

Fig. 11
figure 11

Results of SUS test

Full size image

According to the results shown in the previous tables, we can conclude that all participants could completely perform the defined tasks without hardly needing assistance. The number of errors and assistances in both tasks are low.

We can also see a difference in the results of both tasks, being task 2 the easiest one according to the participants. This is logical given that task 1 required to use more tools and participants were less used to work in a 3D space.

The main results obtained for the improvement of the application on the basis of the test were concerned with aspects such as:

  • The visibility of the state that the system is in.

  • The correspondence between the software product and the real world.

  • The user’s control and freedom.

  • The consistency and fulfillment of standards.

  • An interaction based more on recognition than on memory.

  • Flexibility and efficiency of use.

  • Aesthetic and minimalist design.

Some participants reported the need to add helping tips and more feedback messages after performing certain actions.

A number of problems arose while users were designing some postures, which helped us to improve the design with a more reliable system. We would like to highlight some of these problems that we managed to solve. Firstly, although different views were offered to design the postures, the therapists felt that the system lacked the ability to rotate the camera to achieve a certain perspective. This problem was solved by introducing the ability to rotate the camera until the desired view is obtained. Another problem was identified when the user wanted to move all skeletal joints at the same time since there was no tool to provide this function. To solve this problem, two solutions were implemented. On the one hand, we introduced the x, y and z axes in the virtual editor in order to select along which axis the user wants to move the skeleton’s joints. Furthermore, the physiotherapist can move the skeleton by dragging and dropping the joint called “Spine Mid” (Fig. 12).

Fig. 12
figure 12

Spine Mid Joint to move the whole skeleton

Full size image

In conclusions, all physiotherapists who participated in the assessment considered the system as a good tool to create personalized exercises for their patients. We have mainly focused on the 3D editor to design exercises, but the system also includes the management of patients, giving physiotherapists the possibility of assigning the exercises to a particular patient, among other things. This feature was highlighted by the participants, as well as the possibility of previsualizing the whole exercise to check that the exercise has been properly designed.

5 Conclusions and future work

Technological advances and especially the growing number of new interaction devices are allowing the development of novel systems aimed at improving rehabilitation processes, both from the point of view of the physiotherapists and their patients. Most existing systems simply monitor how patients perform rehabilitation exercises and guide them during the session.

The main novelty of the system presented in this paper resides in the fact that it is designed to allow physiotherapists to easily create -without professional assistance- a personalized set of exercises that can be tailored to the patient’s needs, in a multimodal way.

The main goal is to assist the physiotherapist in the process of designing customized postures and exercises with different levels of difficulty for each patient, and to allow them to follow up on the progress of each patient, as the system records the patient’s performance data, and adapt the rehabilitation exercises to the progress made by the patient.

To achieve these goals, we have developed a virtual 3D editor of rehabilitation exercises with a complete set of toolbars and a 3D skeleton whose joints can be manipulated by the physiotherapist to create the desired postures. Additionally, we provide an alternative mechanism to create postures and exercises by using the Kinect camera device through motion and voice interaction. In this way, the physiotherapist can create the exercises by performing them in front of the Kinect camera. Furthermore, the system allows the physiotherapist to follow the patient’s progress as the system incorporates a module for managing statistics about the patient’s performance results.

The system has been implemented as a Web-based application, which offers the possibility of using the system whenever and wherever the user needs it.

The outcomes of the preliminary evaluation performed on the system have confirmed the validity and acceptance of the system by physiotherapists in rehabilitation settings.

As future work, we aim to increase the capabilities of the system by introducing the new Kinect hand tracking tool to allow the design of exercises that are specifically focused on hand-related rehabilitation exercises, which are currently not possible.