1 Introduction

Therapy is prescribed to elderly population to move certain body joints and muscles through some targeted body motions. This helps in gaining enough muscle power to support daily life activities and keep one healthy. The status of muscle power is monitored by looking at the joint range of motion that is affected by the surrounding muscles [1]. Typical therapy modules targeting elderly people involve activities and gestures to monitor the whole body joint movements. In order to precisely find out the relative joint movement of the affected body, therapists either resort to very complex invasive skeletal devices that are impractical for elderly people or very much expensive to use it for general purpose in-home. For example, therapists use Goniometer to measure the angular displacement of an elbow joint manually during the elbow flexion/extension therapy, which is not scalable and/or practical for many elderly people. For example, in order to track 20 joints of a body, a therapist needs to attach 20 sensors or manual devices in each joint. Moreover, there are complex movements of the gestures that the therapists recommend to build and regain the affected joints and muscles that require high level therapist interventions. Examples of motions that are involved during existing therapies include Abduction/Adduction, Flexion/Extension, Pronation/Supination, Inversion/Eversion, Hyperextension, Dorsiflexion/Plantar flexion, Rotation/Circumduction, Protraction/Retraction, and Elevation/Depression [2]. Existing invasive devices fail to dynamically track above motions of an elderly person and produce live kinematic data that can be used to gain deep knowledge about the status of each joint and associated muscle of affected region.

The non-invasive motion tracking capability could enhance the therapeutic diagnostics to the next level [3]. Therapists could assess the performance of the joints needing improvement. Hence, a therapist could provide feedback during or after a therapy session in a personalized fashion. Although sophisticated therapy facilities are available in medical institutions, in-home therapy is gaining popularity [4]. Therapy in the home is more flexible and convenient for an elderly person by allowing more frequent repetition of therapy exercises with the aid of caregiver family members. In order to keep the pace of improvement, therapists suggest that therapy modules be repeated the required number of times every day. While therapy sessions in front of a therapist are a necessary, a therapist cannot take care of a large number of elderly person, given the fact that our elderly population is very high [5]. Hence, a therapist cannot fulfill the required frequency of practice sessions necessary to increase the quality of improvement, making in-home exercises to complement this goal.

Although sophisticated therapy facilities are available in medical institutions, in-home therapy is gaining popularity [6]. Performing therapy in the home is more suitable for an old age person and their caregiver family members due to movement problems as well as it allows more frequent repetition of therapy exercises with the assistance of caregiver family members. To help during in-home therapy, various off the shelf gesture computing platform such as Microsoft Kinect, LEAP motion, MYO, have recently emerged that help in identifying physiological and gait parameters from a therapy session in real-time [7]. However, to the best of our knowledge, there is no gesture-based multi-sensory solution for cyber-physical therapy system, which supports in-home therapy. In this paper, we present a cyber-physical therapy system, which is based on client-server big data cloud architecture [8]. In the CPS framework, a user having old age motor disability can use multimedia gesture tracking sensors in the physical world and a middleware software as cyber world running on her PC/smartphone and in the cloud [9]. A therapist can suggest certain exercises to an elderly person, which she can perform at home [10]. The therapy CPS framework allows an exercise session to be captured, stored in Amazon Cloud Computing Cyber Environment and analyzed for deducing improvement factors. Moreover, a number of serious games have been designed to fully immerse a user in the cyber world while the multi-sensory physical world environment collects full body range of motion kinematic data [11].

The rest of the paper is organized as follows. Section 2 describes the CPS framework design. Section 3 outlines the implementation technologies while Sect. 4 concludes the paper.

2 Framework Design

The framework comprises of a physical world consisting of elderly people and gesture tracking sensors to support therapies at home, and a middleware interfacing the cloud environment acting as cyber world. The middleware has two main components, one residing at the client side and the other at the cloud environment. The client side is responsible to interact with the multi-sensory gesture data, process the gesture data, map with the therapeutic information, show live feedback and finally send the therapy data to the server side cloud environment for further processing and report generation. We now explain different framework components in details.

2.1 Motivating Scenario

Before we go into the framework design, let us assume a scenario where each elderly person having diversified in-home therapeutic needs has to be dealt separately. We assume a disability care hospital named PhysioPlus, which treats patients who have different levels of disability due to their old age. The hospital specializes in both intra-user and inter-user therapy management, both in-center and remote in-home care. In the former case, intra-user therapy is defined as follows: at the time of admission, an elderly person exhibits certain therapeutic needs in different joints of his/her body. After he/she is exposed to different therapies, the range of motion (ROM) [12] of the person is targeted to improve as close to normal. This development is said to be in-home intra-user therapy management. In the case of inter-user therapy management, it refers to how one elderly person’s disability level is different from others. Since every elderly person has different physiological developments, we define a scenario for the in-home intra-user therapy management needs, which can be extended to inter-user scenarios as well.

We assume an elderly person named Alice suffers from sudden numbness in her left side. Her caregiver family members took her to the nearest hospital. After initial assessment, the doctors classified her sufferings as left Hemiplegic [11] and suggested her caregiver to take her to the Hemiplegia therapist at disability care hospital PhysioPlus to discover details about the type of Hemiplegia and make arrangements for needed therapy so that she can regain her physical strength in the affected body parts. At her first visit to the assigned therapist, the therapist requested Alice to make some gestures and motions, to see the current state of joints and muscles. Based on these initial physical movements, the therapist rates the severity level of Alice’s Hemiplegia and chooses different therapy modules for Alice. The physical condition of Alice is such that she cannot wear complex gloves or wearable devices, rather a non-invasive way of tracking her movements is highly desirable. At her every visit to the therapist, Alice’s kinematic data [13] needs to be recorded in her medical record so that the therapist can monitor the quality of improvement of the targeted muscles and joints that are affected.

Since most of the time Alice is at home due to her old age, her therapist wants that Alice carry on the therapy exercises, with the help of her caregiver family members. In order to assist Alice’s caregiver family member, the therapist has defined some ideal therapy sessions and shared them with Alice such that she can do those exercises in a game-like environment with virtually guided. The therapist also wants that Alice conduct the required numbers of therapy sessions at home and that the results be available to him for review. The therapist wants to keep track of certain quality of improvement metrics that signifies that right therapy module is given to Alice and that she is improving at the desired pace. The therapist is interested to observe statistical data regarding quality of improvement in a weekly, monthly and yearly time period. While Alice is progressing, the therapist wants to increase the complexity and difficulty level to make Alice close to her normal counterpart. To be aligned with the distributed nature of the therapy scenario, the health and session data of Alice needs to be stored in an online repository. What’s more, the therapist wants a visual feedback regarding the exercise sessions to make comments so that Alice can view those comments and act accordingly. Since Alice is old, the therapist wants to provide video and audio feedback on the actual therapy session so that Alice can play the multimedia feedback and make any necessary adjustments to her exercises.

From the above scenario and suggestion from other researchers [14], we can conclude that an elderly person such as Alice is recommended to be tracked by a multimedia gesture tracking in-home environment that can track the necessary joints, muscles, and actions and produce necessary Kinematic metrics that helps in decision making of a therapist. Next we present details about therapy and how it can be tracked by gesture tracking sensors.

2.2 Set Up of Therapy Sensing Environment at the In-Home Physical World

The physical world consists of elderly people and a set of surrounding movement tracking sensors. Figure 1 shows a person within an in-home therapy supported environment. The environment consists of three gesture tracking sensors, Kinect2, MYO armband and LEAP motion, each having different field of view. The physical world also consists of a visualization panel that will allow the therapy data available from the cyber world. Figure 2 shows each of the gesture tracking sensors and the types of joints that can be tracked by each of them.

Fig. 1.
figure 1

In-home therapy management environmental setup at the physical world

Full size image
Fig. 2.
figure 2

Joint tracking capability of (a) Kinect, (b) LEAP, and (c) MYO armband

Full size image

Kinect2 is specialized in tracking the whole body joints [15] (see Fig. 2(a)) while LEAP can only track hand joints [16]. Although Kinect2 can track whole body joints, including the hand joints, but it fails to provide sophisticated PIP, MCP and DIP joint range of motion, which can be obtained from LEAP sensor (see Fig. 2(b)). On the other hand, Kinect2 has greater range of spatial tracking capability while LEAP motion has less coverage (see Fig. 1). MYO has a unique feature such that it can be interfaced with a smartphone through Bluetooth Low Energy protocol, which makes its coverage greater than both Kinect2 and LEAP sensors. Another advantage of MYO is that it does not required line of sight contact with the joint, which allows avoiding occlusion. Once a MYO is worn by an elderly person, it can ubiquitously monitor the wearer’s gestures. A great advantage of MYO and LEAP is that these not only support angular gestures such as flexion and extension (see Fig. 3), these also support rotational gestures such as pronation and supination (see Fig. 3(a)). Hence, combining these three within one framework gives a very rich source of gesture tracking capability. Moreover, all the three sensors can be bought off the shelf by anyone, which makes the in-home therapy monitoring a reality.

Fig. 3.
figure 3

(a–c) Primitive joint motions and (b) breakdown of a high-level therapy in terms of primitive joints and motions around wrist, elbow and forearm movements.

Full size image

2.3 In-Home Therapy Monitoring

Each joint of a human body can produce a fixed subset of primitive motions as allowed by the human body anatomy. Figure 3 shows a number of primitive angular and rotational motions around different joints of human body. Each high-level therapy can be expressed as a set of primitive joint-motions. For example, the therapy “Medial Epicondylitis” can be broken down into the following primitive joint-actions (see Fig. 3(d)) performed in the following order:

  1. 1.

    Wrist FLEXED

  2. 2.

    Forearm PRONATED (palm surface facing downward)

  3. 3.

    Elbow at 90 DEGREE

  4. 4.

    Elbow EXTENDED

  5. 5.

    Wrist EXTENDED

  6. 6.

    Forearm SUPINATED (palm facing upward)

Hence, by individually tracking each of the above primitive joints and then ordering these temporally by the cyber system, the framework can infer the high level therapy. The framework can engage the right sensor for tracking the relevant joints needed to be tracked in a therapy monitoring process. For example, in case of the above therapy, the wrist joint motions can be tracked by LEAP sensor, the forearm motions can be tracked by MYO and the elbow can be tracked by the Kinect2 sensor. Similarly, any therapy can be broken down into a set of primitive gestures and each joint-motion can be assigned to be tracked by a unique gesture tracking sensor.

2.4 Guided Therapy Monitoring

Performing a therapy itself is a boring task, let alone doing at home and in the absence of a therapist. To address this concern, we have designed a model therapist as cyber world entity, which can act as a guide to an elderly person. The idea is that at the onset of the therapy, a virtual therapist will appear at the therapy recording window and guide an elderly person step by step. A scenario of model therapist is shown in Fig. 4. As shown in the top left figure, an elderly person has to first align him/herself with a virtual augmented reality stick diagram to start the therapy. Once the subject moves to the right position, the gesture tracking sensor Kinect2 senses the position and changes the state of the virtual therapist with right posture that needs to be followed by the elderly person. Once the subject’s body posture matches with that of the avatar, the avatar shows the next step to follow. This process continues until the therapy is completed, as prescribed by the therapist. This guidance will help any elderly person to follow any prescribed therapy with right order and required number of frequencies. As shown in Fig. 4, while the subject follows the avatar movements, the kinematic therapeutic data is stored for live feedback as well as report generation.

Fig. 4.
figure 4

A model therapist in the cyber world guides a subject in the physical world

Full size image

2.5 Middleware Design as Cyber World

Framework Cyber World Architecture. We leverage our existing big data architecture detailed in [8]. As shown in the Fig. 5, API server handles the multimedia gesture data traffic which is available from various sensory media sources from the physical world as shown in Fig. 1. We have used the Amazon services as Infrastructure as a Service (IaaS). The Amazon EC2 instances allow quickly scaling capacity both up and down as the computing requirements changes, which is a fundamental case for our application due to variable usage. The API server is deployed on the EC2 instances where scaling and load balancing is configured by creating AWS Auto scaling Group, which is monitored and triggered when the average utilization of EC2 is high or low. This is done using the Cloud Watch for scaling activities. Elastic Load Balancing is used to distribute traffic to instances within Auto scaling Group to get the optimum utilization of the resources and cost.

Fig. 5.
figure 5

Multimedia Big Data at the Cyber world

Full size image

Any newly arrived therapy session file metadata is processed in the In-Memory Database and the complete payload is stored in DynamoDB after pre-processing. To handle the multiple media writes per second in DynamoDB, we use AWS SQS for live data that is directly stored in the queue and offline data is first uploaded to AWS S3 and then processed respectively using AWS Elastic Beanstalk with the support of AWS Auto scaling and AWS Cloud Watch. The Queries and Visualization components receive a therapist or a patient’s request regarding quality of improvement metrics from the server side session data and accordingly interact with analytical layer which processes and returns the desired session statistics.

Serious Game as Therapy for Immersiveness. Serious games offer a way of alleviating the pain and uneasiness of performing therapies at home [17]. While a user plays therapy games, the framework tracks kinematic data and stores in the server for detailed analysis. A sample therapy game that is designed as part of this research framework is shown in Fig. 6. Ladder Man is a simple maze like game that consists of animated main character, the Ladder Man, with some visual goals. It also includes animated model therapy avatar or skeleton that will guide the elderly person step by step. The person’s wrist flexion and extension makes the Ladder man run or walk to a goal based on the range of motion of flexion or extension. The game environment is illustrated in the following figure and illustrated as follows:

Fig. 6.
figure 6

Serious games to assist an elderly person in performing a prescribed therapy (a) ladder man game (b) shoulder movement game.

Full size image

  • Elderly person opens the game

  • An introduction video will show how the game should be played

  • The person will continue to next window

  • Calibration screen will let patient know if the tracking sensor is calibrated and the target joint is in the optimal range for detection

  • Person will continue to next window.

  • A counter will count back to zero from 10, allowing the person to be ready.

  • The game starts. Model therapy avatar/skeleton will indicate or guide the first gestures to be performed.

  • The person performs the gestures

  • Model therapy avatar/skeleton will indicate or guide the next gestures to be performed and so on.

  • Once the therapy session requirement is met, the game will be complete and scores will be displayed

  • The recorded session containing kinematic data will be saved and uploaded to server

  • The person can preview the completed session or play the game again.

3 Implementation

The physical world gesture tracking sensors have been implemented using Kinect2, LEAP and MYO sensors. We have combined these sensors through our custom built websocket API. The sockets read frames from the sensors as they appear as stream. The web application server has been implemented using Laravel while the client side consists of HTML5 with AngularJS and three.js webGL libraries for rendering the gesture 2D/3D animations. We have implemented the cloud environment as shown in Fig. 5 using Amazon web services platform. The relational database has been implemented using PostGreSQL database environment. The framework has been tested with 25 different elderly subjects and found to be engaging themselves while providing in-home therapy data to the therapist online.

4 Conclusion

In this paper we have presented our ongoing research work targeting in-home therapy support for elderly persons. Since it is difficult to follow the prescribe therapies at home without the direct guidance of a therapist, the cyber physical system provides a model virtual therapist who will guide an elderly person while he/she performs a therapy at home. The game-like therapy environment will make sure that the therapy is performed in the correct manner and the required number of times per day. The CPS also records the kinematic data and other improvement metrics while the therapy session takes place at home. The cyber world embeds necessary analytics engine to produce reports showing different range of motion data containing improvement of an elderly person temporally.