Maintaining Mental Wellbeing of Elderly at Home

Abstract

We describe herein the problem of providing the most cost efficient and effective ways of supporting mental wellbeing as well as methods for physical and mental rehabilitation for elderly at home including a recovery from accidents, particularly concentrating on those impacting brain activities, such as aging-related dementia and stroke, illnesses with very high socio-economic influence. Technologies built in several EU funded projects, e.g. FP7-ICT-StrokeBack, FP7-ICT-ARMOR, Artemis-CHIRON, FP7-SEC-AF3 and other ones, are suitable also for other kinds of health issues, such as recovery from injuries, restoring mobility etc. A common part is stimulating engagement through entertainment, rivalry and “real feeling” of gaming environment, motivating compliance with rehabilitation rules. The automated home system combining progress in ICT and applied clinical know-how allows patients, their direct care providers and family to back the effective use of rehabilitation procedures in their familiar home surroundings instead of unfriendly clinical settings. Our system integrates a set of state of art technologies ranging from augmented/virtual reality gaming, merged with immersive user interfaces for providing mixed reality exercise setting, innovative embedded micro sensor devices with improved power autonomy through use of the newest Bluetooth Smart communication transceivers, combined together into a Personal Health Record (PHR) system supporting the delivery of individual, patient-centred e-health services both at home, at hospital or when mobile. The use of mixed-reality systems, merging interactive virtual components with realistic settings of patient’s home, is linked with multi-modal user interfaces stimulating operation of his/her body to accomplish the objective of the exercise, while self-motivation is inspired by rivalry with oneself and other people. This gears to achieving better individual’s contribution to rehabilitation procedure, leading to attaining meaningfully quicker regaining of one’s earlier abilities. The physiological data is combined with and related to the detected body motion sensing using novel feature extraction and classification procedures handled within a wearable unit, to determine the precision of performed workouts. By using physical intervention only when essential, this disregards expensive human involvement and thus significantly decreases related expenses of Public Health Care services.

We start with describing the motivation and needs for such system in Sect. 2, which gives raise to deriving system specifications and architecture as described in Sect. 3. In following Sects. 4 and 5 we described specific technologies developed in our projects, namely Mixed-Reality Rehabilitation Training System integrated into a Personal Health Record (PHR) platform. We then provide the overview of the overall system integrated into a portable unit in Sect. 6, concluding with report on evaluations with users in Sect. 7.

1 Motivation and Objectives

The Public Health Care devotes more than 3% of their whole healthcare spending for dealing with effects of brain associated diseases among diverse countries in Europe and USA. The usual retirement age and life expectation has increased over recent years, whereas risk of evolving mental issues have raised to almost 40–50% of population in and over the retiring age. Investigations show that expenses of long-term care have raised from 13% to 49% of average global expenses over recent years. Hence, creating an effective policy for sustaining mental wellbeing, providing continuing care and rehabilitation approach for people at risk, actively engaging patients in this process, at the same time lowering expensive human involvement turn out to be a matter of urgency.

The pervasiveness of getting old in the European civilizations is projected to lead to enlarged population suffering from mental illnesses, where peak risk of dementia may reach almost 40% of the retired population and a raising risk of strokes for elderly. For instance, [2] forecasts that the total stroke patients in Hessen (Germany); may rise from 20,846 in 2005 to over 35,000 in 2050 that equals a surge of near 70% during the next four decades. The German Aerztezeitung forecasts a growth of more than 2.5 times, meaning a vast burden on expenditure of healthcare organisations. The consequence on healthcare could be even more important as the present tendency shows a ratio of young and healthy peoples to elderly folks also lowers, such that the casual care price could be reducing and thus directly pointing to raised direct healthcare expenses. This could become a weight for economies.

Consequently, there is a serious need for improving medical care, specifically at home, engaging elderly into their care process for accomplishing best result in terms of both medical care and quality of life. Furthermore, the results of individual strokes have significant effect on our society too. The entire expense of stroke care within EU has been estimated over 38 billion euros in 2006. This amount involved healthcare expenses (about 49% of entire cost), production loss due to incapacity and deaths (23% whole cost) and casual care charges (29% of whole cost).

Our work targets both of the mentioned difficulties. The target was to develop a telemedicine system that would provide medical recuperation at home for elderly with negligible human involvement. Using StrokeBack services, elderly could perform therapy at their home, a place where they feel emotionally more comfortable than at clinics. Furthermore, interactions with physiotherapists could be reduced thus reducing costs of medical care. By helping in appropriately performing the physiotherapy with automatic guidance enhanced by suitable medical data and overseen by professionals only when necessary, we intended to stimulate elderly to train more and more efficiently than it is likely nowadays. Henceforth, we aimed to improve the speed of rehabilitation, as well as the quality of life for the elderly, at the same time reducing overall costs of healthcare. The system has been accompanied with a Patient Health Record (PHR), in which exercise parameters and important patient data would be stored. Therefore, the PHR delivers all essential medical info that rehabilitation professionals might require to assess success of the rehabilitation exercises, for example to presume links between chosen trainings and rehabilitation results for various people. Also, to assess a general patient wellbeing. Moreover, PHR is used to offer mid-term feedback to patients, such as speed and effect of therapy related to usual and recent progress, thus maintaining high motivation.

We intended to improve keeping mental fitness and where mental illness develops, speeding up a recovery in case of stroke and reducing progress of age-related mental degradation for elderly living alone. We expected a twofold advantage from using our system. Most people feel emotionally healthier in their familiar home setting rather than in hospices while recovery speed is much faster. Moreover, we intended to exploit an improved enthusiasm of elderly during training with tools resembling game consoles. An ability to perform workouts without supervision by physiotherapists benefits the reduction of medical expenses by lowering costly human interaction time. Nowadays, amount of time needed for conducting occupational (means therapeutic) and physiotherapeutic sessions are constrained by costs for patient’s lodging, transportation for therapists visiting patients’ homes. Our objective is to provide new technological capabilities and service assemblies for enabling elderly to improve their wellbeing through increase of the amount of exercises.

Over recent years game consoles attracted much attention when used for rehabilitation trainings. Articles prove achieving significant improvement in speeding up rehabilitation process and mostly being an incentive for patients [1, 2, 14, 26]. We target mostly elderly people at early phases of cerebral decay, supporting them with initial therapy whereas empowering also more affected ones to benefit from using our specialist care services too. We designed our services with clinical use in mind with aim to be adjustable to ones’ capabilities (patient-oriented). This could be used both by hemiplegic and paretic persons, as well as those using wheelchairs. This way we intended to decrease the time, static therapy and treatment process, allowing elderly to maintain conducting their normal life for as long as possible.

The concept of the described system centres on patients treated as subjects of the recovery process (Fig. 1), based on a proven premise that elderly feel more comfortable at home. It is known that people in general train more and better within stimulating surroundings. A person follows more instructions from a therapist. They can exercise at home and the system oversees their workout, providing feedback in real time if they execute them properly or not. The system saves results and core physiological data, which is then examined by medical professionals to evaluate ones’ progress. The elderly may get immediate response about their individual condition. To guarantee correct guidance, therapists also receive information from the PHR about patient progress to be able to assess the recovery and decide if to use alternative training exercises, being introduced to one’s training regime.

Fig. 1.

The overall architecture of the foreseen therapy system

For remote therapy training at gold standard, that is as good as during face-to-face exercise with therapists, advanced sensor body area networks (BAN) were employed. The BAN worn by elderly supports continuous supervision of elderly’s movement and main body measurements. The objective is to check and save patients’ condition such that they can regularly, train autonomously without instructions from therapists. Using appropriate sensors, we can assume to detect also undesirable extra actions. Elderly may wear sensors during a whole day that permits professionals to relate ones’ daily activities with accurate required patterns defined by trainings. To ease system configuration, we only use and assess self-learning algorithms for analysing exercises, that is the system learns correct actions (patient movements) during training as they are performed under supervision of the training therapist. Such algorithms are based on the record of “correct” body motions performed by therapist during the learning process, while correctness of performing exercises is determined by assessing a match between the patient’s body movements and previously recorded training data.

A typical scenario is when the therapist needs to care for an elderly person once a week just to assess the effectiveness of the therapy, to examine the outcomes of last training using recorded info and to change the regime if required. Additionally, therapists may need to teach new movements and arrange a new training plan. This way, we aim to improve a therapy effectiveness at home. Our objective is also to assess a viability and needs for making use of electronic PHR for recording and documenting one’s condition as well as to remotely follow up on the training, for example by the visiting physician. This contains the data captured during training workouts and during everyday activities. Such an information is kept in database and can then be examined by medical experts. The assessment may be exploited for determining actual effects of personal training. Such an advice may be beneficial for choosing training plans for other persons, for reviewing efficiency of training for given groups of people etc. Physiological data could be helpful in assessing one’s condition and could help in examining chances of future events.

2 Specifications and Architecture

Many articles have been published from user point of view about home-based rehabilitation systems. When building such a system for elderly suffering from age-caused cerebral problems, it is important to consider how to promote them to such target audience, developing and evaluating home training platforms, form of user interface, types of wearable sensors and a feasibility of the home-based therapy technologies. User needs and requirements, sensor accessories and usability constraints have been identified through interviews with target user groups including a total of 16 physiotherapists, 39 patients and 13 care takers. Consultations have been done with new volunteers and some experienced ones, who were already familiar with the system and could suggest new features. The first system prototype was evaluated with 5 therapists and 4 patients who had used the system over the period of two-weeks. Clinical requirements were assessed based on responses to a postal investigation from 28 therapists. The subsequent version has been evaluated on 8 patients. The result of this investigation was that home-based therapy needs to:

• Supplement a therapy – patients did not like it as alternative type of therapy

• not to be focussed on one “type of rehabilitation”

• be adaptable to personal needs as no two people are the same

• be simple and small to use, flexible to fit personal requirements of one’s home

• anticipate problems with memory, awareness, language, understanding info and losses of attention linked with old-age dementia

• provide response on outcome of therapy, even if progress may be slow

The developed system comprises two components:

• 1. “Expert clinical system” installed at the clinic, although it might be portable as well. This might require technicians with expert clinical knowledge to set it up to fit the needs of a particular patient.

• 2. “Home system” significantly less complex and easier to use, permitting to be installed and configured by anyone, such as family or care takers.

Our system was produced and its performance evaluated on the subset of ‘Wolf Motor Function Test’ (WMFT). The actions involved a variety of skill levels and practical activities. The system produced extra impairment information such as: spasticity versus stiffness, motor control, rapidity, smoothness of movements, repeatability, fatigue and endurance, as well as the effort put in. The essential assumption was for the expert system to evolve into an analytic tool for recognition of core impairments, flagged through study of data created during execution of common activities. It incorporates a BAN of wearable physiological and activity checking sensors with objective to gather rich and highly useful data for real time data examination. Real-time advice and power autonomy were not critical at the time. The essential objective of a home-based system was usability as crucial to ensuring user compliance with prescribed therapies. It was anticipated to be as small as possible and employ only the most necessary set of sensors permitting elderly to easily interact with it. Here the power consumption was critical, similarly to real-time data examination and display. It included immersive user interfaces such as Leap Motion [4], MS Kinect [3], EEG from Emotiv [5] and others, integrated with many virtual and augmentation technologies to allow fully immersive gaming experience, through e.g. supported Smart 3D TVs, 3D projectors and AR/VR visors (Fig. 2).

Fig. 2.

Physical composition of a gaming sub-system

There is a general resistance observed among elderly to using technologies on their bodies, to some level accepted when part of the medical process. The highest problem with acceptance is when it comes to visors, especially closed Virtual Reality ones. In many cases users accept such technologies when used of short periods of time and being part of a routine medical examination and/or a test, but they tend to avoid using it alone when performing training at home. On the other hand, open Augmented Reality glasses, though generally more expensive, are more accepted by elderly since they resemble more standard prescription glasses. In such cases acceptance and compliance with prescribed exercise regimes is much higher.

A generic architecture of the rehabilitation system is presented in Fig. 3. It shows a Patient Module installed at home and offering remote physiological supervision of one’s health signs, running therapeutical games and offering a full integration with the online PHR service used as a database for sharing information among the elderly and their leading physician(s).

Fig. 3.

Overall rehabilitation system architecture

The architecture has been aimed to support also use of mobile devices such as tablets, smartphones, portables etc. We developed an inexpensive combined gaming solution suitable both for full-body and near-field trainings. The clinician sub-system offers link to back-office PHR database for continuous supervision of patients’ wellbeing. Current version combined two sections, both including Kinect and Leap Motion depth sensor (refer to Sect. 6 for more details). One set of sensors is integrated into the horizontal table and is aimed for tracking the use of physical objects and another one is built into a vertical section for tracing body movements. Two displays have been used, a horizontal one for physical objects and a vertical one for displaying conventional board games controlled either through Kinect sensor or other user interfaces. A progress of therapy and other relevant physical information, such as audio-visual teleconferencing, are supported as required. Back-office services were open-source based platforms like Open EMR [6] services.

Eventually, all those services have been migrated to intLIFE core PHR service platform from Intracom Telecom. The overall gaming platform used a client-server approach composed of a repository for games and serving them directly from the PHR server, in such a way greatly reducing load on client devices. This permitted us both to execute games on common everyday devices such as Smart TVs or Smartphones, at the same time allowing to maintain the most recent versions of games with no need for updating them on client devices. Still, as with all network connected system, one needs to expect that connection among networked devices may not always be sustained. Therefore, we have anticipated two operational scenarios in our system: one when network is persistent and another one when it is not (Fig. 4). In the former case when the network is continuously available, the server is built into a home gateway, while the client device was a game unit including a game server (repository of games and results for each user) operates remotely from the same physical server as the one hosting PHR services. The home unit did not need to bother about updating games to their latest versions or managing game results. Nevertheless, when network may or may not be available at all times, the game server needs to be hosted locally on a home gateway, together with the Kinect Server. In the first case, the game server can be operated remotely. In the second scenario, the server could be run locally and use games downloaded earlier. Similarly, physical data and game results can be either uploaded on the fly or stored locally and uploaded as soon as the network connectivity is re-established.

Fig. 4.

Online & offline use of Kinect Server for “game” management

An expert system has been designed to be working in a controlled environment under the supervision of a physiotherapist, or other suitably competent clinician, trained in its use. This will normally take place at a medical centre though it might be used in a home environment too if suitable personnel are present. The expert system consists of a range of body-worn wireless sensors (sensor BAN) and a Motion Capture (MoCap) system using depth sensing devices (Kinect or ASUS [7]) deployed on a PC. A choice of sensors includes kinematic sensor units (containing tri-axial accelerometers, tri-axial gyroscopes and tri-axial magnetometers. When combined they give 9 degrees of freedom of measurements). Electromyography (EMG) sensors monitor muscle motion in biceps and triceps, while electrocardiograph (ECG) sensors monitor the heart rate. A kinetic sensor unit are positioned both on forearms just above the wrist and on upper arms just above the elbow as well as on the sternum. The motion capture module is used to trace patients as they complete given tasks and exercises, whereas video records are used as qualitative measures of the progress of patient’s therapy. Suitable ways to secure and achieve repeatability of sensor placement on the body was investigated during research phase as well as possible effects of sensor orientation and misalignment.

Evaluations with expert system starts with a face-to-face discussion of the patient with a clinician. Based on this, clinician can assess patient’s level of impairment, while the patient informs the clinician about their personal expectations from rehabilitation training. Such a process allows the clinician to choose a range of upper limb training tasks and exercises that might be most suitable for a particular patient. Then measurement phase starts with a patient stimulated to complete a set of exercises and training tasks selected from the WMFT database that have been explicitly selected by clinicians to for the patient’s needs and apparent abilities.

A motion capture system keeps a history of patient’s actions during trainings. Via appropriate signal processing, a motion capture system produces temporal and spatial-kinetic data, such as limb placement in space, limb velocity and acceleration, joint angles and time required to complete particular tasks. Hence, motion capture system could be perceived as a ‘standard’ against which the data resulting from other sensors could be compared with. During the training, physiological data from body-worn wireless sensors is transmitted to server PC in real-time and processed in two different ways. Raw sensor data is converted into 3-dimensional spatial info that can be immediately compared against data generated from motion capture unit. With depth-based sensing and taking advantage of extra information from wearable Shimmer accelerometers, body motion sensing accuracy of movements with centimetre accuracies can be practically achieved from a distance of 2–3 m. Subsequent data processing approach includes looking for patterns in the sensor data, which shows a high correlation with expected actions of the upper limb or specific training tasks that are performed. Home-based system uses such a data as templates for determining if such actions are performed by patients as part of their everyday activities. Other features are also used as indexes for assessing progress and effectiveness of the training, e.g. including calculation of energy spending out of EMG data or assessing metabolic rate out of ECG data. A clinician can judge patients’ condition when they perform prescribed training and make professional judgement of the patient state according to the WMFT scoring scheme. Such a score is sent to PHR database. All raw data gathered by body-worn wireless sensors and motion capture unit, including processed data are saved in PHR as well [8, 14].

Subsequent visits to leading physicians may re-assess patient’s performance for the prescribed set of training tasks out of the WMFT, permitting a long-term assessment of the progress of rehabilitation to be determined with clinical reliability, in our case corresponding to measure of progress of rehabilitation as determined by physiotherapists though classical set of training exercises. Since home-based systems are needed to be used without professional involvement, they may be less complex than expert ones and use less sensors. For example, EMG and ECG sensors may not be incorporated into home-based BAN. Ease of use is most important as such a system needs to be run exclusively by patients, with a support of their carers or family members.

3 Mixed-Reality Training System

The core idea of the telemedicine platform for supporting clinical training at home for elderly people with negligible professional support has been supplemented with a Patient Health Record (PHR) platform where training data, vital physiological and personal data of the patients were stored. Thus, the PHR offered required medical and personal info about the patient, which rehabilitators might require for evaluating effectiveness and progress of the training. This means to assess the relation between chosen exercises and speed of rehabilitation for various persons and to determine their overall wellbeing. The PHR could be employed to inform patients about their mid-term performance e.g. her/his, speed of rehabilitation as compared to a clinical one involving visits to physiotherapists, including their progress over last day/weeks, thus keeping patients’ enthusiasm high.

Benefits we expected from our method is twofold, since most people feel emotionally easier training in their familiar environment, this improves and speeds up their rehabilitation. Moreover, focusing on exercising with tools resembling game consoles, we are able to maintain patients’ motivation. A proposed notion puts patients into a centre of the training procedure, exploiting the fact that people feel more comfortable at home. It has been proven that patients exercise more when training is linked with attractive atmospheres [1]. Firstly, elderly may learn physical training exercises from therapists at care centres. Subsequently, patients can train at home while the system monitors their progress and provides real-time response if they perform their exercises correctly or not. Furthermore, recorded results of the training and vital parameters of the patient are readily available to physicians. Such a data may be then analysed for assessing patient’s progress and determine the level of recovery. Patients can also obtain midterm feedback about their individual recovery process. To warrant suitable supervision, therapists get information from the PHR allowing them to assess progress of recovery allowing them to decide if other types of exercises may need to be introduced into the patient’s training schedule.

3.1 Game-Based Training System

Using virtual, augmented and mixed-reality immersive systems for training at home unlocks an attractive path to improving several adverse effects happening due to brain traumas. Such comprise assisting in a recovery of motor skills, limb-eye coordination, orientation in space, daily routines etc. Exercises can vary from simple goal-oriented limb moves intended to achieve a specific goal (e.g. to put a coffee cup on a table), improving decreased motor skills (e.g. simulated driving), and many other ones. In order to boost effectiveness of training exercises, cutting-edge haptic user interfaces have been produced, permitting a direct body stimulus and using physical items inside of the virtual environments, complementing visual stimulus.

Immersive interfaces have quickly found to be attractive for remote home-based training, both performed independently and remotely supervised by therapists. Dependent on physical interfaces, several training approaches are possible. User interfaces like Cyber Glove [10] or Rutgers RMII Master [11] permit a handover of patient’s limb moves to virtual gaming environments. They use pressure-sensing servos, one per finger, integrated with motion sensors. This lets therapists to accomplish a range of motions with variable speed, fractionation (e.g. moving separate fingers) and strength (through pressure sensors). Games consist of two main classes: physical training (e.g. DigiKey, Power Putty) and practical training (e.g. Peg Board where objects of different shapes and sizes need to be fitted into matching holes or Ball Game requiring manipulation of different balls). Computer monitors the progress of exercises and is used for providing a visual feedback. Cyber Gloves have been used by the Rehabilitation Institute of Chicago [8] for evaluating patterns of finger actions when grasping and for assessing a space of movement for diverse circumstances after stroke. Virtual environments are gradually introduced for practical exercises and simulation of natural surroundings, e.g. home, work, etc. Training types can include simple goal-oriented movements [9] for recovering ability to execute everyday activities.

Modern rehabilitation systems, though taking advantage of the latest immersive technologies appear to focus on proprietary and closed range of exercises, missing comprehensively addressing a complete range of disabilities and providing a all-inclusive set of rehabilitation setups. Use of technologies is also limited and fluctuates from one system to another. Though there are systems of using avatars aimed to provide more intuitive feedback, using many complex wearable devices (as in Fig. 2) might become tiring for users and could reduce an effectiveness of the rehabilitation. In our method we provided novel technologies for body motion tracking that take advantage of the information captured by correlating info from wearable sensors with visual feedback that have been recently available commercially, such as MS Kinect [3], Leap Motion [4] user interfaces, and 3D mixed reality visors.

The developed system provides a full 3D visual and physical feedback via Mixed-Reality interface and visor technologies, putting the user into a training space. Since detecting muscle activity may not be achieved without wearable sensors, IHP GmbH has developed a custom embedded lightweight sensor for short-range wireless communication of most common parameters such as EMG, critical medical signs like ECG, Blood Pressure, heart rate etc. This way, rehabilitation exercises became more intuitive by using exercise templates with feedback displaying level of compliance with prescribed exercises. Therapists are able to prescribe exercises as treatments in the EHR/PHR platform, offering means of correlating data with changes of patient’s condition, improving efficiency of patients’ recovery.

3.2 Body Sensing and User Interfaces

We developed an automated way to automatically track the correctness of performed exercises and be able to compare patient’s body movement against correct ones (templates). Most of the existing methods use complex sets of wearable sensors and/or expensive visual monitoring methods. In our project we investigated modern commercial 3D scanning sensors using IR-LED technologies, such as MS Kinect [3] released in version 2 and recently being commonly phased out by Intel’s RealSense depth cameras [30], Prime Sense [31] and Leap Motion [4] devices. For improved accuracy additional embedded micro sensors can be used, such as gyroscopes (often at a price of a need for frequent position and tilt calibration) as well as more common inertial sensors and accelerometers, detecting changes in speed. Many of such sensors are available on the market. Furthermore, brain wave sensing with devices like EPOC EEG U/I from Emotiv, currently used in our system as user interface, although also beneficial for detecting brain problems such as risk of seizures, while generic sensors devices like Shimmer allow deploying a range of other modalities including electromyographic (EMG) ones used for detecting activity of muscles during training. Bearing in mind their very small sizes (often less than 5 × 5 mm in case of gyros and accelerometers, while sensor boards often smaller than 3 × 4 cm) a development of wireless and very low weight wearable sensors is feasible, able to be energy-autonomous by using energy harvesting techniques.

Monitoring activity of muscles poses a problem with sensing since it has been well known since long ago [26] that EMG represents exertion rather than the result. Therefore, it may be unreliable as an indicator of muscle strength when they get fatigued. As a result, measuring the force, alongside the EMG, is a significant advancement in determining the efficiency of rehabilitation plans and may show that not only the fatigue occurs, but also if the origin of the process is central or peripheral [12]. Standard surface EMG sensing needs precise placing of sensors over target muscles, and so placing them in “smart” clothes for home use could make it simpler for patients to use them and avoid wrong placement of electrodes. Electrode arrays are commonly being developed for sensing and processing of EMG signals and can be used to optimise received signals. Various options have been researched to provide adequately reliable, though economic muscle activity sensing as well. We decided therefore to employ EMG sensors on 2R Shimmer device for development purposes, whereby use a purpose devised sensor from IHP GmbH.

Nevertheless, more types of sensors are needed for providing reliable home care for patients apart of the EMG ones. Novel approaches are required to combine construction nodes in a body sensor network. Commercial systems in existence offer basic info about activity, e.g. movement and direction of speed and postures. Offering reliable info about performance, e.g. corresponding to movement and muscle activity during a specific task and sensing abnormalities, anticipated patterns or small changes related to recovery, needs a higher level of complexity of data acquisition, interpretation and processing. The challenge is to devise and build a unified multi-modal system together with high-level analysis algorithms for extracted signal and data optimisation. The Kinect device shows a potential for being employed as a haptic user interface [23]. It has been used in many earlier projects, with Open Source libraries available for various browsers, like Chrome [13], and demos compatible with Windows 7 and higher [14] platforms. Subsequent versions improved compatibility and performance of Kinect drivers on Windows platforms. Even that Kinect itself has been discontinued, its technology has been transferred to such systems like HoloLens, Cortana, Intel RealSense, Windows Hello biometric facial ID system, as well as context-aware user interfaces. Microsoft tends to suggests users to move to Intel Prime Sensor cameras for using their cloud-powered solutions, based on Project Kinect for Azure, which combines the next generation of Microsoft’s category-defining depth sensor with Azure AI services [32].

Existing body-wearable approaches to acquiring physiological measurements are commonly considered quite adequate though they are frequently bulky and awkward to mount, for example electro-goniometers. They may be also expensive to implement, for example the VIACON camera. Capabilities for being used at home is then quite limited. Consequently, we decided to resolve those disadvantages by:

• Expanding the usability of current sensor technologies: for example, by using MEMS accelerometers with wireless capabilities that are readily available on the market for measuring joint angles for upper and lower limbs, thus enabling us to offer low-cost sensors without cabling, optimized with respect to their info-content and spatial position.

• Innovation in sensing procedures thus to lower the number of sensors that needs to be worn on a body, at the same sustaining sufficiently good quality of data received. Since many users have games consoles at home (e.g. Xbox, Nintendo Wii etc.) for family entertainment, they can be also used to host rehabilitation applications. With evolution of home game consoles like Xbox, human body motion will be easily to monitor with low-cost cameras fitted to e.g. Smart TV sets.

• Effortless installation and calibration even by not technically skilled users for home use, making such a solution applicable for home use even for first timers who are care for by untrained family members and/or unskilled caretakers.

• Seamless validation if exercises are done correctly by patients could be based on data captured by Body Area Networks (BAN). Those would be correlated with treatments prescribed for individual medical condition, allowing for determining the effectiveness of patient’s training, if it is negative or positive.

3.3 The Prototype

A prototype system implementation, integrating a range of technologies, such as physiological monitoring with both Shimmer and Ghost sensors, user interfaces for controlling games, not to mention rehabilitation games themselves, built using Unity3D engine. Assembly of individual sub-systems of the “Patients home training place” is shown in Fig. 5, and its placement on a patient in Fig. 6.

Fig. 5.

Integration of the overall “home” system

Fig. 6.

ECG sensing with Shimmer2R (L) and wearable EEG system (R)

The PTZ-camera used for supervision of the patient during exercises has pan-tilt-zoom capabilities. Considering that very intensive training may in some cases of recovering patients increase a risk of causing traumas, such as stroke or epilepsy, for safety reasons we included an EEG Insight sensor from Emotiv, monitoring the brain waves and searching for “flashes” of activity between two brain spheres, being indicated by our involved physiotherapists as signs of increased risk of pre-event condition (Fig. 7). Such a sensor has an extra benefit to be used as a catchy gaming interface, shifting patient’s perception from its intended use as a preventive sensor to enjoying playing games free-hands using the “power of the mind”. Emotiv offers a Unity3D support, not to mention a more powerful INSIGHT [5] EEG sensor with smaller number of detectors than earlier EPOCH one, 5 + 2 compared to 14 + 2.

Fig. 7.

Emotiv Insight EEG U/I (a), brain activity mapping with Emotiv (b)

The prototype version uses the Insight EEG sensor for assessing a relationship between the intention to move e.g. a hand and the actual action, in addition to seeking within brain waves for any indications that might indicate risks of approaching stroke as well as being used as a user interface similar to a “mouse”. Correlating with data obtained from EMG sensors allows to detect instances when patient’s brain correctly sends a signal to e.g. move an hand, which could not be executed because of a broken nerve links.

3.3.1 The “Kinect Server”

The primary user interface for controlling rehabilitation games in our system is MS Kinect. We take advantage of its distance sensing capabilities combined with RGB camera, which have shown to be effective both for full body motion tracking (using its skeleton matching algorithm) and for near-field training e.g. for hands and legs. As Kinect had not been originally indented to be used for close range detection and only parts of the bodies being visible, such a skeleton tracing is not effective enough and therefore we developed a custom algorithm for being able to determine reliably movements of arms, hands and fingers, being also able to separate those from the objects behind. This led to building a “Kinect server”, extending available open source algorithms. The application used Open NI drivers for Linux at the beginning, then transcoded to MS Kinect 2 drivers for Windows operating systems. The “Kinect Server” enables connections remotely to MS Kinect device and then making use of the sensor data from a variety of client devices, which have previously not been supported by the MS Kinect SDK. The Kinect Server prototype used initially Open NI drivers for Xbox, later transcoded to more generic drivers provided by Microsoft in Kinect SDK 1.7 and later versions. The Kinect Server based system is composed out of two sub-modules:

• Server – requires to run on an operating system supported by MS Kinect drivers. Its role is to receive the data from the Kinect sensor and provide it in a suitable form via network to remote client devices.

• Client – can be installed and executed on ANY device and operating system as long as it offers WEB accessibility with support for Java Scripts. This implies that nearly any device able to access the WEB, such as tablets and smartphones, not to mention Smart TVs, and other devices are therefore natively supported and can take advantage of MS Kinect sensor capabilities.

Range of data and info available from Kinect Server by connected clients includes: RGB feeds, depth maps (both natively offered by Kinect sensor) and a list of detected objects (custom data offered by Kinect Server). Custom options are offered, e.g. ability to limit the area in which objects are detected, permitting applications to remove background objects and be able to focus only on objects of interest (e.g. directly in front and/or closest ones to the sensor). As the sever was built as a generic enabler, additional gesture recognition capability has been offered as well. To enable interoperability with various commercial devices and operating systems, we selected Python for developing our “palm_controls” scripts that are able to detect explicit motions and map those to selected keystrokes and mouse “clicks”.

3.3.2 Embedded Kinect Server (EKS)

The disadvantages of the Kinect sensor related to the lack of seamless compatibility with many Operating Systems, vide range of drivers that are often incompatible with one another, need to be connected to two USB ports on different physical controllers, as well as the requirement to be run-on high-performance workstations, made us to investigate different ways to interact with MS Kinect sensors. Initially we aimed to run our Embedded Kinect Server (EKS) on embedded microcomputers, such as the Raspberry PI [28] or similar ones, thus enabling client devices to run games while taking advantage of data received directly from EKS via local wireless network or Ethernet. This allowed us to “break” the restriction of the physical (wired) connection between the Kinect sensor and the game console, thus allowing 3D sensing capabilities to be available on any networked device.

A range of embedded devices have been examined: from Raspberry PI and eBox 3350 [15], to Panda boards (Fig. 8) [16] and a number of other ones. We discovered though our trials that there was an inherent issue with Kinect’s design existing in all versions, starting from Xbox one to later Windows one. The USB ports could not supply sufficient amount of current and so additional power supply was required. Attempts to increase current supplied by USB ports on Raspberry PI devices, using external powered USB hubs, not to mention other work-arounds, they were all unsuccessful. Until recently, the only embedded platform able to stably operate the Kinect sensor (keeping connectivity and be able to run EKS) was Panda Board. It has been successfully used in our tests to execute the Mario Bros game on various Android operated smartphones and Samsung Smart TVs, which could wirelessly connect to the Kinect sensor for controlling the game with body gestures [35].

Fig. 8.

Embedded Kinect Server deployment: Panda board (a) and physical prototype integrated with the MS Kinect for Windows sensor (b)

3.3.3 Rehabilitation Games Using the “Kinect Server”

The core functionalities supported by our Kinect Server enable limiting the field of view, removing background outside pre-defined area, separating and classifying between individual objects etc. This enabled us to build a Kinect-based user interface, where compliant with needs and requirements of physiotherapists, we traded classical keyboard keys and mouse strokes with hand gestures equivalent to arrows up/down/left/right and mouse clicks action by making a fist. This way we were able to develop our game-based training system, the first of its kind, for rehabilitation of stroke patients who had mobility issues after their stroke episodes. We conducted evaluations, first using Mario Bros game controlled entirely using hand movements. Algorithms detecting position of the hand and producing fake key presses were first developed with Matlab and subsequently ported to PERL for distribution together with Kinect server to embedded devices. They assumed that hand(s) were fixed at a known distance from the sensor on a stable support (requirement from physiotherapists) support, thus we knew the common position of the hand with fingers facing the Kinect sensor. This allowed for easy recognition of the direction and movements of fingertips. This way no calibration of the Kinect sensor was ever required, being a commonly known problem for this device. Removal of the surrounding objects was also simplified by ignoring anything more or less distant from the hand and allowing us to focus our attention entirely on monitoring a 3D space from the fingertip to the end of the wrist. The position of the bounding box around the hand allowed detecting changes in hand position, while horizontal and vertical direction to the hand section mostly extruding from the centre of gravity allowed to determine which finger was moved, in which direction and how far. We then extended the algorithm to more elaborated hand gestures and various combinations of movements. A custom delay between “reads” was used to set the detection “speed”. Such a recognition algorithm, enhanced with recognition of full arm movement, could be easily adapted to e.g. sign language recognition. To evaluate such a capability and to allow patients to play with their full body, we have developed a test game mixing real and virtual objects to form a mixed-reality gaming environment. In this game we requested patients to throw a ball made of paper at the imaginary balloons (circles projected on to the wall) as presented in Fig. 9. The Kinect device was easily able to detect the paper ball leaving the hand and hitting a specific area of the wall. This was correlated with projections to determine a collision, which was rewarded with “balloons” being loudly blown into pieces to a great joy of the gamers.

Fig. 9.

Throwing real paper ball at virtual targets

Such games enabled our test patients to rehabilitate their entire body, not only their limbs. It was very entertaining not only for our users, but for their care takers alike. It had significant benefit for rehabilitation, allowing patients to focus on improving movements of their hand and the whole body and forgetting their disabilities at least for some time, resulting in the increased effectiveness of their training.

3.3.4 Full-Body Exercising Through “Avateering”

We subsequently examined various other, more advanced games that could be used for full-body rehabilitation. They were developed using 3D gaming engines and took advantage of the avateering, i.e. featuring transfer of the physical body movements to the virtual character in the game (avatar). Our first approach required “hacks” built by Kinect developer communities to enable embedding games into WEB pages. The most suitable one for our use was ZigFu [17]. Its advantage was that it was compatible with Open NI drivers Unity3D [18] gaming engine.

It was simpler in use than other commercial ones like Brekel [19] or Autodesk Motion Builder [20]. To make games more intuitive and familiar for our users (often elderly and people unfamiliar with computer games) we modelled gaming environments to resemble natural spaces with photorealistic quality [21]. The example presented in Fig. 10 (left) shows a “Virtual Room” built in Unity3D [24], with test subject testing avateering algorithm in Fig. 10 (right).

Fig. 10.

Avateering in a “home” environment

An inherent benefit from using 3D gaming engines such as Unity3D, Cry Engine or Unreal Engine was an opportunity to produce games that could be executed both as stand-alone applications on supported computers and consoles, or embedded into WEB pages and executed using a classical WEB browser. This made it easier to integrate games into the PHR platform, where they were prescribed by physicians and physiotherapists as therapies, distributed and operated using WEB browsers on any networked client device. Games contained embedded versions of custom-built Kinect Server plug-ins. More recently we integrated also the electromyographic (EMG) “Myo” [22] sensor from Thalmic Labs. It allowed us to detect electrical signals on the skin that were caused by movement of the muscles. This offered two main advantages, one to allow physiotherapists to get an indication if control signals from the brain reach the muscles. Furthermore, this offered us additional data for being used as an additional user interface. With provision of various support software by the manufacturer, offering support for programming 3rd-party applications making use of this sensors, with plugin for Unity3D and the application translating various muscle signals to gestures, we were able to map them subsequently to keystrokes and mouse clicks for explicit use in our games. Furthermore, the SDK gives direct access to raw signals from all eight EMG sensors (around a band that can be placed on a forearm or an arm), which enabled us to process raw signals as well in order to improve the accuracy and reliability of our user interface. Using such an approach we adapted the “Amazing Skater” game template from Ace Games [23] that was built in Unity3D and added the EMG as a user interface (screenshots are presented in Fig. 11). This was an incentive-based game combining rehabilitation with entertainment, allowing to avoid using Kinect sensor in favour of a more compact and easier controlled user interface, detecting muscle activities.

Fig. 11.

adapted from a template by Ace Games

The “Skater Game”

The game can be played using either a keyboard or a Myo sensor. The latter is supported through a Unity3D plugin, though it can be also operated using custom StrokeBack application profile via Myo Application Manager.

4 Personal Health Record Systems

The Personal Health Record (PHR) solutions define a manageable, integrated, flexible and expandable system for provision of care management services and management of patient data. According to NAHIT report [36] regarding definition of key technological terms related to e-health, the following definitions apply:

• Electronic Medical Record (EMR): An electronic record of health-related information on an individual that can be created, gathered, managed, and consulted by authorized clinicians and staff within one health care organization.

• Electronic Health Record (EHR): An electronic record of health-related information on an individual that conforms to nationally recognized interoperability standards and that can be created, managed, and consulted by authorized clinicians and staff across more than one health care organization.

• Personal Health Record (PHR): An electronic record of health-related information on an individual that conforms to nationally recognized interoperability standards and that can be drawn from multiple sources while being managed, shared, and controlled by the individual.

The core feature of the PHR, which distinguishes it from the EMR and EHR, is that info contained within it is under the control of the individual. The above definition names such individuals as controllers, but leaves room for other bodies to act in the individual’s interest, having a control over the access to PHR. Such agents may be expressly declared by the individual, though not in all cases. For example, agents acting on behalf of an individual include parents for their children, and later in life, children acting for parents. The individual is distinctively a guardian of information stored or accessible within a PHR, who decides what volumes of information to include, how it is maintained and ordered, and who can read it or “check it out”. Standards and policy will need to determine if and how individuals can delete or modify information in a PHR that originated from an EHR and how these modifications are communicated to other providers with whom the data in the PHR are shared. Having control also means that an individual’s PHR can exist independently of the entity that sponsors it—the PHR is portable. This requirement for portability excludes models in which sponsors such as health insurers or health care providers give individuals access to health-related information that is dependent on the individual remaining with that sponsor.

The long-term goal of a PHR is to be a lifelong resource of pertinent health information for an individual. Thus, it should have both the depth and breadth of information to enable individuals to become more engaged in their own healthcare as they move from being passive recipients to active participants in their personal health management. The health information in a PHR can be drawn from a broad range of possible sources. The sum of these and other inputs is a well-rounded picture comprising clinical information, administrative information, and wellness information for individuals to employ and impart to others at their discretion. Significant sources may include, but are not limited to:

• Individuals—Self-generated information for personal management or information for care providers, including information about allergies, prescribed medications, eating habits, exercise objectives, the progression of an illness or recovery from it, and preferences regarding care in various circumstances.

• Health care providers—Including hospitals, skilled nursing homes, long term care, and other facilities; pharmacies, lab, and facilities reporting test results.

• Health care clinicians—Including physicians, nurses, behavioural health professionals, registered dieticians, chiropractors, and other licensed or certified care providers.

• Medical devices—Instruments, machines and implanted devices monitoring clinical indices, for immediate use as well as for historical purposes.

• Wellness promoters—Entities supplying services or information to generate and maintain good health, such as proactive medicine centres, fitness centres, rehabilitation experts, and complementary/alternative medicine practitioners.

• Health insurers—Information arising from claims for insurance payments, disease management programs recommending certain actions and collecting results, updated information on drugs in a formulary, and other coverage policies specific to an individual.

• Public health—Government health departments, disease surveillance and immunization programs, school-based care providers and social workers, and nongovernmental organizations engaged in health and wellness.

• Research institutions—Information about opportunities to engage in clinical trials and studies, and recently published results of interest to the individual.

The HL7 [8] standard from the HL7 EHR Work Group [9], describes PHR as the patient-centric system that is mostly controlled by the individual and governs the form and technical development of interoperable PHR/EHR systems.

The overarching theme of a PHR-S involves a patient centric tool that is controlled for the most part, by the individual. It should be immediately available electronically, and able to link to other systems, either in a “pull-push” or “push-pull” method. The PHR-S is intended to provide functionality to help an individual maintain a longitudinal view of his or her health history, and may be comprised of information from a plethora of sources—i.e., from providers and health plans, as well as from the individual. Data collected by the system is administrative and/or clinical, and the tool may provide access to a wealth of forms (advance directives) and advice (diet, exercise, disease management). A PHR-S would help the individual collect behavioural health, public health, patient entered and patient accessed data (including medical monitoring devices), medication information, care management plans and the like, and could be connected to providers, laboratories, pharmacies, nursing homes, hospitals and other institutions and clinical resources. At its core, the PHR-S should provide the ability for the individual to capture and maintain demographic, insurance coverage, and provider information. It should also provide the ability to capture health history in the form of a health summary, problems, conditions, symptoms, allergies, medications, laboratory and other test results, immunizations and encounters. Additionally, personal care planning features such as advance directives and care plans should be available. The system must be secure and have appropriate identity and access management capabilities, and use standard nomenclature, coding and data exchange standards for consistency and interoperability. A host of optional features have been addressed over the course of this initiative, including secure messaging, graphing for test results, patient education, guideline-based reminders, appointment scheduling and reminders, drug-drug interactions, formulary management, health care cost comparisons, document storage and clinical trial eligibility. The effective use of a PHR-S is a key point for improving healthcare in terms of self-management, patient-provider communication and quality outcomes.

The PHR provides all necessary functionalities to assist the individual in maintaining a continuous insight into his/her medical history, including info coming from a number of sources. It assists an individual in collecting vital physiological data (e.g. from medical monitoring devices), health info, care management plans and alike, potentially connecting also to providers, labs, pharmacies, nursing homes, clinics and similar organisations and medical resources. The PHR encompasses the whole health history, including health issues, conditions, symptoms, allergies, medicines, lab test results, immunizations and visits. Considering the sensitivity of data on record, such platforms need to be secure and employ sufficient access management, authentication and authorisation mechanisms. There are two distinct dimensions to be highlighted:

• Integrated care management plans or integration with related third-party systems: Current PHR systems do not adequately support this need. Take for example Google Health or Microsoft Health Vault: notwithstanding the fact that these products offer very attractive web interfaces for the patients to edit and store their personal record of health-related information they lack of any functionality related to the lay out and maintenance of a rehabilitation plan. Typically, this need is covered by special purpose software solutions that are entirely clinicians-oriented and do not actively put the patients in the loop (i.e. not Personal Health Systems). In contrast, our approach is patient-centric: the intention of the project is to build a PHR-S that facilitates personal care planning. This does not mean that the clinicians are made subordinates in the rehabilitation process: the proposed PHR-S facilitates their tele-supervision, if granted such a right by the patients.

• Standard nomenclature, coding and data exchange standards for consistency and interoperability: The significance of these parameters is that they ensure human readability and machine processability of health-related data. In HL7, the former is guaranteed by a Clinical Document Architecture (CDA) HL7 standard, while the latter is guaranteed by v2.x or v3 HL7 Messaging standard. HL7 CDA is an XML-based mark-up standard intended to specify the encoding, structure and semantics of clinical documents for exchange. The CDA specifies that the content of the document consist of a mandatory textual part (which ensures human interpretation of the document contents) and optional structured parts (for software processing). The structured part relies on coding systems (such as from SNOMED and LOINC) to represent concepts. The consortium is not aware of any PHR-S that claims full adoption of the HL7 CDA. For example, Google Health supports a subset of CCR [3], a competing standard to CDA, while Microsoft Health Vault claims to support a subset of CCR and CDA [4], but actually only for importing information from other systems, and not for exporting [5]. As for the underlying messaging scheme, to our knowledge, none of these PHR-S claim support of a widely used standard messaging scheme, such as those of HL7.

On top of the above one should add the need for a controllable, integrated, yet fully open ICT solution that ensures the smooth execution of the project trials. The accumulated experience indicates that while a significant amount of yet unmet ICT-related end-user related requirements arise whenever a new medical issue is examined within an R&D project, in most of the cases neither the legacy ICT systems in the trials sites are open and accessible nor the IT personnel easily accepts intervention and links to such systems. From this perspective, by embedding within the project ICT environment a novel, open PHR-S allows StrokeBack to deliver a self-contained ICT solution able to be deployed in both rich and virgin e-Health environments.

4.1 Overview of Personal Health Record (PHR) Systems

There are various open-source as well as commercial implementations of PHR platforms available in the market. One of the most well-known ones is Microsoft HealthVault [27], which offers online service for storing and maintaining various types of health-related information. The HealthVault platform has capabilities of a typical search engine, permitting users to scan specifically for medical information. Patients may store their personal medical records as well as the prescription history, manage their records, upload medical data from such devices as blood pressure monitors, glucose meters, weights, thermometers and then process this and manage this data. The information can be then shared with other types of medical and health management WEB portals and/or immediately with their physicians or general practitioners. The HealthVault includes also a desktop client allowing medical information to be received from various types of personal physiological measuring devices to be then sent to the PHR. From most common open source implementations, Tolven [21] is the most characteristic one that focuses on providing an electronic Personal Health Record (PHR) enabling users to capture and individually share their health information in a secure way. Another open-source PHR solution one is Indivo X [25], which is promoted as Personally-Controlled Health Record (PCHR), a system, which offers means of creating a PHR, with links to other PHRs as well.

4.2 Generic Architecture of the PHR System

The functions are supported by a Personal Health Record (PHR) system enable individuals to manage information about his or her healthcare. They provide direction as to the individual’s ability to interact with a Personal Health Record in such a way to individualize the record and maintain a current and accurate record of his or her healthcare activities. They include activities such as managing wellness, prevention and encounters. Such functions are designed to encourage and allow an individual to participate actively in his/her healthcare and better access the resources that allow for self-education and monitoring [8]. The principal users of these functions are expected to be individuals referenced as account holders; the patient or subject of care and healthcare providers will have access to certain functions to view, update or make corrections to their Personal Health Record. The Account Holder will receive appropriate decision support, as well as support from the PHR-S to enable effective electronic communication between providers, and between the provider and the account holder or account holder’s designated representative.

Intracom Telecom has implemented its own proprietary version of the PHR platform, named intLIFE, which integrates a range of healthcare application, targeting a number of chronic diseases as well as supports generic wellbeing services. Custom-made adaptations were made to ensure safe sharing of users’ medical data with proper procedures, ensuring sufficient level of private data protection. It secures a controlled access to such data via custom authentication ad authorisation mechanisms, thus guaranteeing that all data that may be circulated around, could neither be traced back to nor be used to identify the person from whom such data had been obtained from. In such a way, a health system that is based on Intracom’s PHR services can be used securely and safely for production of clinical models created from massive amounts of raw data from vast number of patients. This permits a more reliable feature-based medical diagnosis for other patients and determining a range of conditions that had not been possible before. In order to safeguard the privacy and security of information stored within the EHR/PHR to essential levels, both the Management subsystem and the Vital Signs Monitoring one are linked together via encrypted interfaces employing authentication, authorisation and data anonymization modules. The PHR developed by Intracom has a modular architecture supporting effortlessly adding and removing any functional sub-systems. This was achieved by separating business layers from underlying technical frameworks. The former one corresponds to end user needs in a given domain, whereby the latter horizontal layers offer generic, business agnostic functionalities (Fig. 12).

Fig. 12.

PHR-S platform architecture

The Middleware Framework provides a flexible and configurable link between internal and external modules. The capability maintained by this layer is associated with API management, rights management, user interfaces and the presentation layer. The Presentation Framework provides auxiliary application guidance and access from various WEB browsers. The foundation for the design of this layer was ensuring independence of the graphical user interface from the business domain.

In order to facilitate the independent development of applications and services by third parties that can be easily integrated and glued with the core PHR system, we will follow a modular programming approach. The advantage of this approach is that it provides us with great flexibility in assigning discrete functions to each of the modules and allows easy integration through appropriate interfaces. The PHR includes a set of core modules that are necessary independently of the value-added services that are provided to end users. Such core modules include:

• External adapter controller module responsible for directing calls to appropriate ESPs components; it also includes libraries for translation to HL7 messages, so as to support standard based communication with external systems.

• User management module providing functionalities related to the administration and management of the users who have access to the PHR-S and the various applications and services built on top of it. It includes aspects related to access rules and user roles. Appropriate interfaces of the User Management Module enable access by third party applications and facilitate a centralized approach in users management.

• Permission management module enabling account holders (health record owners or their designees) to manage access rights (permissions) to their PHR. and controls access to protected resources.

• Audit management module enabling logging of information related to the transactions being executed in the PHR-S. Such transactions may include log in attempts, modifications of permissions, modification of PHR content, etc. It logs timestamps and respective user who is involved for each transaction.

On top of the above, one can easily build independent business services around the core PHR-S. The exact services to be built will be based on the outcome of the end user requirements collection and elicitation process. Nevertheless, we can rather safely present in this paragraph some indicative services that are in line with the general objectives of the project as apposed in the Description of Work:

• General Health Record Service providing patients with access to their medical data (general demographics, family history data, vaccinations, allergies, etc.).

• General Health Record Service Extensions providing extra functionality to the patients, correlated with disease-specific aspects. Indicative such extensions provide functionality related to the management of post-incident health interventions, diabetes, COPD, nutrition and general lifestyle, etc.

• Exercise Guidance Service guiding patients while executing a predefined set of activities and/or physical exercises –possibly set up by the clinician using the Care Management Service Module see below-. It may provide direct –online- feedback to the subject, during the execution of the activities or offline feedback, after the end of an activity session.

• Care Management Service providing clinicians with the necessary functionality in order to set up and monitor plans related to the management of the disease of their patient. This module has different extensions, depending on the disease: e.g. it is different for stroke patients, diabetics, COPDs, etc.

• Videoconference Service enabling patients and clinicians to keep undergo live videoconference sessions. Service configuration parameters are dedicated to defining who and how often can establish such a session. This feature addresses the need described by the clinicians to be able to put relatively strict rules as to how often their patients may call them in the course of the day.

• Quality of Life Surveys Service: This service enables the patient to participate in questionnaire-based surveys related to her quality of life.

5 Overall System Integration

The prototype architecture compliant with expressed user requirements, led to te selection of relevant use cases and a range of components needed for each of those use cases. Moreover, features of the “look and feel” for the developed system and the deployment at patient’s homes and within their life has been built into the design. As a consequence of additional research on the minimum space between patient and distance sensors, the first prototype of the home unit for patients’ use has been created. This server as a reference for matching advances in other subsystems.

Various versions of training games were tested and have been enhanced following a feedback from physiotherapists and end users. Similar approach has been followed in the development of PHR services, “Exercise and Compensation Analyse (ECA)” tool as well as user interfaces embedded into the mobile unit. Regarding events envisaged to be used for using rehabilitation games, an event capture algorithm has been created using close-range distance sensors, including Kinect and Leap Motion, which concentrated on movements of wrists and palms, such as open and close actions for the latter ones, both being indicated by clinicians as most important for the effective rehabilitation of post-stroke patients. After putting together all components into the integrated system, combining range/distance sensors with ECA tool for body motion capture, the performance of the close-range Kinect devices reduced significantly when both were used at the same time. Therefore, we decided to make all components to use data from just one Kinect device, integrating it as far-field sensor while applying Leap-Motion one for detection of near-field motions and interactions with games.

The experimental assessment of the integrated Mobile Patient Unit and its corresponding training regimes started from the review of the overall physical design with vertical and horizontal displays. The first version of the system was bulky and weighty; therefore, a second smaller variant was built (Fig. 13). The former one has been designed with clinical (stationary) use in mind, while the latter one offers few innovative technical enhancements. Specifically, the close-range detection with Kinect sensor was replaced with the Leap Motion one built into the base of the unit, to take over the tracking and recognition of hand gestures. In this way, the overall size of the mobile unit could be lowered by 30 to 40% with weight 50% lower.

Fig. 13.

Prototypes of the mobile patient station for home use.

6 Deployment and Validation

A dedicated pilot study has been organized in Germany in order to validate the usability and practicality of the developed PHR system and the Care Management services. The system has been installed at the Brandenburg Klinik Berlin-Brandenburg [29] and evaluated with a control group of 25 patients. The aims of those evaluations were to check if the developed framework was “usable and useful”, and if the implemented Care Management services are beneficial for the physiotherapists in conducting their rehabilitation work and help patients in achieving improvement of their motor functions. Project participants were all involved in performing both the technical validation of the system functionalities and the initial assessments with end users. We followed a specific process to validate the rehabilitation system:

• Users switching on the rehabilitation unit.

• Users launching “Tele-rehabilitation” process with the “touch table”.

• Users selecting “autonomous training” that is the training mode when exercises are performed without direct involvement of physiotherapists. The “auto training” mode was selected automatically in case of no connection to server.

• User selected a game and executed it, custom selection depending on earlier scores and adjusted a level of difficulty. In case that exercises included background music, user could select a favourite one. The whole selection process had been configured earlier by physiotherapists who decoded which options were to be made available to patients.

• Users started the exercise in an autonomous mode. The PHR physiological monitoring sub-system assessed and analysed user operation and generated feedback. In the end, results and scores achieved were sent to the PHR.

• Users run exercises with online supervision by physiotherapists. They were constantly supervised, could see their clinicians via teleconferencing and could instantly receive instructions how to correctly perform their exercises.

• Users could check their scores and evaluation results after completing them.

Evaluation results confirmed our earlier expectations that Kinect could not be used reliably at close range and for tracking movement of upper limbs, especially when it concerned movements of fingers and even hand gestures. It also required calibration every so often. On the other hand, the Leap Motion sensor has proven more beneficial in such situations, providing a reliable and precise tracking of hands and fingers. Based on such results, the final (commercial) solution combining advantage of both sensors and certainly reducing their respective disadvantages, could be produced, in which Kinect was placed in vertical section of the mobile unit to monitor full body motion and interaction with the surrounding environment, while the Leap Motion being installed in the flat part of the unit was used for short range interactions with games using hand gestures and interaction with virtual objects using fingers, whose movements were transferred via avateering to virtual hands.

We integrated a teleconferencing functionality into our system to evaluate benefits of teletherapeutic interventions. In most cases, young users could achieve better results, whereby interactions with aphasic users has proven more complex, in some cases impossible, with neuropsychological issues causing a barrier. Specifically, complicated verbal directions were often misinterpreted by users. In this context the “Selective Repetitive Movements” and “Arm Ability Training” types of exercises have shown best user compliance, which could be explained by exercises being easier to demonstrate visually. Therapists used the following procedures:

• 1. Restricting rehabilitation games to upper limbs

• 2. Employing evidence-based interventions during exercises

• 3. Focusing on motor functions, cognitive skills and fast system response

The produced three types of tests based on earlier defined technical specs:

• “Selective Repetitive Movements”, a set involving joints and movements of upper limbs with main attention to hands, especially distal movements. We postulate that such exercises could be readily linked with mixed-reality games. Using such gaming methodology helps in overcoming the problem of high number of monotonous repetitions, thus improving users’ motivation, compliance and attention on correctly performing their rehabilitation exercises.

• “Arm Ability Training”, a set where we promote use of everyday objects, where we see a potential to motivate our users. Specifically, we address here problems with users who lacking everyday exercises as part of their daily routines. Therefore, interventions considered here required focussed attention not only to motor capabilities, but also to cognitive and sensitivity ones.

• “Music Supported Training”, asset not needing any previous musical tutoring. It uses music as means of improving user motivation. Various musical devices may be used, whereby for simplicity of use we chose a synthesiser keyboard.

The three above mentioned types of training procedures made it possible to combine patients with different impairments into the same intervention group and apply the same types of telerehabilitation exercises. This also provide sufficient variability in case that some users might chose other rehabilitation exercises, while ensuring maintaining his/her motivation and compliance. We provide a full set of guidelines to users to ensure that they can follow correctly the rehabilitation training routines.

In order to speed up rehabilitation speed, exercises require investigation from point of view of: visual and interaction attractiveness, effective operation of the exercises and probably the most difficult one to achieve, the real time assessment of the correctness of performing exercises by end users. Constraints to be taken into consideration are mainly how well patients perform their rehabilitation and how much improvement is achieved, physiological data taken both during exercises and in daily living activities, cognitive abilities and so on. The analysis we do follows clinical models relating wellbeing with physiological data, based on the results from Virtual Physiological Human (VPH) project [33], to which we have added new variables corresponding to the improvement of physical and cognitive capabilities based on clinical experiences and used as indicators for remote rehabilitation, periodically verified in clinical environments by medical professionals. An extensive historical overview of physiological models can be found in [34].

Since using only few classical rehabilitation exercises was not suitable from therapeutic perspective because of the complex nature and varying types of impairments faced by individuals, we opted for creating a system where it could be individually configured and geared to fit specific clinical rehabilitation needs of the patients. The system had to be able to “learn” new training types in real-time, meaning adaptation to changing movements of the patient, thus allowing the system to assess body motion with respect to pre-defined exercises at a later time.

We appreciate the fact that patients may not always perform in the similar manner and movements may not be repeated exactly as they had been recorded. Furthermore, assessment and measuring of the position of joints using Kinect exhibits certain level of inaccuracies. Therefore, the system needs to be adjustable to user’s rehabilitation conditions and level of progress, including custom precision of detecting movements. The latest prototype calculates a deviation from the reference set of body motions as a variation in the off-axis angles for every vector associated with a specific part of the body, e.g. upper or lower limbs. The acceptable deviation from the reference movements in the pre-recorded training may be customised, before launching the supervisory application. The latter one was enhanced with a support for overseeing several exercises simultaneously. Each of them may be adjusted to provide a pre-defined type of feedback when performing correctly the exercise.

6.1 Execution of Rehabilitation Studies

To ensure high reliability of study results, we strictly followed a certain set of procedures and had provided guides to patients and physiotherapists before starting the evaluations. They comprised relevant info for patients and “informed consent forms” outlining the scope of the tests and requesting permission to e.g. capture and use personal data for the sole purpose of the tests. This has followed a face-to-face interview after which users were requested to sign the “informed consent forms” before being able to take part in the evaluation tests. Following the tests, a separate set of questionnaires was filled for each patient to individually assess their performance during the studies. Their responses provided a base for assessing the results of the rehabilitation from the perspective of achieved outcomes, usability and technical practicability of our proposed method.

6.2 Evaluation of Pilot Results

The evaluations were combined with a number of classical rehabilitation exercises and means to judge the effectiveness of the newly adopted procedures. Each user from the group of 25 patients engaged in the evaluations took part in the trials for a period of six weeks. Over this period, evaluations have been performed 4 times every two weeks throughout the whole evaluation period. We employed the WMFT as standard means of evaluating patients’ physical capabilities. The WMFT process that we used consisted out of 17 smaller tasks that each patient was expected to execute, while the physiotherapist rated their performance in a scale from 0, corresponding to very bad achievements, to 5, representing normal scores, towards perfect achievement. The aim of the task was to place nine (9) little timber slats into corresponding holes on a flat board. Then they were to place them back at their original places. The goal was for patient to achieve this task as quickly as they could.

The second test has been performed when seated and its purpose was to move dices from one box to another one over a small obstruction wall within one minute, from right to left and back again. The score was calculated as the number of dices moved during the time limit. The Barthel Index was used to assess patient’s motor skills and as a result also the effectiveness of the rehabilitation. This is a kind of analysis using an observer judging and providing a score for a trainee, in this case by the physiotherapist or a clinician. It offers a way to evaluate patient capabilities associated with daily activities. It includes such common tasks as going to toilet, eating and drinking, movability, personal hygiene, dressing and undressing, incontinence and ability to climb staircases. The tests of this kind were assessed in scale between 0 corresponding to a person completely independent to 10 representing worst case scenario of a person totally dependent of care from third persons. The total of the scores from all tests provided the final evaluation results and the actual condition of the person. The rating has been performed by interviewer by filling a specific questionnaire. The questionnaire was filled by the patient who was expected to answer 49 questions about their everyday activities, personal characteristics, family relations, speech abilities, movability, social activities etc. Every question was scored from 0 to 5, corresponding to bad and normal respectively. The higher the score was, the better the effectiveness of the rehabilitation was.

7 Conclusions

The development of the Electronic Health Record (EHR) clinical systems and the Personal Health Record (PHR) ones were the outcome of the intrinsic problem of the medical professionals in effectively managing the increasing amount of paper archives and all types of printed medical records. The absence of unified way of transferring information among electronic medical systems made the situation even more difficult. The introduction of the HL7 standard [8, 9] provided reference rules and procedures as a means to resolve this situation, though since the beginning the HL7 was treated only as guidelines and not a factual standard of rules to be strictly followed. This caused electronic systems to be developed and used that employed only different subsets of the HL7 specification [9] that suited the given medical service provider and not the complete standard. Transferring information among systems build in such a way proved to be difficult, causing much information to be misinterpreted or made incomplete. This issue has been identified early as a critical one for future wide-spread of electronic communication among clinical systems. Now the HL7 is considered more as a factual base for providing interoperability criteria for smooth operation of medical systems. However, since it still misses device level interoperability, more work is still needed for ensuring seamless mobile physiological monitoring with links to diverse clinical systems. In our projects, the work progressed towards merging both areas of electronic health record interoperability with device level certification, which has resulted in creation of the proprietary implementation of the PHR core services by Intracom, the intLIFE core platform, where core interoperability components are fully compatible with HL7 specification, with remote monitoring ensured to be performed with devices certified for use for critical medical applications. The intLIFE platform embeds purpose defined device profiles allowing linking any certified medical device with its electronic repository of medical measurements and clinical information.

The formal technical validation tests confirmed the usability of the developed system. It proves to be beneficial to clinicians for acquiring and storage of physiological information about their patients, not to mention integration of various applications and services for processing and assessment of such a data. We specifically validated the usability of user interfaces such as Leap Motion operating gaming application, especially for those using immersive and natural interactions within virtual and mixed reality 3D environments. After the successful technical validation tests, the evaluations with real users, patients and clinicians have been conducted. They concentrated on body motion capture and acquiring movements of the real person (physiotherapist) to be used later for demonstrating the correct way of performing exercises by avateering as presented in Fig. 10. Those were integrated into the complete system along with server offering custom selection of games, treated as therapies prescribed by clinicians in the same way that medicines are. Links with PHR online platform provided a clinical base for managing both the game selections, personal and clinical patient data as well as games scores and physiological data obtained from medical devices. It also offered means for correlating all this data to produce the best overall view of the current condition of the patient, both physical and mental.

Evaluations with real users allowed to refine all aspects of therapeutic interventions. Although physiotherapists may still strongly depend on classical occupational therapies and focussing on personal rehabilitation, they commonly agree with the added benefits from employing the automatic clinical assessment of subjcts’ motor skills and progress of recovery compliant with WMFT, as well as computer-based occupational exercises with both real and physical objects in mixed-reality environments. Apart from classical exercises, everyday activities can be effortlessly supervised using Body Area Networks (BAN) of sensors throughout the day with proper consideration for rights to privacy of the individuals, by restricting direct access to certain private raw data by physicians and care takers, instead offering processed diagnosis and relevant physiological data only. Data gathered from long-term monitoring of basic movements in semi-natural scenarios (over three weeks period) has been used to improve the accuracy of ADL recognition algorithms. They were employed as basis for developing novel application-specific integrated circuits (ASIC) used for automated detection of various types of movements. Individualized training and novel compensational analysis components have also been integrated into a single Exercise Evaluation Tool (EET). The EET offers benefits for exercising e.g. upper limbs with recorded movement traces, which patients are expected to follow. The EET can automatically analyse the correctness of such exercises. The previous version has been extended by fine-tuning for individual sections of the limbs (e.g. shoulder, upper and lower arm, etc.). For examples we can determine how much the patient’s movement varies from the correct ones. Similar features were also implemented for detecting compensational moves during exercises, such as if patient remains standing during exercises with specific objects.

In conclusion we could notice significant changes among medical communities fter the introduction of electronic health records, enhancing the efficiency of medical services at a lower cost, while continuing to offer a wide range of research challenges, which we expect to be pursued and likely resolved in the near future.