Keywords

1 Introduction

Clinicians often rely on self-reporting of patients for the interval between the last hospital-visit to the present hospital-visit, for the assessment of performance of the patient to track the neurological disease progression. The self-reporting assessment system has been criticized by many clinicians and is prone to discrepancies in many areas of healthcare system. In neurological disorders such as Multiple Sclerosis (MS) and fronto-temporal dementia (FTD) personality changes occur e.g. dramatic increase in social submissiveness and introversion. There are no studies which can quantify the physical activity changes at home for such FTD or MS population. It is of paramount importance to clinicians to objectively quantify the behavioural features (over an extended period of activity in a home situation) for the accurate diagnosis of the disease. Moreover, the movement assessment of patients living with neurodegenerative disorders such as MS is an important factor for clinicians to monitor disease progression and respond for timely intervention. A low-cost, unobtrusive sensorized system capable of autonomously detecting patients behavioral state at home could help significantly improve accuracy of assessments and hence improve quality of care for these patients. In order to achieve this goal we need to develop a system which can relate the in-home physical activity to activity within the clinical test.

Wearable devices unarguably can detect the patient’s living state at home, these include pedometers, specialized accelerometers, etc. [7]. However, the biggest disadvantage of wearable devices is compliance with the need to wear or carry them. There are also issues with collecting continuous sets of data as well as battery life. Environmentally mounted sensors that make passive observations of patients while at home overcome many of these deficits. Several studies have discussed the non-wearable preference of older adults [4]. Numerous researchers have looked at the use of environmentally mounted sensors, such as vibration sensors mounted on the floor [16], infrared passive sensors [15], acoustic sensors [10], and video streaming (an intrusion into privacy), including traditional cameras [9] and thermal imaging sensors [12]. Studies have found that privacy concerns of older adults to vision-based monitoring systems may be addressed by the use of appropriate privacy preserving processing techniques, such as silhouettes [3].

Early reports suggest the Kinect can identify pose [1, 6, 11], simple stepping movements [5] and postural control [2] in healthy adults, although some have raised concerns about the accuracy of the skeleton model estimation during unconventional body postures or when using wheelchairs or walkers [8, 13]. There is also growing evidence for the use of exercise-based computer games (exergames) to retrain motor function in people with Parkinson’s disease (PD) [14], although evidence of their safety and efficacy are yet to be established [11]. Exergaming as a therapeutic tool that incorporates functional, purposeful and engaging exercise in a quantifiable and reliable way that also encourages high volumes of practice and potentially improved motivation and adherence [12]. A player’s movement can be recorded whilst playing a game using the Kinect, allowing clinicians to ensure the patient perform exercises correctly.

In this research paper, we demonstrate proof of concept of obtaining early stage experimental results of physical activity obtained in a domestic setting. We demonstrate an unobtrusive continuous gait monitoring system based on the Microsoft Kinect sensor that could measure the gait of older adults, in their homes, during normal daily activity. However, the output of the system, measures of in-home gait speed, stride time, stride length, etc., is not easily interpretable, as such parameters have never before been available. Thus, either a large scale study to directly relate these in-home gait parameters to health status is needed, or a methodology to relate the parameters to existing well studied and understood domains needs to be developed. For this work, the Kinect-based in-home gait system was deployed in a residential setting. While the systems were installed, the participants also completed preliminary walking assessment consisting of standard mobility test, such as a short maximum speed walk (SMSW). The SMSW test has been widely studied and shown to be a good measure of functional ability as well as an indicator of falls risk in the elderly [7, 15]. As such, mapping of the residential gait data to this well understood domain would facilitate interpretation of the data by a clinician.

This paper presents a methodology for and results from estimating walk time from in-home gait data, specifically walking speed, collected by the Kinect-based systems. The purpose is not to measure SMSW time directly, but to map the in-home gait data to a domain that clinicians understand; and to assess the accuracy of the mapping with regards to what could theoretically be expected. Although this paper focuses on mapping in-home gait speed to SMSW time, the approach could be used with any data source, and any well studied domain. Section 2 of this paper discusses the materials and methodology used in the area of in-home gait and functional ability assessment. Section 3 contains the results of estimating SMSW time from in-home gait speed and compares it to estimates made using SMSW measured by a clinician at the same time. Finally, a discussion of the results and their implications is given in Sect. 4.

2 Materials and Methods

2.1 Ethics Statement

No video recordings during any task were made in this study. The Kinect sensor’s numerical data was recorded that directly related to walking movements. The numerical data was de-identified, representing joint positions, were stored in the database for further analysis. Volunteers were researchers at the University of Salford, Manchester and Salford Royal NHS Foundation Trust.

2.2 Second Generation Microsoft Kinect V2 Based Sensing

The Microsoft Kinect is a camera-based sensor primarily used to directly control computer games through body movement. The Kinect tracks the position of the limbs and body without the need for handheld controllers or force platforms. Use of a depth sensor also allows the Kinect to capture three-dimensional movement patterns. We propose that this system has the potential for remote assessment of movement symptoms in people with neurological disorders, exemplified by MS and FTD symptoms.

Fig. 1.
figure 1

Planar view of Microsoft Kinect - (a) The 3D planar view associated with Microsoft Kinect. (b) Skeletal tracking and joints in x,y - plane.

Full size image

2.3 Data Collection

The data acquired during subject monitoring included 3-D body joint information provided by Microsoft Kinect V2 at a rate of 30 frames per second. The data acquisition was supported by an in-house developed application written in MS Visual studio 2015 C++ using the Kinect SDK v2.0. Microsoft Kinect sensor with the use of its software development kit (SDK) provides three-dimensional skeletal data on 26 joint positions over time. We chose to record the positions of four skeletal joints namely, spine-base, hip, knee and ankle center of gravity or leg movements, which are potential walking velocity indicators. With 7 joints (spine-base, left and right joints for hip, knee and ankle) and 3 floating point values (real numbers) representing the x, y, and z positions for each joint, each motion frame was expressed as a 21-element position vector. The 3D position (x, y, z) of a joint is expressed in the position coordinate system of the Kinect sensor and the units are in meters (as shown in Fig. 1). The walking speed rely on motion dynamics so our system is view-independent, i.e., there is no need to express the recorded positions in the coordinate system of the subject’s body. The dynamic of each joint is computed using the variation of position of the joint over time. In the first step, each joint motion, defined by the sequence of 3D positions, is replaced by the distance between each frame particular joint that varies from \(0 \ldots k\), where k is the number of frames in a performed motion. The instantaneous velocity of motion for a particular joint is calculated as the resultant of x, y, and z positions over all frames that represent a motion. The instantaneous velocity \(V_{inst}\) for a given 3D motion is computed as follows:

$$\begin{aligned} V_{inst} = \frac{d(x, y, z)}{dt} \end{aligned}$$
(1)
$$\begin{aligned} V_{inst} = \frac{1}{T}\sqrt{(x(k) - x(k - 1))^2 + (y(k) - y(k - 1))^2 + (z(k) - z(k - 1))^2} \end{aligned}$$
(2)

where T is the sampling interval and equals the reciprocal of the sampling frequency (1/30 = 0.0333) s and k is the number of joint data points. The subject’s kinematic properties can be extracted using the value of the joint positions, and if required joint angles, angular velocities and accelerations can be calculated based on the time history of the joint positions. Moreover, to remove possible artefacts in the joint position data, a 2D moving average filter was used to smooth the data (as shown in Fig. 2).

Fig. 2.
figure 2

Plot of correlation spine-base position data during walk; used to identify when steps occur. A moving average smoothed filter output is also shown in black. (Color figure online)

Full size image

2.4 Kinect Based Short Maximum Speed Walk (SMSW) Assessment

The short maximum speed walk (SMSW) assessment at Salford Royal NHS Foundation Trust (hospital), is usually carried out in a narrow corridor. The Kinect sensor was mounted on 2.5 m height tripod at the end of the same corridor of the hospital, thereby covering a rectangular area of roughly 3.5 \(\times \) 2.5 m (see Fig. 3(a)). Each subject was tested in an evenly lit environment in a single session of sequential tests as shown in the Fig. 3(a) and (b). All subjects were given the same instruction as specified in a standardized test procedure: “Walk as fast as you can towards the sensor”. The starting point for walking was approximately three meters outside the detection range of the sensor. We postulated that this would allow the subject to reach maximum walking speed before reaching the measurement zone. The start was given as a voice command. An automatic computer-generated sound signalled the subject to end the experiment after leaving the opposite edge of the sensor measurement zone. Similarly, the same software was used to calculate the time taken from the initial point of start to the end point 25-foot mark. The time taken to complete a timed 25-feet walk (T25-FW) gives a quantitative mobility and leg function measure, based on a 25 feet walking distance.

Fig. 3.
figure 3

Schematic system setup (in xy-plane). (a) The Kinect sensor was positioned 2.5 m above ground. Subjects walked with maximum speed towards the Kinect camera. (b) Sample screenshot of a healthy subject during the test with skeleton projection (green lines). The spine-base joint was used as the data source for analysis. (Color figure online)

Full size image

2.5 Residential Autonomous Sensing System

A Microsoft Kinect sensor and computer were deployed in a residential apartment as part of a volunteer study at researchers’ house. The Kinect was installed on a 2.5 m height tripod, which can easily be placed on any shelf a few inches below the ceiling (height 2.5 m), above the front door. This arrangement has proven to be unobtrusive to the residents, with most indicating that they do not notice the equipment after a short period of time. The output of the Kinect based residential autonomous systems was a dataset in which each entry corresponds to a joint position of the skeletal movement that occurred in the apartment (Fig. 4).

Fig. 4.
figure 4

Kinect sensor installed in a residential setting. (a) The background is subtracted while depth silhouettes and skeletons are determined by the Kinect SDK. (b) Infra-red and depth sensing from the similar image.

Full size image
Table 1. Mean RMSE (meters), Correlation (r) between the joint positions during assessment at hospital and in-home monitoring system.
Full size table
Fig. 5.
figure 5

Joint positions and moving average filtered output. (a) Left and right hip joint position and filtered output. (b) Left and right ankle joint position and filtered output. (Color figure online)

Full size image
Fig. 6.
figure 6

Root mean squared error and correlation between the walking speeds, obtained during clinical assessment at hospital and at residential setting. (Color figure online)

Full size image

3 Results

The participants performed a set of walking movements which involved spine base, hips, knee and ankle joints. Figures 2 and 5 represent the time series of spine base, hip and ankle positions (which compound the walking movement) and the root mean squared errors (RMSE) are shown in Fig. 6. Joint movements are reported in Figs. 2 and 5 represents the evolution of the walking movement where as the continuous curve is the filtered output of the each joint position. The results in Figs. 2 and 5 show the walking movement from one of the experiments using the Kinect based system during the assessment at the hospital. A correlation coefficient r and RMSE was calculated between the joint (spine-base, hip and ankle) positions obtained during the clinical assessment and the in-house monitoring as tabulated in Table 1. On average, the RMSE was less than 0.012 for all joint positions and the correlation coefficient varied between \(0.90 \le r \ge 0.95\). Correlations between time series of walking assessment at Table 1. Mean RMSE (meters), Correlation (r) between the joint positions during assessment at hospital and in-home monitoring system as tabulated in Table 1. The results obtained correspond to what we expected. Walking readings at assessment and in-home monitoring are determined by joint positions activity, which do not necessarily occur simultaneously.

4 Conclusions

In this an early stage of proof of concept study, we investigated the applicability and feasibility of Microsoft Kinect v2 assisted motion analysis during the hospital walking gait assessments and in-home monitoring. Our primary question was whether skeletal tracking data recorded by the Kinect based system would yield reliable information for clinicians to assess the walking gait from in-home monitoring system. Using the spine-base, hip and ankle joints, we established the test SMSW to analyse person’s gait at maximum walking speed. The linear distance covered by the recognition area was only 3.5 m (see Fig. 1 and Table 1), and consequently only a few steps of each subject were analysed. Despite this short walking and recording time, the overall detection quality of the target hip-centre joint of the SMSW over time was excellent.

The noise from the Microsoft Kinect based system was filtered out by applying a custom-built moving average filter developed during the analysis of this experiment. After filtering the SMSW average walking speed parameter was excellent and on par with T25FW. An analysis of skeletal data from spine-base, hip and ankle joints was performed for the walking assessment in healthy subjects. Our findings show that Kinect-based motion analysis is also feasible in Multiple Sclerosis (MS) patients and can detect gait differences in comparison to healthy controls.