Keywords

1 Introduction

Exoskeletons are wearable devices supporting tasks commonly found in industrial applications [1]. They modify the wearer’s internal load distribution by means of active [2] or passive [5] elements, or by a combination of both [7]. Aiming for the defined reduction of joint torques and forces, the question arises how to reliably determine these quantities. Joint forces and torque are defined by the sum of the individual muscle forces and their levers acted upon segments and cannot be measured in vivo.

Our proposed method to determine and manipulate internal load distributions with the aid of exoskeletons is to replace the human wearing an exoskeleton with a mechanical mock-up of a simplified shoulder-arm-system, restricted to planar movement. Additionally, the affect of the exoskeleton is simulated with a collaborative robot, mechanically coupled to the upper arm of the analog simulator, imposing forces for establishing a control goal, like constant shoulder torque over time while carrying out a pre-defined task, see also Fig. 1. The advantage of this approach is to enable for direct readings of the torque and forces acting on the shoulders via appropriate sensors, and by measuring motor currents. The cobot, used as a substitute for the actual exoskeleton, allows for distinguished control inputs, thus simulating support of the exoskeleton.

Fig. 1
figure 1

Different layers of simulation. (1): Base case of a test person (a) wearing an exoskeleton (c). (2): The wearable is replaced and simulated by a collaborative, serial kinematic robot (b). (3): The shoulder-arm-system is mimicked by a servo drive with double pendulum attached (d)

Full size image

2 Related Work

Lower-dimensional surrogate models for the prediction and feature extraction of human motion data is an actively researched area where principal component analysis, neural networks, and statistical methods are among the most popular [6, 8, 9]. Gaussian process latent variable models (GPLVM), also considered as probabilistic nonlinear PCA, has been used by Marin [13] to create a low-dimensional surrogate model embedded in an optimization problem to minimize ergonomic scores of drilling tasks. DMD-based methods have not yet established in analyzing and predicting human motion. The work of Enes [12] isolates the reason for this, and introduces delay-embedded DMD algorithm to remedy issues associated with the drawbacks of exact DMD. Patil [4] fused LiDAR and inertial measurement unit (IMU) sensor data to track human motion data in real-time. In [25], a motion-controlled mechanical mock-up of the shoulder joint is introduced, exhibiting a rotational degree of freedom of the scapula.

Table 1 Components of the mechanical mock-up
Full size table

3 Mechanical Mock-up

The mechanical mock-up of the shoulder-arm system comprises of a gearless servo drive, a double pendulum attached to its shaft, and sensors to account for force and angular readings. Upper arm and forearm are made of milled aluminum parts, and reflect mass and dimensions of its human counterpart. At hand position, additional mass may be mounted for different load scenarios. Rotary encoders for absolute angular measurements are integrated into the servo drive, and mounted to the (elbow) joint connecting upper arm and forearm. The muscles are modeled as McKibben fluidic muscles, i.e. fiber-reinforced elastomers contracting when pneumatically pressurized [17]. The muscles’ insertion points are at 50 mm from elbow joint center for the biceps, and 25 mm from the elbow joint center for the triceps. Table 1 lists used components and its specifications. For the mechanical simulation of the impact of an exoskeleton, a collaborative serial kinematic robot is used. It introduces pressure force via a link to the shoulder-arm-system teststand (Fig. 2).

‘

4 Model

The mathematical model that describes the behavior of the planar shoulder-arm-system is a Langrangian of 1st kind description of a double pendulum with lumped masses, as depicted in Fig. 3.

Fig. 2
figure 2

Mechanical mock-up of the shoulder-arm-system comprising of servo drive (a), upper arm (b), forearm (c), biceps (f), triceps (g), rotary encoder elbow joint (e), force sensor mounted in base plate (h), and additional mass at hand position (d)

Full size image
Fig. 3
figure 3

Double pendulum schematic and characteristics for modeling according to Lagrangian of 1\(^{st}\) kind. \((x_1, z_1)\) is position of lumped upper arm mass, \((x_2, z_2)\) is position of lumped fore arm mass, \(\varphi _1, \varphi _2\) are respective angles enclosed with the z axis. \(M_1\) is torque introduced by shoulder servo drive, \(M_2\) is torque introduced by biceps/triceps pair about elbow joint, \(M_{d1}, M_{d2}\) are respective damping torques, proportional to angular frequency. \(F_{sup}\) denotes the support vector imposed by the exoskeleton, g is gravitational acceleration, and \(m_1, m_2, l_1, l_2\) are lumped masses and length of limbs, respectively

Full size image

The governing equations are [22]:

$$\begin{aligned} m_i \ddot{x}_i = \sum _j^{n_F} F_j + \sum _a^{n_c} \lambda _a \frac{\partial f_{a}}{\partial x_i}, \qquad i = 1..2N, \end{aligned}$$
(1)

where m denotes mass, x is a cartesian coordinate, F is applied force (gravitation, actuation, damping, support). \(n_F\) is the number of forces, \(n_c\) is the number of holonomic constraints, \(\lambda \) is Lagrange multiplier, f is holonomic constraint, N is the number of mass points.

For the integration of the differential equations we are using an implicit Runge-Kutta method which has proven to be numerically more stable than explicit schemes. The chosen parameters are \(m_1 = 3\) kg, \(m_2 = 2\) kg, \(l_1 = l_2 = 0.3\) m, \(g = 10\) m/s\(^2\), \(d_1 = d_2 = 0.4\).

The model is used for testing control and identification algorithms before deploying the code on the test stand, and to get a qualitative understanding of the underlying dynamics and characteristics of the system.

5 Surrogate Model

Surrogate models are small scale approximations of full-scale descriptions of system dynamics. Their main purpose is to adequately estimate and predict the motion in phase space, usually in a given subset of possible states, limiting the application range and accuracy of the surrogate model.

In this article, we advocate the use of linear regression techniques, particularly the Hankel Alternative View of Koopman (HAVOK) [18], for two major reasons. Firstly, it preserves the physical meaning of the states, rendering the computational overhead of an observer obsolete. Secondly, the obtained linear discrete time model integrates very well into the model predictive controller framework. Due to the linearity of the surrogate model it is computationally feasible, and embeddable [16], even for optimization-based control strategies, as MPC is.

The HAVOK method for deriving linear surrogate models of nonlinear systems on basis of measurement data is, in its foundations, a time-delay embedding with a Koopman-theory-motivated linear propagation of singular right eigenvectors over discrete time, closely related to the Eigensystem realization algorithm (ERA) [20], or the more recent dynamic mode decomposition with delay (DMDd) [3] (Fig. 4).

Fig. 4
figure 4

The method Hankel Alternativ View of Koopman (HAVOK) [18] for creating linear surrogate models from measurements of nonlinear systems. (a) Time-shifted measurements are stacked into a Hankel matrix H, and decomposed into its left singular eigenvectors U, right singular eigenvectors V, and singular values S. (b) Only the first r right singular eigenvectors \(\tilde{V}\), corresponding to the largest singular values, are stored, the remaining vectors are discarded. (c) Dynamic mode decomposition, a linear regression technique, is applied to truncated versions of \(\tilde{V}\), denoted X and \(X'\) (d) The best-fit matrix \(\Xi \) propagates the right eigenvector \(\tilde{v}_k\) one time step. \( (e)\) From the singular value decomposition of the Hankel matrix we have \(v = S^{-1}U^H\). (f) The closed-form solution for the propagation of physical states in a time window of length r can be explicitely stated as a linear mapping of the truncated versions of U, S, and the best-fit matrix \(\Xi \)

Full size image
Fig. 5
figure 5

Cascaded strategy of cobot trajectory control: In an open loop system identification process (blue), the plant follows a given, periodic motion pattern. This movement is perturbed by force signals imposed on the plant, and the resulting shoulder torque is read. From the i/o data, a surrogate model is derived, using linear regression techniques, like Hankel Alternative View of Koopman (HAVOK), Eigensystem Realization Algorithm (ERA), and Subspace Identification (SSI). This model (A, B) then forms the basis for a model predictive control algorithm to close the loop of measured state \(x_k\), and computed input \(u_k\)

Full size image

6 Control

Figure 5 shows the schematic of how to arrive at a linear surrogate model-based controller of the identified i/o behavior of an exoskeleton’s support vector to shoulder torque. The procedure is divided into an open loop and a closed loop branch. The open loop is really about system identification. While doing a trajectory-tracking controlled predefined task, i.e. keeping the hand position of the mechanical mock-up on a motion path, we impose force perturbations to the upper arm, and read the resulting shoulder torque. This input/output mapping will subsequently be used for a linear-regression-based method to create a small linear surrogate model suitable for real-time control. For the closed loop branch, we have chosen a model predictive control (MPC) strategy as it seamlessly integrates the discrete-time linear model description obtained from the system identification part. Despite its optimization-based nature, and therefore computationally expensive, it is still applicable for real-time control due to the linear description of the model, and performant algorithms optimized for embedded systems [16]. In contrast to frequency-domain methods, MPC control goals can be formulated explicitely as cost functions and state constraints on physical values. Additionally, the discrete time setting aligns well with cycle times used in threads of programmable logic controllers.

7 LiDAR Sensors

For measuring the planar movement of the shoulder-arm-system, actuated by a serial kinematics robot, a LiDAR multi-sensor system, specifically developed for the task of tracking human motion, is applied. It basically consists of eight Intel RealSense L515 time-of-flight sensors 30 Hz frame rate, a depth accuracy of approx. 5 mm, and an integrated RGB camera for color information. The sensors are spatially distributed to capture the scene from different angles with their individual point clouds registered into an integrated scan based on an extrinsic calibration in a postprocessing step. Wiring and components are depicted in Fig. 6. Challenging tasks are sensor placement for a trade-off between minimizing occlusion effects due to shadowing, and minimization of interference between individual sensors as a side effect of their active measurement principle. To account for the interference, the sensors are triggered with temporal delays. The main advantage, and inherent characteristic, of a LiDAR measurement system is its ability to collect surface information of the captured object, and therefore contribute greatly to the classification, and identification of movement patterns.

Fig. 6
figure 6

Wiring and components of LiDAR multi-sensor system

Full size image

8 Results and Discussion

The double pendulum system described in Sect. 4 was stabilized with an LQR controller by linearizing about an operating point with torques \(M_1 = -1.2\) Nm, \(M_2 = 2.3\) Nm, representing the lower hand position of the trajectory of the task of picking up workpiece, and mounting it overhead. Figure 7 shows the damping effect of the controller when opposed to inputs introduced by the supporting structure. The input signal is a normalized measurement of an XSENSOR pressure mat, located at the load introduction area of the exoskeleton’s arm shell, recorded over a full motion path when carrying out the task of picking up workpiece, and mounting it overhead, and integrated over the area [24]. The controlled shoulder-arm complex serves as a model for the real behavior of a human arm when exposed to external disturbances.

Fig. 7
figure 7

Simulation of forearm x position and phase plot of forearm (x, z) position. Blue represents free dynamics of the forearm when opposed to small signal support inputs, orange shows the oscillation-attenuated forearm dynamics using LQR control

Full size image

We were planning to apply the HAVOK with control (HAVOKc) method, described in Sect. 5, to create a linear surrogate model for mapping the exoskeleton force input to shoulder torque, but up until now we were not successful implementing it. Python code for model, input data, controller, and attempt for HAVOKc are available at https://bitbucket.org/maxherrmann/havok.

For the LiDAR system, the current state of development allows for capturing point clouds with four sensors measuring simultaneously. Figure 8 shows a sequence of images taken of a person taking a seat in a chair. The accuracy of the system has not yet been evaluated.

Fig. 8
figure 8

Point cloud sequence of person taking a seat recorded by four mutually registered LiDAR sensors

Full size image

9 Summary and Outlook

A mechanical twin of the human shoulder-arm-system coupled to a serial kinematic roboter introducing pressure force into the upper arm to support lifting set out to answer the question “Can shoulder torque of a mechanical mock-up be controlled with an appropriately chosen support vector over time based on data-driven linear surrogate models and a LiDAR motion capture system?”. The mechanical twin is a cybernetic arm, an analog simulator, equipped with rotatory and translatory actuators, and designed with equal dimensions and mass distribution of a human arm, mimicking its motion. Thus, enabling for the real-time reading of torque in the shoulder and elbow joints, and reaction forces sensed with distinguished force sensors integrated in the fixed bearings of the motor. The collaborative robot simulates the impact of the exoskeleton on the upper arm via a mechanical coupling for pressure force transduction.

A concept of how a HAVOKc-based system identification can be carried out while the simulated shoulder-arm-system is moving on a trajectory-controlled periodic path is outlined. The resulting transfer function from introduced load at the upper arm to shoulder torque is obtained as a linear surrogate model. The surrogate model evaluates faster than the full model while preserving the dominant characteristics, and can thus be incorporated into a trajectory-tracking controller.

Simulations were carried out validating the mechanical model, and testing the performance of a linear-quadratic controller that stabilizes the hand position. For small signal perturbations imposed by a load vector acting upon the forearm, the resulting oscillations observed at the uncontrolled arm were effecitvely attenuated.

We are introducing a novel multi-sensor LiDAR system, merging individual sensor measurements into an integral point cloud by mutually registering the data sets. In a subsequent post processing step, features, i.e. segment positions and orientations, are extracted and used as reference signals for downstream control loops.

All the mentioned teststands, sensors, and algorithms are still in the development phase such that this article sketches an outline and a concept of the investigations to come.

In future research, the mechanical mock-up can be replaced by a mapping from kinematics to kinetics. This is usually accomplished by introducing human motion data to a musculo-skeletal model, and, by means of inverse kinematics and inverse dynamics, compute the internal load state of a human [10, 11]. Since this approach is computationally expensive and infeasible for real-time control, surrogate models might as well be a suitable measure for addressing this problem.