Keywords

1 Introduction

Skin is the thin layer of tissue forming the natural outer covering of the body. It is a protective barrier against mechanical, thermal and physical injury and hazardous substances. It also acts as a sensory organ (touch, temperature, strain, moisture). The electronic skin (e-skin) and its applications in many domains have attracted researchers worldwide for over three decades [8]. E-skin as an interface that mimics a biological tissue is being developed in medicine (smart health care [18], prosthesis [20]), and robotics [3]). In robotics, it allows for increasing safety in human-machine cooperation, the agility of robotic manipulation [19], and soft robotics [2, 4].

To calibrate and establish the measurement characteristics of e-skin sensors, a standard approach base on the measurement of the force exerted by a reference device. In the known research, linear, quadratic, and cubic polynomial models were fitted to data to calibrate the example sensor [5, 9, 21]. The best results were obtained using a Huber regression with a quadratic polynomial model. In [15] e-skin semi-automatised calibration procedure using industrial robot FANUC LRMate 200iC equipped with a reference force sensor was presented where second-order exponential function and logistic function were used. To assess fitting results, two parameters were analysed: adjusted coefficient of determination (ARS) and root-mean-square error (RMSE). However, in all the abovementioned research [15] the e-skin graphene sensor’s characteristics manifest hysteretic behaviour, and the visible effect of hysteresis on the modelling accuracy has not yet been discussed.

Hysteresis is the dependence of the state of a system on the state’s history, and we can find this phenomenon in many science fields as physics, chemistry, and engineering. Besides the subject-specific models, some hysteretic models capture general features of many systems with hysteresis that can be classified as algebraic models, transcendental models, differential models and integral models [16]. One of the promising solutions is to use artificial neural networks to model the system’s data with hysteretic behaviour. They have distinct advantages over linear identification methods, i.e., the approximation of multivariable nonlinear functions, the simple gradient-based adaptation of model parameters and a rapid calculation of neural network equations. In contrast to analytical models, the design procedure of the ANN does not demand an exact knowledge of model physical equations and physical parameters that describe the model, but only values of model variables in the causal form. Several neural network architectures with recurrent layers and memory capacity were proposed recently, e.g. recurrent neural networks for ultra-capacitors [1], physics-informed deep neural networks for mechanical dampers [10] or extended Preisach neural network [6] among others.

The presented research aims to propose a novel method of e-skin graphene sensor modelling using the NARX recurrent neural network that can describe the hysteretic behaviour of the sensor.

2 Graphene-Based Electronic Skin

The e-skin used for the research consisted of two layers [13, 14]. The first is a conductive layer of comb electrodes printed on plastic foil and connected along columns and rows. In contrast, the second one comprises FSR graphene sensors arranged in a rectangular pattern placed on a plastic foil. In the research, a matrix with the size of a single sensor (approx. 5 \(\times \) 5 mm) was used. The e-skin controller measures the pressure for each sensor and transmits the position and touch force exerted on the active surface. The data is sent to a computer, processed and saved. The computer software enables the visualization of the touch results as a colour-coded image. Figure 1 presents the hand touch measurement for the 16 \(\times \) 32 FSR matrix.

Fig. 1.
figure 1

Example hand touch visualization for the size of 16\(\,\times \,\)32 cells; Gaussian blur used for post-processing.

Full size image

The measurement acquisition setup comprised the FANUC LR Mate 200iC manipulator, the R-30iA Mate manipulator controller, the e-skin with a driver, the OnRobot Hex-e 6-axis force and torque measuring device with a controller, and a general-purpose PC. Data from e-skin sensors and the reference Hex-e sensor were acquired during the calibration procedure. Data acquisition was subdivided into the ‘loading phase’, when the force exerted on the particular sensor of the e-skin by the robotic arm increased, and the ‘unloading phase’, when the force decreased (Fig. 2).

Fig. 2.
figure 2

Illustrative representation of the calibration station: 1. LRMate 200iC manipulator, 2. reference e-Hex sensor, 3. robot tool, 4. e-skin electronic system, 5. e-skin.

Full size image

3 Neural-Network Modelling

Nonlinear AutoRegressive eXogenous model (NARX) in the form of the recurrent artificial neural network (R-ANN) was used to model the e-skin graphene sensor to estimate the pressure (force exerted on sensor) based on the sensor readouts. NARX-RNN nonlinear model extends the autoregressive linear model with the exogenous input (ARX), popular in time-series nonlinear modelling. NARX-RNN model is the input-output model, where the output in each step is described by input signal with noise. NARX-RNN has memory capabilities. It memorizes previous data and can be used to model hysteretic behaviour. While making a decision, it considers the current input and what it has learned from the inputs it received previously. Output from the previous step is fed as input to the current step creating a feedback loop.

In such a case, the nonlinear part of the NARX-RNN model was described as

$$\begin{aligned} y_{\text {NN}}(k)=F_{\text {NN}}(k)=f\left( V(k), V(k-1), F_{\text {NN}}(k-1)\right) \end{aligned}$$
(1)

where \(y_{\text {NN}}(k)\) - output of the NARX-RNN, \(F_{\text {NN}}(k)\) - estimated output (touch pressure) at step k, V(k) - sensor readouts at the step k, k - sample number, \(t=kT_{p}\) - time, \(T_p\) - sampling time.

Exemplary scheme of used in the research NARX-RNN model is presented in Fig. 3.

Fig. 3.
figure 3

Used in the research NARX-RNN architecture (2 neurons in the nonlinear hidden layer, D - delay (\(k-1\))).

Full size image

The NARX-RNN comprises the nonlinear hidden layer that processes the input data as

$$\begin{aligned} \left\{ \begin{array}{l} y^{(1)}{ }_{(j) \text {NL}}(k)=f^{(1)}{ }_{(j) \text {NL}}\left( x^{(1)}{ }_{(j)\text {NL}}(k)\right) ; f^{(1)}{ }_{(j)_{\text {NL}}} \in [-1,1]\\ x^{(1)}(n)_{\text {NL}}(k)=b_{(n)}^{(1)}+w_{p+1, n}^{(1)} y_\text {L}^{(2)}(k-1)+\sum _{i=1}^p w^{(1)}{ }_{i, n} x_i(k) \end{array} \right. \end{aligned}$$
(2)

where \(y^{(1)}{ }_{(j) \text {NL}}(k)\) - output signal of the j-th neuron in the nonlinear layer, \(x^{(1)}{ }_{(j)\text {NL}}(k)\) - input signal to the j-th neuron in the nonlinear layer, n - the number of neurons in the nonlinear layer, p - the number of neuron inputs in the nonlinear layer, \(w^{(1)}{ }_{i, j}\) - the weight of the i-th input to j th neuron in the nonlinear layer, \(x_i(k)\) - i-th input to the network (with tapped delay line, D inputs in Fig. 3), \(b_{(n)}^{(1)}\) - the threshold offset of the n-th neuron in the nonlinear layer. The second term of (2) describes the recurrent feedback loop with delay (D in Fig. 3). In the considered case, the output of the RNN network \(y^{(1)}{ }_\text {L}(k)\) is the output signal from a linear output layer, with 1 linear neuron described as follows

$$\begin{aligned} \left\{ \begin{array}{l} y_{\text {NN}}(k)=y_{\text {L}}^{(2)}(k)=f_{\text {L}}^{(2)}\left( x_{\text {L}}^{(2)}, k\right) =x^{(2)}, f_{\text {L}}^{(2)} \in R\\ x_{\text {L}}^{(2)}(k)=b_{\text {L}}{ }^{(2)}+\sum _{i=1}^n w_{\left( \text {L}\right) i, 1}{ }^{(2)} h_i(k) \end{array} \right. \end{aligned}$$
(3)

where \(y_{\text {NN}}(k)\) - the network output, equal to the linear layer output, \(x^{(2)}_{\text {L}}(k)\) - the input signal to the neuron in the linear output layer, \(w_{\left( \text {L}\right) i, 1}^{(2)}\) - the weight of the ith input to neuron in the linear output layer, and \(b_{\text {L}}{ }^{(2)}\) - the neuron’s bias in the linear output layer.

For the training and evaluation of the NARX-RNN model, the estimation error was calculated as follows

$$\begin{aligned} e_{\text {NARX-RNN}}(k)=y_{\text {NN}}(k)-y(k) \end{aligned}$$
(4)

4 Results

4.1 Research Method

The Mean Squared Error (MSE) has been used to describe the performance function \(E_{\text {NARX-RNN}}\) of the NARX-RNN model during training and testing. The changes in the weights in the i-th iteration were used in the NARX-RNN models according to the Levenberg-Marquardt algorithm, and variable metrics method [12]. The training algorithm stop conditions were defined because of the possible large step of each iteration. The mentioned conditions are usually estimated as the assumed minimum value of the performance function and the maximum number of training iterations. In the case described in this article, the stopping conditions were: epochs \(=1000\), \(E_{\text {NARX-RNN}} \le 10^{-6}\). The early stop method was used during training of the NARX-RNN model in a controlled way by segmentation of the dataset into three subsets namely: Training subset used during NARX-RNN training (70% of data), Validation subset used for NARX-RNN validation during training and to prevent the data overfitting (15% of data), Testing subset not used in the training phase, only used for comparison of the models during final evaluation (15% of data). The NARX-RNN models were trained per iteration in batch mode [7], while weights and biases were initialized using the Nguyen- Widrow initialization procedure [17]. The values of the sensor readouts and the touch pressure measured by the HEX device were normalized to the range [0, 1] to avoid the early stop due to neuron saturation. The quality of the NARX-RNN model of the e-skin graphene sensor was evaluated based on the MSE performance function, and the goodness of fit between the estimated data and the reference data was calculated as Root Mean Square Error (RMSE) [11].

Fig. 4.
figure 4

(a) E-skin sensor measurement characteristic. (b) Residuals for e-skin sensor RNN model (1 neuron described with satlins function).

Full size image
Table 1. Quality indices of the RNN models for ‘loading’ and ‘unloading’ phases.
Full size table

4.2 Modelling

The proposed NARX-RNN model was evaluated on the exemplary sensor (row 6 and column 16 of the sensor matrix). The sensor-measured characteristic is presented in Fig. 4. The two types of nonlinear hidden layer neuron transfer functions were used: hyperbolic tangent sigmoid transfer function (tansig) and symmetric saturating linear transfer function (satlins). The RMSE values for the NARX-RNN models with the different numbers of neurons in the hidden layer before and after training are presented in Table 1.

4.3 Discussion

Obtained results, presented in Table 1, indicate that touch pressure estimation improves after training around 200–300 times. The estimation errors do not depend significantly on the number of neurons in the hidden layer, even if only one neuron is used. Moreover, the number of iteration epochs is a few times lower in the case of satlins function describing neurons in the hidden layer (14–21 epochs), compared with tansig function (68–187 epochs). The obtained quality indices are similar for the ‘loading’ and ‘unloading’ phases, thus properly modelling the hysteretic behaviour of the sensor.

5 Summary

The article presents the possibility of touch pressure estimation of the e-skin graphene pressure sensors with hysteretic behaviour using NARX recurrent neural networks. Increasing the number of neurons in the nonlinear hidden layer did not improve the generalization properties of the model. Moreover, the neurons described by the simple satlins function give similar results with fewer epochs than those described with tansig.

The presented method provides a simple alternative to the e-skin graphene pressure sensors models known in the literature. Further research should be done to extend the developed model to the sensor matrix and the layered deep neural networks for e-skin calibration and data compression.