Online handwriting trajectory reconstruction from kinematic sensors using temporal convolutional network

Abstract

Handwriting with digital pens is a common way to facilitate human–computer interaction through the use of online handwriting (OH) trajectory reconstruction. In this work, we focus on a digital pen equipped with sensors from which one wants to reconstruct the OH trajectory. Such a pen allows to write on any surface and to get the digital trace, which can help learning to write, by writing on paper, and can be useful for many other applications such as collaborative meetings, etc. In this paper, we introduce a novel processing pipeline that maps the sensor signals of the pen to the corresponding OH trajectory. Notably, in order to tackle the difference of sampling rates between the pen and the tablet (which provides ground truth information), our preprocessing pipeline relies on Dynamic Time Warping to align the signals. We introduce a dedicated neural network architecture, inspired by a Temporal Convolutional Network, to reconstruct the online trajectory from the pen sensor signals. Finally, we also present a new benchmark dataset on which our method is evaluated both qualitatively and quantitatively, showing a notable improvement over its most notable competitor.

Appendix

1.1 DTW alignment

The DTW algorithm [25] is based on dynamic programming to assess the similarity between time series. For two multivariate time series $x \in \mathbb {R}^{T_x \times z}$, and $y \in \mathbb {R}^{T_y \times z}$ of equal feature dimensionality z and respective lengths $T_x$ and $T_y$, the DTW is written as follows:

$$\begin{aligned} DTW(x, y) = \min _{\delta \in E(x, y)} \sum _{(i,j)\in \delta } d(x_i, y_j) \end{aligned}$$

(1)

where E(x, y) represents the set of all admissible alignments between x and y and d is a distance metric in $\mathbb {R}^z$. Commonly, the squared Euclidean distance $d(x_i, y_j) = \Vert x_i -y_j\Vert ^2$ is used.

An alignment is a sequence of pairs of timestamps that is admissible if (i) it matches the first (and, respectively, the last) indices of time series x and y together, (ii) it is monotonically increasing, and (iii) it connects the two time series by matching at least one index of each series. Using dynamic programming, the admissible alignment paths can be computed according to the following recurrence formula:

$$\begin{aligned} DTW(x_{\rightarrow i}, y_{\rightarrow j})&= d(x_{i}, y_{j}) \\&\quad + \min \left\{ \begin{array}{lll} DTW(x_{\rightarrow i}, y_{\rightarrow j-1}) \\ DTW(x_{\rightarrow i-1}, y_{\rightarrow j}) \\ DTW(x_{\rightarrow i-1}, y_{\rightarrow j-1}) \end{array}\right. \end{aligned}$$

where $x_{\rightarrow i}$ denotes time series x observed up to timestamp i.

The DTW method on timestamps links multiple sensor signal points to at least one tablet signal data point. In our case, we do not allow the association of multiple points of the tablet signal to one point of the sensor signal. In other words, in the previous recurrence formula the predecessor $DTW(x_{\rightarrow i}, y_{\rightarrow j-1})$ is removed. According to the resulting alignment path, the timestamps from the sensor signal are aligned with their corresponding timestamps from the tablet signal.

1.2 Datasets description

A group of 35 adult writers has contributed to the dataset acquisition. One recording is composed of 34 samples which are randomly selected scripts of characters, words, equations, shapes and word groups. To the next, this new dataset is called IRISA-KIHT. A writer can make several recordings that makes the IRISA-KIHT dataset to be writer imbalanced.

Table 8 IRISA-KIHT and IRISA-KIHT-S datasets descriptions

Fig. 10

Some examples of IRISA-KIHT data. Pen-up (hovering) strokes are in gray and pen-down (touching) strokes are in blue

To evaluate the generalization capacity of our proposed approach, we decided to train and test our model on different writers. The IRISA-KIHT dataset consists of 25 writers’ recordings in the training set (3,770 samples) which represents around $71.5\%$ of the writers’ total number. The rest 10 writers’ recordings never seen in training refer to the test set, representing $28.5\%$ of writers (8.3% of samples). Figure 10 shows some samples of the IRISA-KIHT datasets.

The IRISA-KIHT-S dataset is a subset of the IRISA-KIHT dataset which is available on request.³ This dataset is composed of 30 recordings and it is writer balanced as there is one recording per writer. Table 8 presents the IRISA-KIHT and IRISA-KIHT-S statistical description.

Every 34-sample recording session generates files from the data acquisition mobile app. The sensor signals file has 15 columns and N rows, where N is the number of IMU signals, timestamps, and sensor values. The table has 13 columns: milliseconds, accelerometer front (x, y, z), accelerometer rear (x, y, z), gyroscope (x, y, z), magnetometer (x, y, z) and force signals. Tablet signal files contain milliseconds, position coordinates (x, y, z) and pressure force signals. The transcription (labels) file contains labels and the start and stop timestamps for every sample. Additional files concerning the sensor calibration and recording meta data are provided. The dataset website describes the format and meta data.

1.3 TCN architecture

A TCN (Temporal Convolutional Network)-like layer consists of dilated, residual non-causal 1D convolutional layers with the same input and output lengths. The input tensor of our TCN implementation has the shape (batch_size=None, input_length=None, channels_num=13) and the output tensor has the shape (batch_size=None, output_length=None, channels_num=2). Since each TCN layer has the same input and output length, only the third dimension of the input and output tensors varies ("None" here denote the learning with batches of different size). The TCN layer consists of 4 blocks in TCN-49 and 3 blocks in TCN-373, with each block containing 100 filters followed by a RELU activation function and padded to the "same" input shape. A batch normalization layer was introduced between each pair of consecutive blocks. The performance of a TCN architecture-based model depends on the size of the receptive field (RF) of the network. Since a TCN’s receptive field depends on the network depth N as well as filter size k and dilation factor d, stabilization of deeper and larger TCNs becomes important [2]. The receptive field of our TCN based model is computed as :

$$\begin{aligned} RF = 1 + 2 \cdot (k - 1) \cdot N \cdot \sum _{i}d_{i} \end{aligned}$$

(2)

The kernel size k is selected so that the receptive field covers enough context for predictions. For a regular convolution filter, the dilation d is equal to 1. Using larger dilation enables an output at the top level to represent a wider range of inputs. In the equation, there is a multiplication by 2 because there are two 1D CNN layers in a single residual block of the TCN architecture. For the purposes of the trajectory reconstruction task, we set the stride equal to 1. For the TCN-49 we choose to set k=3, N=4 and d=1, 2 and for the TCN-373 k=3, N=3 and d=1, 2, 4, 8, 16.

Fig. 11

Metrics comparison. On the top, the ground truth and on the bottom two predictions coming from different models

1.4 Fréchetdistance

The formal definition of the Fréchet distance is: Let S be a metric space, d its distance function. A curve A in S is a continuous map from the unit interval into S, i.e. $A:[0,1]\rightarrow S$. A reparameterization $\alpha$ of [0, 1] is a continuous, non-decreasing, surjection $\alpha :[0,1]\rightarrow [0,1]$ Let A and B be two given curves in S. Then, the Fréchet distance between A and B is defined as the infimum over all reparameterizations $\alpha$ and $\beta$ of [0, 1] of the maximum over all $t\in [0,1]$ of the distance in S between $A(\alpha (t))$ and $B(\beta (t))$. The Fréchet distance F is defined by the following equation:

$$\begin{aligned} F(A,B)= \inf _{\alpha , \beta } \max _{t\in [0,1]} \left\{ d(A(\alpha (t)), B(\beta (t)) \right\} \end{aligned}$$

(3)

Appropriate assessment metrics are needed to evaluate trajectory reconstruction to give students useful feedback. Fréchet distance doesn’t disagree with visual evaluation like DTW and RMSE. Figure 11 indicates that the left upper loop of the f is better reconstructed. Fréchet distance is needed to observe this.