Introduction

Speech and handwriting are probably the most difficult tasks performed by human beings, because they differentiate us from animals. Handwriting requires very fine motor skills, probably more so than speech, because some animals can imitate human sounds but no animal can write. In addition, we learn to speak first and then we learn how to read and write, when the brain is more mature.

Handwriting analysis is a good way to study the human brain in a non-invasive way. This knowledge, once acquired, can be applied to artificial systems that emulate the human brain. We consider that handwriting movements are more complex by far than what has been analyzed in the past. In fact, some parts of the movements have been neglected. With this paper, we will analyze this kind of movements, which will be defined in posterior sections as in-air at long distance. This kind of movements can be used to improve artificial intelligence for biometric applications such as health and security [1,2,3,4].

In the past, the analysis of handwriting had to be performed in an offline manner. Only the writing itself (strokes on a piece of paper) were available for analysis. Nowadays, modern-capturing devices like digitizing tablets and pens or online whiteboards can gather data without losing its temporal dimension. When spatiotemporal information is available, its analysis is referred to as online. A typical modern-digitizing tablet (Fig. 1) not only gathers the x-y coordinates that describe the movement of the writing device as it changes its position, but it can also collect other data, mainly the pressure exerted by the writing device on the writing surface, the azimuth (the angle of the pen in the horizontal plane), and the altitude (the angle of the pen with respect to the vertical axis) (see (Fig. 2)). From now own, x-y coordinates, pressure, azimuth, and altitude will be referred to as features of the handwriting.

Fig. 1
figure 1

Intuos Pro L digitizing tablet and pen

Full size image
Fig. 2
figure 2

Azimuth and altitude angles of the pen with respect to the plane of the writing surface

Full size image

A very interesting aspect of the modern online analysis of handwriting is that it can consider information gathered when the writing device was not exerting pressure on the writing surface. Thus, the movements performed by the hand while writing a text can be split into two classes:

  1. 1.

    On-surface trajectories (pen-downs), corresponding to the movements executed while the writing device is touching the writing surface. Each of these trajectories produces a visible stroke. We will call this kind of movement on-surface.

  2. 2.

    In-air trajectories (pen-ups), corresponding to the movements performed by the hand while transitioning from one stroke to the next one. During these movements, the writing device exerts no pressure on the surface. This class can be split into two subsets:

    1. a.

      In-air at short distances (in-airS), when the distance from the tip of the pen to the writing surface is lower or equal to 1 cm. In this case, the digitizing device can track the (x, y) coordinates during the pen movement.

    2. b.

      In-air at long distances (in-airL), when distances from the tip of the pen to the writing surface are higher than 1 cm. In this case, the digitizing device is not able to track the movements and we only know the time spent at high distance.

In our previous research, we have focused on on-surface and in-airS movements discarding in-airL movements because they do not provide the same amount of data as the previous ones. In fact, the unique parameters are just the number of strokes at long distance and time spent at long distance. For instance, in [5], we applied information theory to demonstrate that on-surface and in-airS contain almost the same amount of information and they are not redundant. This was an important milestone because in-air trajectories had received almost no attention at all, even in online approaches where spatiotemporal information is available.

Figure 3 shows two examples of on-surface and in-airS trajectories taken from two executions of the pentagon test performed by two different writers from the Emothaw database.

Fig. 3
figure 3

On-surface (top) and in-air (bottom) trajectories from two executions of two crossed pentagons

Full size image

In-airL can be detected looking at the time stamp provided by the digitizing tablet. During in-airL time, the tablet is unable to track the tip of the pen and no samples are acquired. Nevertheless, time stamp is increasing and the next time that the pen touches the surface, the samples are stored again in the file and the time jump can be detected. Figure 4 shows the difference of consecutive time stamps for an example file. For most of the samples (on-surface and in-airS), this value is small (typically two units). However, there are some peaks, which correspond to in-airL movements. Figure 4 reveals 11 strokes of the type in-airL. Sometimes, this time is abnormally long. This is probably due to some acquisition problem, where the user started to speak with the database acquisition supervisor for minutes. We will label these cases and will not include them in the average computation of time spent at in-airL. We consider these cases when time in-airL is greater than 70% of the total time. In particular, we have found this phenomenon in 5 files from the analyzed databases (total amount of analyzed files is 17,951 files) (e.g. see Fig. 5).

Fig. 4
figure 4

Time stamp difference of consecutive samples for an example of accepted file from PaHaW database task write lektorka word twice

Full size image
Fig. 5
figure 5

Time stamp difference of consecutive samples for an example of discarded file from PaHaW database task write lektorka word twice

Full size image

Experimental Databases

In this paper, we have analyzed a set of different databases that contain different tasks and user profiles. The databases share the existence of handwritten tasks. In this section, we will summarize the main characteristics of the analyzed databases.

BIOSECUR-ID

This database is a multimodal biometric one and includes eight biometric traits: speech, iris, face (still images and videos), handwritten signature and handwritten text, fingerprints, hand, and keystroking. This database acquired inside the Biosecur-ID project was developed by a consortium of six Spanish Universities, more details can be found in [6]. With respect to handwriting and signatures, this database defines five different tasks: a Spanish text in lower-case, ten digits written separately, 16 Spanish words in upper-case, four genuine signatures, and one forgery of the three precedent subjects.

EMOTHAW

As described in [7], this database includes samples of 129 participants who are classified on the basis of their emotional states: anxiety, depression, and stress or health. This classification is assessed by the Depression–Anxiety–Stress Scales (DASS) questionnaire. Seven tasks are recorded through a digitizing tablet: pentagons and house drawing, words in capital letters copied in handprint, circles with left and right hand, clock drawing, and one sentence copied in cursive writing.

PAHAW

The Parkinson’s Disease Handwriting Database (PaHaW) consists of multiple handwriting samples from 37 Parkinson’s disease patients, and 38 gender and age matched controls. Eight different tasks were recorded through a digitizing tablet: spiral drawing, letters, words, and a sentence. The details about this database can be found in [8].

OXIGEN-THERAPY

This database described in [9] includes eight patients with hypoxemia who performed two tasks: house and clock drawing, before and after breathing 30 min with O2 with the aim of evaluating changes in psychomotor functions.

SALT

As described in [10], the database includes samples of 52 participants: 23 with Alzheimer’s disease, 12 with mild cognitive impairment (MCI), and 17 healthy controls. Seven tasks were recorded: crossed pentagons, spiral, 3D house, clock drawings, spontaneous, copied, and dictated handwriting.

Experimental Results

The first experiments consisted of analyzing the three kinds of time in absolute and relative values as well as the number of strokes in all the scenarios. Tables 1, 2, 3, 4, and 5 summarize the results for the analyzed databases. It is worth remarking that different databases contain different tasks described in the previous section.

Table 1 BIOSECUR-ID database. Time in absolute units and relative time in parenthesis
Full size table
Table 2 EMOTHAW database. Time in absolute units and relative time in parenthesis
Full size table
Table 3 PAHAW database. Time in absolute units and relative time in parenthesis
Full size table
Table 4 OXIGEN-THERAPY database. Time in absolute units and relative time in parenthesis
Full size table
Table 5 SALT database. Time in absolute units and relative time in parenthesis
Full size table

For a given user, the number of strokes is an integer number. However, the table shows the average number of strokes for a specific database and task (in addition to the number of strokes done by the whole set of users split by the number of users). This number is not integer anymore.

Experimental results of BIOSECUR-ID database, which is the largest one according to the number of users and files, reveal that in-airL is almost negligible in the case of signatures, but interestingly, it is three times larger for skilled forgeries than for genuine signatures. For uppercase words, the time in-airL is larger than for the other tasks but still quite modest (10.4%). Thus, this kind of movement is less important than the other two and can probably be ignored without sacrificing a lot of information. For the other databases, a statistical test will be performed after presenting the experimental results.

From all the databases related to diseases, we computed the Mann-Whitney U test between study and control groups to determine the existence of statistically significant difference (p < 0.05) in the studied features (time and strokes). The results are shown in Table 6.

Table 6 EMOTHA (Mann-Whitney U test)
Full size table

We can observe in Table 6 (a. Depression/control) how in crossed pentagon task, the values are very close to the threshold for long distance time and strokes. In house draw, the near time and on-surface strokes show statistical significance. In Table 6 (b. Anxiety/control), house draw shows again that near-distance time is statistically signficant. Finally, in Table 6 (c. Stress/control), we obtain p < 0.05 for on-surface time in clock draw only.

As is shown in Table 7, for PaHaW database we obtain statistically significant results in letter l long distance time and in bigram le for near-distance time and on-surface strokes.

Table 7 PaHaW (Mann-Whitney U test)
Full size table

In OXYGEN THERAPY database, the times and number of strokes do not show statistical significance and do not seem to offer a valid classification pattern between pre- and post-O2 results (Table 8).

Table 8 OXYGEN THERAPY (Mann-Whitney U test)
Full size table

In Table 9 (SALT, a. Alzheimer/control), we can observe how on crossed pentagons draw, statistical significance can be found in on-surface time and long distance time. Also, on-surface time presents significance on the sentence copied. No results with p < 0.05 were obtained for mild cognitive impairment (MCI)/control (Table 9, b).

Table 9 SALT (Mann-Whitney U test)
Full size table

Discussion

Although most of the results in previous tables are not significant, even for on-surface and in-airS information, we should point out that this kind of measurements offers a large set of features that can be extracted, such as speed and acceleration of trajectories and complexity measurements extracted from coordinates x, y. In fact, a classifier would not be based on a single measurement. It will take advantage of a set of measurements. Thus, high p values for on-surface and in-airS do not imply the impossibility to perform a classification. These values are provided just for comparison purpose with in-airL values. In-airL extracted features are limited to time and number of strokes. Thus, the analysis of relevance of this information is simpler.

Nevertheless, this paper points out the tasks and pathologies where more potential improvements can be achieved, because in some tasks, p < 0.05 has been obtained.

Looking at the experimental results of pathologies, we can establish that in-airL movements are not relevant but there are some exceptions: crossed pentagon task for depression patients in EMOTHAW, which is near significance (p = 0.0589 for time and p = 0.0561 for strokes), letter l task for PaHaW database (p = 0.0157 for time), and crossed pentagons task for Alzheimer/control comparison (p = 0.0122 for time). We consider these results especially interesting because crossed pentagons are a very useful measurement in pathological analysis, in fact, it is the only drawing that subjects must perform in the well-established mini-mental state examination, also known as the Folstein test [11].

Conclusions

One of the main goals of this paper was to study if in-airL information can be discarded in handwritten tasks analysis. Looking at the experimental results, we can conclude that little time is spent by healthy writers at long distance so most of the information is contained on-surface and in-airS distances. This implies that the development of a new acquisition device able to track x and y coordinates and long distances will probably not be very useful, because few samples will be acquired in this condition. However, experimental results reveal that time spent at long distance is more than three times higher for skilled forgeries than for genuine signatures. This opens a possible research line in security biometrics. A similar consideration can be established for the number of strokes, which is doubled in the case of skilled forgeries with respect to short distance in-air movements. Thus, the existence of long distance movements can be indicative of a signature forgery.

On the other hand, when looking at pathologies, we have found statistically significant differences in the pentagon tasks for Alzheimer/control comparison. This result opens the possibility of investigating in-air at long distance movements further.