1 Introduction

Handwriting requires cognitive, visual-motor, and memory skills to master. Due to the complexity of human perception, a growing trend in the design of handwriting assistive technologies involves using multimodal interfaces [5, 10]. Traditional assistive technologies for handwriting acquisition have focused on audio and visual modalities, but recently there has been a trend to exploit the haptic modality to further improve sensorimotor abilities [1, 3, 8, 11].

In order to resemble real-life handwriting experience, haptic-visual feedback in touchscreen with stylus-based interaction are proposed [2]. A vibration motor is attached to the stylus to provide vibrotactile feedback based on the interaction with the touchscreen device [13]. Results showed that users benefit greatly from the vibrotactile feedback. Some combined vibrotactile feedback with visual guidance [9, 15], and showed improved performance.

Although promising, existing methods for tactile and/or force feedback guidance are based on using mechanical attachments and do not address ergonomic factors on handwriting performance (such as visual occlusion and flexibility in stylus grip and writing pressure). In this paper, we propose a multimodal system using contactless force feedback guidance based on magnetic forces that overcomes variations in grasping styles and applied pressure forces. Therefore, the contributions of this paper include the following: (1) proposing the hardware and software design of KATIB, a multimodal handwriting system with magnetic-based haptic guidance for improved ergonomics, (2) developing rendering algorithms for full and partial haptic guidance, and (3) characterization and technical evaluation of the proposed platform.

2 Related Work

Magnetically-driven haptic guidance is desirable since magnetic forces can be felt at the tip of the handwriting stylus without having mechanical attachment that may occlude the visual display. A common approach is to utilize an array of electromagnets to guide users to appropriate screen locations [16]. Actuated Workbench [7, 12], Proactive Desk II [16], and Fingerflux [12] are tabletop systems that can make physical objects placed on the table move using an array of electromagnets. For instance, Fingerflux provided near-surface haptic feedback to guide the user’s finger to appropriate locations on a touchscreen device [12]. However, this approach reported a drifting error of more than 10 mm which makes it unsuitable for handwriting tasks (literature suggests less than 3 mm error for handwriting tasks [4]).

An interesting approach is to attach a magnet or an electromagnet to the end effector of a two DoF motorized linkage mechanism to control the stylus, altogether placed underneath a writing surface (paper) to provide haptic guidance [6, 14]. In the dePENd system, a computer controls the xy position of the magnet under the writing surface in order to move the pen and present haptic guidance [14]. As the linkage mechanism moves the magnet along a desired trajectory, it attracts the stylus through magnetic forces to move along the same trajectory. No visual guidance is provided. A recent study demonstrated a system to deliver dynamic guidance in drawing and sketching via an electromagnet placed underneath a pressure sensitive tablet [6]. The system allows the user to move the pen freely and renders pull back forces using a closed-loop time-free approach to minimize the error between the pen position and the desired trajectory.

These systems do not provide synchronized haptic-visual guidance for handwriting. Driven by previous findings that multimodal guidance is more effective for motor learning [10], KATIB provides mechanically decoupled, multimodal feedback system with synchronized haptic-visual guidance to support multi-stroke handwriting. Compared to electromagnetic-based guidance, the permanent magnet provides concentrated magnetic flux and thus higher fidelity of haptic guidance. Furthermore, larger haptic guidance forces are achievable with permanent magnet, compared to electromagnet. Finally, the heating effects of the electromagnet weakens the magnetic flux and thus reduces the haptic guidance force over usage.

3 KATIB System

A schematic diagram of the KATIB system design is shown in Fig. 1(a). The system comprises a 3D printed pen-like stylus that hosts cylindrical 5 \(\times \) 5 [mm] (or more) and one cylindrical 2 \(\times \) 2 [mm] vertically polarized N42 neodymium magnets that are stacked on top of each other forming the stylus tip, a low-cost 4 mm thick resistive touchscreen 640 \(\times \) 480 pixels display to provide visual guidance and capture interactions between the writing surface and the stylus, a N42 neodymium magnet underneath the screen that is attached to the end-effector of a 2 DoF parallel manipulator for moving the magnet along a particular trajectory and provide magnetic haptic guidance, and a board computer that runs an application to provide synchronized audio, visual, and haptic guidance for the learner. A snapshot of the system prototype is shown in Fig. 1(b).

Fig. 1.
figure 1

(a) KATIB system design, (b) KATIB prototype.

Full size image

3.1 Magnetic Force Acting on the Stylus

The magnetic force due to a non uniform magnetic field can be calculated through the following equation:

$$\begin{aligned} \mathbf{F} =\nabla \left( \mathbf{m} \cdot \mathbf{B} \right) ,~\mathbf{m} =\frac{1}{\mu _0}{} \mathbf{B} _rV \end{aligned}$$
(1)

whereas m is the magnetic moment vector. In case of a permanent magnet the magnetic moment can be expressed through the residual flux density of the magnet \(B_r\). The residual flux density of the magnet is usually provided by the manufacturer and V is the volume of the magnet. The values of the parameters above, as well as the derivation of the magnetic flux distribution B, are discussed in the magnetostatic analysis in Sect.  4.2

3.2 Hardware Implementation

Katib sytem was designed around a Raspberry Pi Model B+ single board computer which is running Linux software. It drives two NEMA17 stepper motors which are equipped with a 5:1 reduction planetary gearbox. The stepper motors are PID controlled through the uStepper driver platform which is based on an ATMEL ATMEGA328 MCU, the Trinamic TMC5130 Motor Driver and the Broadcom AEAT8800-Q24 Hall effect encoder. The motors can operate at a maximum update rate of 1800 Hz. The rotation of the end effector is achieved by the use of a custom solution comprised of a 15k RPM micro DC motor equipped with a 300:1 reduction gearbox and a quadrature hall effect encoder which is driven from a Texas Instruments DRV8838 driver and an ATMEL ATMEGA328. The Rasberry PI is also connected to a 640 \(\times \) 480 TFT display which is driven by a generic HDMI-TFT driver. The display is equipped with a resistive touchscreen driven by the MICROCHIP AR1100 touchscreen controller to detect contact with the stylus. The haptic guidance force can be rendered at a frequency of 1800 Hz. A schematic diagram of the hardware implementation is shown in Fig. 2.

Fig. 2.
figure 2

Hardware implementation.

Full size image

3.3 Graphical User Interface Design

As shown in Fig. 3, KATIB provides two interfaces: an instructor window and a learner window. The instructor window enables teachers to record a handwriting task, assign it to one or more learners, and examine the learners performance. On the other hand, the learner window provides learners with a list of handwriting tasks to exercise and record. The recorded handwriting tasks are evaluated and a report about the learner’s performance is sent back to the instructor.

Fig. 3.
figure 3

(a) Learner interface, (b) Instructor interface.

Full size image

3.4 Haptic-Visual Guidance

Visual guidance is implemented by showing a visual trace of the entire handwriting task as well as a highlighted visual target for the immediate next move along the handwriting trajectory. In multi-stroke tasks, a flashing visual dot of different color is displayed to show the starting point of the subsequent stroke. As for haptic guidance, full and partial haptic-visual guidance methods are developed for KATIB system. In the full haptic guidance method, the system leads the movement by providing visual feedback about the next point to move to along the trajectory and applies a maximum force to move the stylus to the desired position. Once the user is at the desired position, the next point is identified and haptic-visual guidance is provided for the next point. The full guidance method is detailed in Algorithm 1. Figure 5 demonstrates high fidelity of full haptic guidance where the average root mean square (RMS) error is 2.73 mm.

Partial haptic-visual guidance is user-led. The KATIB system provides visual feedback for the next point to move the stylus to and provide force guidance only when the user deviates significantly from the desired trajectory. The magnetic force is switched on or off by rotating the magnet by 180\(^\circ \) from its original position. If the user completes the entire handwriting task within the trajectory error threshold, no haptic feedback is applied. The partial haptic-visual guidance method is shown in Algorithm 2.

figure a

4 System Characterization

4.1 Haptic Guidance Workspace

The work space of the 2DOF parallel manipulator can be calculated by solving the forward kinematics problem. The stepper motors can be driven in up to 16 microsteps and since the planetary gearbox has a 5:1 reduction rate, the final number of steps per motor shaft revolution is 16000 steps. A Matlab scrip was written to solve the kinematic problem for the specified step resolution for a range between −45 to 45\(^\circ \) per motor and 100 mm as the length of each parallel manipulator arm. The measured workspace for the device is 80 mm (width) by 60 mm (height), which is sufficient for most handwriting tasks.

4.2 Magnetostatic Analysis

In order to calculate the magnetic force acting on the stylus according to Eq. 1, we need to derive the magnetic field distribution for the specific configuration (magnet, glass screen, stylus magnet) under a steady current, which in this case is equal to zero. The magnetostatic equations are derived from Maxwell’s equations under the assumption of either fixed or moving charges with a steady current. In such cases, Maxwell’s equations can split into two pairs of equations: two equations describing the electric field (electrostatics) and two equations describing the magnetic field (magnetostatics).

The ANSYS Magnetostatic Analysis module was utilized to solve the magnetic field distribution for the different translational and tilt cases. The geometry consisted of a stack of 2 cylindrical magnets for the stylus(5 \(\times \) 5 mm and 2 \(\times \) 2 mm respectively) and one cylindrical (5 \(\times \) 5 mm) magnet for the end effector, separated by a surface of glass material with 4 mm thickness (\(\mu _{r/glass} = 5\)). The structure was enclosed in a volume of simulated air (\(\mu _{r/air} = 1\)). Neodymium N42 (NdFeB) was used to simulate the magnets with a residual induction of 1300 mT and a coercive force of 955 KA/m. The geometry was meshed using a “sphere of influence”, a heavily refined spherical mesh volume encapsulating the region whereas the magnets where interacting, to ensure a stable and accurate approximation. The final mesh comprised about 250k elements. Finally, the simulation was repeated for the different translation and tilt configurations of the stylus and end effector. The tilt axis was defined by the tangency between the base of the stylus magnet and the glass surface.

Effects of Stylus Displacement/Tilt on Guidance Force. As the magnet moves to the next position along the trajectory, it attracts the stylus to move into the same direction. The normal and tangential attraction forces are analyzed as function of the horizontal distance between the magnet and the stylus. The effects of displacement is simulated as shown in Fig. 4 (a). The results, shown in Fig. 4 (b), demonstrate a peak tangential force of 0.43 N from 2 mm to 5 mm displacement, which produces a perceivable haptic guidance. Furthermore, the effects of tilting the stylus on the amplitude of the attraction force is also studied. As shown in Fig. 4 (c), tilting the stylus up to 40\(^\circ \) from the normal direction would results in a negligible tangential force.

Fig. 4.
figure 4

(a) Magnetic interaction modeling, (b) magnetic forces against displacement, (c) magnetic forces against tilt angle.

Full size image
Fig. 5.
figure 5

Sample handwriting tasks with stylus drifting along the writing surface (EE for end-effector).

Full size image

Friction Effects on Stylus. The effects of friction between the stylus and the handwriting surface is examined. The stylus freely drifts along the handwriting trajectory due to the attraction force by the moving magnet. The magnet and stylus positions are recorded for three different handwriting tasks and plotted in Fig. 5 where differences in traces are mainly due to friction effects. Since the static friction \(f_s\) is larger than the dynamic friction \(f_d\), it must be true that the tangential force Fx is larger than the static friction force fs because the stylus is moving. Given the results from Fig. 4(b) showing that the tangential force maximizes at around 0.43 N at 3.5 mm away from the magnet implies that the static friction must be less than 0.43 N to be able to move the stylus (see Fig. 5). It must be noted that this is an extreme case and in reality the user is holding the stylus and applying some forces along the desired trajectory. Also, while the user is grasping the stylus there is always a tilt angle that reduces further the effects of the friction.

5 Conclusion

In this paper, we proposed a haptic-visual guidance system that utilizes magnetic forces for haptic guidance to support handwriting. By using magnetic forces, the system improves the ergonomics of handwriting (support for grasping styles and force profiles) while maintaining a highly fidelity of haptic guidance. As for future work, we will conduct a usability study to thoroughly evaluate the ergonomic benefits for learning as well as augmenting handwriting skills.