The Importance of Pen Motion Pattern Groups for Semi-Automatic Classification of Handwriting into Mental Workload Classes

  • Murad Badarna
  • Ilan Shimshoni
  • Gil Luria
  • Sara Rosenblum
Article

Abstract

In this paper, we introduce the pen motion pattern groups (PMPGs) and their contribution to the classification of handwriting into cognitive mental workload classes. We demonstrate the importance of PMPGs by providing an efficient semi-automatic machine learning-based classification framework that distinguishes between handwritten texts written by the same person under different mental workloads. Our evaluation framework is non-language-dependent since we used stroke features, which are not language-specific, and it takes into account the variability in behavioral biometrics between different writers. The handwritten text data was collected using the Computerized Penmanship Evaluation Tool. This digitizer provided accurate temporal measures throughout the writing. As a first stage, the participants were asked to write a given text in the Hebrew language. Then, as a second stage, the participants’ cognitive workload was manipulated by asking them to hold a number in their memory during the entire writing task. In our experiments, we show that incorporating the PMPGs into the classification process yielded an average cognitive load discrimination accuracy of 92.16%, which decreased to 72.90% when the PMPGs were not considered. The separation of handwritten strokes into PMPGs allows us to account for the fact that the strokes are affected differently under different cognitive mental workloads. This novel distinction between PMPGs is important since the handwriting process in each PMPG is different from a perceptual motor and brain-hand control point of view. Moreover, most of the features that are influenced by cognitive workload are those that cannot be discerned by an expert when looking at a handwritten text on paper, such as azimuth, tilt, velocity, acceleration, and pressure.

Keywords

Handwriting Classification Computerized measures Mental workload Digitizer 

Notes

Compliance with Ethical Standards

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee.

Conflict of Interest

The authors declare that they have no conflict of interest.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

References

  1. 1.
    Alonso-Martinez C, Faundez-Zanuy M, Mekyska J. A comparative study of in-air trajectories at short and long distances in online handwriting. Cogn Comput 2017;9(5):712–20.CrossRefGoogle Scholar
  2. 2.
    Breiman L. Random forests. Mach Learn 2001;45(1):5–32.CrossRefGoogle Scholar
  3. 3.
    Cambria E, Hussain A. 2012. Sentic computing: techniques, tools, and applications, vol 2. Springer Science & Business Media.Google Scholar
  4. 4.
    Drotar P, Mekyska J, Rektorova I, Masarova L, Smekal Z, Faundez-Zanuy M. Decision support framework for Parkinson’s disease based on novel handwriting markers. IEEE Trans Neural Syst Rehabil Eng 2015;23(3):508–16.PubMedGoogle Scholar
  5. 5.
    Drotár P, Mekyska J, Rektorová I, Masarová L, Smékal Z, Faundez-Zanuy M. Evaluation of handwriting kinematics and pressure for differential diagnosis of Parkinson’s disease. Artif Intell Med 2016;67:39–46.CrossRefPubMedGoogle Scholar
  6. 6.
    Faundez-Zanuy M, Sesa-Nogueras E, Roure-Alcobé J. On the relevance of age in handwritten biometric recognition. IEEE international carnahan conference on security technology (ICCST); 2012. p. 105–109.Google Scholar
  7. 7.
    Faundez-Zanuy M, Sesa-Nogueras E, Roure-Alcobé J, Esposito A, Mekyska J, López de Ipiña K. A preliminary study on aging examining online handwriting. 5th IEEE conference on cognitive infocommunications (CogInfoCom); 2014. p. 221–224.Google Scholar
  8. 8.
    Faundez-Zanuy M, Hussain A, Mekyska J, Sesa-Nogueras E, Monte-Moreno E, Esposito A, Chetouani M, Garre-Olmo J, Abel A, Smekal Z, Lopez de Ipina K. Biometric applications related to human beings: there is life beyond security. Cogn Comput 2013;5(1):136–51.CrossRefGoogle Scholar
  9. 9.
    Fierrez J, Ortega-Garcia J, Ramos D, Gonzalez-Rodriguez J. Hmm-based on-line signature verification: feature extraction and signature modeling. Pattern Recogn Lett 2007;28(16):2325–34.CrossRefGoogle Scholar
  10. 10.
    Froese T, Suzuki K, Ogai Y, Ikegami T. Using human–computer interfaces to investigate ‘mind-as-it-could-be’ from the first-person perspective. Cogn Comput 2012;4(3):365–82.CrossRefGoogle Scholar
  11. 11.
    Gomez-Barrero M, Galbally J, Fierrez J, Ortega-Garcia J, Plamondon R. Enhanced on-line signature verification based on skilled forgery detection using sigma-lognormal features. 2015 international conference on biometrics (ICB). IEEE; 2015. p. 501–506.Google Scholar
  12. 12.
    Górska Z, Janicki A. Recognition of extraversion level based on handwriting and support vector machines. Percept Mot Skills 2012;114(3):857–69.CrossRefPubMedGoogle Scholar
  13. 13.
    Gramzow RH, Gaertner L, Sedikides C. Memory for in-group and out-group information in a minimal group context: the self as an informational base. J Pers Soc Psychol 2001;80(2):188–205.CrossRefPubMedGoogle Scholar
  14. 14.
    Hussain A, Niazi M. Toward a formal, visual framework of emergent cognitive development of scholars. Cogn Comput 2013;6(1):113–24.CrossRefGoogle Scholar
  15. 15.
    Kandel S, Valdois S. Syllables as functional units in a copying task. Lang Cogn Process 2006;21(4):432–52.CrossRefGoogle Scholar
  16. 16.
    Koole SL, Dijksterhuis A, van Knippenberg A. What’s in a name: implicit self-esteem and the automatic self. J Pers Soc Psychol 2001;80(4):669–85.CrossRefPubMedGoogle Scholar
  17. 17.
    Kotsavasiloglou C, Kostikis N, Hristu-Varsakelis D, Arnaoutoglou M. Machine learning-based classification of simple drawing movements in Parkinson’s disease. Biomed Signal Process Control 2016;31(1):174–80.Google Scholar
  18. 18.
    Kotsiantis SB. Supervised machine learning: a review of classification techniques. Proceedings of the 2007 conference on emerging artificial intelligence applications in computer engineering. Amsterdam, The Netherlands; 2007. p. 3–24.Google Scholar
  19. 19.
    Likforman-Sulem L, Esposito A, Faundez-Zanuy M, Clémençon S, Cordasco G. Emothaw: a novel database for emotional state recognition from handwriting and drawing. IEEE Transactions on Human-Machine Systems 2017;47(2):273–84.CrossRefGoogle Scholar
  20. 20.
    Likforman-Sulem L, Esposito A, Faundez-Zanuy M, Clémençon S, Bassis S, Esposito A, Morabito CF. Extracting style and emotion from handwriting. Advances in neural networks: computational and theoretical issues; 2015. p. 347–355.Google Scholar
  21. 21.
    Longstaff MG, Heath RA. The influence of motor system degradation on the control of handwriting movements: a dynamical systems analysis. Hum Mov Sci 2003;22(1):91–110.CrossRefPubMedGoogle Scholar
  22. 22.
    López-de Ipiña K, Alonso JB, Solé-Casals J, Barroso N, Henriquez P, Faundez-Zanuy M, Travieso CM, Ecay-Torres M, Martínez-Lage P, Eguiraun H. On automatic diagnosis of Alzheimer’s disease based on spontaneous speech analysis and emotional temperature. Cogn Comput 2015;7(1):44–55.CrossRefGoogle Scholar
  23. 23.
    Luria G, Kahana A, Rosenblum S. Detection of deception via handwriting behaviors using a computerized tool: toward an evaluation of malingering. Cogn Comput 2014;6(4):849–55.CrossRefGoogle Scholar
  24. 24.
    Luria G, Rosenblum S. Comparing the handwriting behaviours of true and false writing with computerized handwriting measures. Appl Cogn Psychol 2010;24(8):1115–28.CrossRefGoogle Scholar
  25. 25.
    Luria G, Rosenblum S. A computerized multidimensional measurement of mental workload via handwriting analysis. Behav Res Methods 2012;44(2):575–86.CrossRefPubMedGoogle Scholar
  26. 26.
    Malik ZK, Hussain A, Wu J. Novel biologically inspired approaches to extracting online information from temporal data. Cogn Comput 2014;6(3):595–607.CrossRefGoogle Scholar
  27. 27.
    McClelland JL. Is a machine realization of truly human-like intelligence achievable? Cogn Comput 2009;1(1):17–21.CrossRefGoogle Scholar
  28. 28.
    Mergl R, Tigges P, Schröter A, Möller H -J, Hegerl U. Digitized analysis of handwriting and drawing movements in healthy subjects: methods, results and perspectives. J Neurosci Methods 1999;90(2):157–69.CrossRefPubMedGoogle Scholar
  29. 29.
    Meulenbroek RGJ, Vinter A, Mounoud P. Development of the start-rotation principle in circle production. Br J Dev Psychol 1993;11(3):307–20.CrossRefGoogle Scholar
  30. 30.
    Mohan V, Morasso P, Sandini G, Kasderidis S. Inference through embodied simulation in cognitive robots. Cogn Comput 2013;5(3):355–82.CrossRefGoogle Scholar
  31. 31.
    Pan SJ, Yang Q. A survey on transfer learning. IEEE Trans Knowl Data Eng 2010;22(10):1345–59.CrossRefGoogle Scholar
  32. 32.
    Paulhus DL, Graf P, Van Selst M. Attentional load increases the positivity of self-presentation. Soc Cogn 1989;7(4):389–400.CrossRefGoogle Scholar
  33. 33.
    Poon KW, Li-Tsang CWP, Weiss TPL, Rosenblum S. The effect of a computerized visual perception and visual-motor integration training program on improving chinese handwriting of children with handwriting difficulties. Res Dev Disabil 2010;31(6):1552–60.CrossRefPubMedGoogle Scholar
  34. 34.
    Rosenblum S. Handwriting measures as reflectors of executive functions among adults with developmental coordination disorders (dcd). Front Psychol 2013;4(1):357–67.PubMedPubMedCentralGoogle Scholar
  35. 35.
    Rosenblum S. Do motor ability and handwriting kinematic measures predict organizational ability among children with developmental coordination disorders? Hum Mov Sci 2015;43(1):201–15.CrossRefPubMedGoogle Scholar
  36. 36.
    Rosenblum S, Epsztein L, Josman N. Handwriting performance of children with attention deficit hyperactive disorders: a pilot study. Physical & Occupational Therapy In Pediatrics 2008;28(3):219–34.CrossRefGoogle Scholar
  37. 37.
    Rosenblum S, Livneh-Zirinski M. Do relationships exist between brain-hand language and daily function characteristics of children with a hidden disability? Recent advances of neural network models and applications: proceedings of the 23rd workshop of the italian neural networks society (SIREN); 2014. p. 269–281.Google Scholar
  38. 38.
    Rosenblum S, Luria G. Applying a handwriting measurement model for capturing cognitive load implications through complex figure drawing. Cogn Comput 2016;8(1):69–77.CrossRefGoogle Scholar
  39. 39.
    Rosenblum S, Parush S, Weiss PL. Computerized temporal handwriting characteristics of proficient and non-proficient handwriters. Am J Occup Ther 2003;57(2):129–38.CrossRefPubMedGoogle Scholar
  40. 40.
    Rosenblum S, Simhon HAB, Gal E. Unique handwriting performance characteristics of children with high-functioning autism spectrum disorder. Research in Autism Spectrum Disorders 2016;23(1):235–44.CrossRefGoogle Scholar
  41. 41.
    Rosenblum S, Werner P. Assessing the handwriting process in healthy elderly persons using a computerized system. Aging Clin Exp Res 2006;18(5):433–39.CrossRefPubMedGoogle Scholar
  42. 42.
    Rosenblum S, Werner P, Dekel T, Gurevitz I, Heinik J. Handwriting process variables among elderly people with mild major depressive disorder: a preliminary study. Aging Clin Exp Res 2010;22(2):141–7.CrossRefPubMedGoogle Scholar
  43. 43.
    Sesa-Nogueras E, Faundez-Zanuy M, Mekyska J. An information analysis of in-air and on-surface trajectories in online handwriting. Cogn Comput 2012;4(2):195–205.CrossRefGoogle Scholar
  44. 44.
    Sesa-Nogueras E, Faundez-Zanuy M, Roure-Alcobé J. Gender classification by means of online uppercase handwriting: a text-dependent allographic approach. Cogn Comput 2016;8(1):15–29.CrossRefGoogle Scholar
  45. 45.
    Smekal Z, Mekyska J, Rektorova I, Faundez-Zanuy M. Analysis of neurological disorders based on digital processing of speech and handwritten text. International symposium on signals circuits and systems (ISSCS); 2013. p. 1–6.Google Scholar
  46. 46.
    Squartini S, Esposito A. Co-worker: toward real-time and context-aware systems for human collaborative knowledge building. Cogn Comput 2012;4(2):157–71.CrossRefGoogle Scholar
  47. 47.
    Squartini S, Hu S, Liu Q. Advances on brain inspired computing. Cogn Comput 2013;5(2):161–3.CrossRefGoogle Scholar
  48. 48.
    Stone JV. Learning perceptually salient visual parameters using spatiotemporal smoothness constraints. Neural Comput 1996;8(7):1463–92.CrossRefPubMedGoogle Scholar
  49. 49.
    Suzuki Y, Kazuo M, Jiro T. Stylus enhancement to enrich interaction with computers. Proceedings of the 12th international conference on human-computer interaction: interaction platforms and techniques, HCI’07; 2007. p. 133–142.Google Scholar
  50. 50.
    Swann WB, Hixon JG, Stein-Seroussi A, Gilbert DT. The fleeting gleam of praise: cognitive processes underlying behavioral reactions to self-relevant feedback. J Pers Soc Psychol 1990;59(1):17.CrossRefPubMedGoogle Scholar
  51. 51.
    Taylor BT, Bove V M Jr. Graspables: Grasp-recognition as a user interface. Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’09; 2009. p. 917–926.Google Scholar
  52. 52.
    Tabatabaey-Mashadi N, Sudirman R, Guest RM, Khalid PI. Analyses of pupils’ polygonal shape drawing strategy with respect to handwriting performance. Pattern Anal Applic 2015;18(3):571–86.CrossRefGoogle Scholar
  53. 53.
    Taylor JG. Cognitive computation. Cogn Comput 2009;1(1):4–16.CrossRefGoogle Scholar
  54. 54.
    Teulings H -L, Contreras-Vidal JL, Stelmach GE, Adler CH. Parkinsonism reduces coordination of fingers, wrist, and arm in fine motor control. Exp Neurol 1997;146(1):159–70.CrossRefPubMedGoogle Scholar
  55. 55.
    Travieso CM, Alonso JB. Special issue on advanced cognitive systems based on nonlinear analysis. Cogn Comput 2013;5(4):397–8.CrossRefGoogle Scholar
  56. 56.
    Werner P, Rosenblum S, Bar-On G, Heinik J, Korczyn A. Handwriting process variables discriminating mild Alzheimer’s disease and mild cognitive impairment. J Gerontol Ser B Psychol Sci Soc Sci 2006;61 (4):228–36.CrossRefGoogle Scholar
  57. 57.
    Xiaojun B, Tomer M, Gonzalo R, Ravin B, Ken H. An exploration of pen rolling for pen-based interaction. Proceedings of the 21st annual ACM symposium on user interface software and technology, UIST ’08; 2008. p. 191–200.Google Scholar
  58. 58.
    Zhao Z, Chen Z, Chen Y, Wang S, Wang H. A class incremental extreme learning machine for activity recognition. Cogn Comput 2014;6(3):423–31.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Murad Badarna
    • 1
  • Ilan Shimshoni
    • 1
  • Gil Luria
    • 2
  • Sara Rosenblum
    • 3
  1. 1.Department of Information Systems, Faculty of Social SciencesUniversity of HaifaHaifaIsrael
  2. 2.Department of Human Services, Faculty of Social Welfare and Health SciencesUniversity of HaifaHaifaIsrael
  3. 3.Laboratory of Complex Human Activity and Participation (CHAP), Department of Occupational Therapy, Faculty of Social Welfare and Health SciencesUniversity of HaifaHaifaIsrael

Personalised recommendations