User and System Cross-Learning of Gesture Commands on Pen-Based Devices

  • PeiYu Li
  • Manuel Bouillon
  • Eric Anquetil
  • Grégoire Richard
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8118)


This paper presents a new design and evaluation of customizable gesture commands on pen-based devices. Our objective is to help users during the definition of gestures by detecting confusion among gestures. We also help the memorization gestures with the guide of a new type of menu “Customizable Gesture Menus”. These menus are associated with an evolving gesture recognition engine that learns incrementally, starting from few data samples. Our research focuses on making user and recognition system learn at the same time, hence the term “cross-learning”. Three experimentations are presented in details in this paper to support these ideas.


Handwritten gesture recognition Marking Menus Customizable gesture interfaces 


  1. 1.
    Kurtenbach, G., Buxton, W.: The limits of expert performance using hierarchic marking menus. In: Proc. INTERACT 1993, pp. 482–487. ACM Press (1993)Google Scholar
  2. 2.
    Kurtenbach, G., Moran, T.P., Buxton, W.: Contextual animation of gestural commands. Computer Graphics Forum 13(5), 305–314 (1994)CrossRefGoogle Scholar
  3. 3.
    Zhao, S., Agrawala, M., Hinckley, K.: Zone and Polygon Menus: Using Relative Position to Increase the Breadth of Multi-stroke Marking Menus. In: Proc. CHI 2006, pp. 1077–1086. ACM Press (2006)Google Scholar
  4. 4.
    Zhao, S., Balakrishnan, R.: Simple vs. Compound Mark Hierarchical Marking Menus. In: Proc. UIST 2004, pp. 33–42. ACM Press (2004)Google Scholar
  5. 5.
    Bau, O., Mackay, W.E.: OctoPocus: a dynamic guide for learning gesture-based command sets. In: Proc. UIST 2008, pp. 37–46. ACM Press (2008)Google Scholar
  6. 6.
    Delaye, A., Anquetil, E.: Hbf49 feature set: A first unified baseline for online symbol recognition. Pattern Recognition 46(1), 117–130 (2013)CrossRefGoogle Scholar
  7. 7.
    Delaye, A., Sekkal, R., Anquetil, E.: Continuous Marking Menus for learning Cursive Pen-based Gestures. In: Proc. IUI 2011, pp. 319–322. ACM Press (2011)Google Scholar
  8. 8.
    Bailly, G., Lecolinet, E., Nigay, L.: Wave Menus: Improving the Novice Mode of Hierarchical Marking Menus. In: Baranauskas, C., Abascal, J., Barbosa, S.D.J. (eds.) INTERACT 2007. LNCS, vol. 4662, pp. 475–488. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  9. 9.
    Almaksour, A., Anquetil, E., Quiniou, S., Cheriet, M.: Evolving Fuzzy Classifier: Application to Incremental Learning of Handwritten Gesture recognition Systems. In: Proc. ICPR 2010 (2010)Google Scholar
  10. 10.
    Almaksour, A., Anquetil, E.: Improving premise structure in evolving Takagi-Sugeno neuro-fuzzy classifiers. Evolving Systems 2, 25–33 (2011), doi:10.1007/s12530-011-9027-0CrossRefGoogle Scholar
  11. 11.
  12. 12.
    Guimbretiére, F., Winograd, T.: FlowMenu: combining command, text, and data entry. In: Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, San Diego, California, United States, November 06-08, pp. 213–216 (2000), doi:10.1145/354401.354778Google Scholar
  13. 13.
    Igarashi, T., Matsuoka, S., Kawachiya, S., Tanaka, H.: Interactive beautification: a technique for rapid geometric design. In: Proc. UIST 1997, pp. 105–114. ACM Press (1997)Google Scholar
  14. 14.
    Norman, D.A.: Natural user interfaces are not natural. ACM Magazine Interactions 17(3), 6–10 (2010)CrossRefGoogle Scholar
  15. 15.
    LaViola, J., Zeleznik, R.: A Practical Approach to Writer-Dependent Symbol Recognition Using a Writer-Independent Recognizer. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(11), 1917–1926 (2007)CrossRefGoogle Scholar
  16. 16.
    Wobbrock, J.O., Wilson, A.D., Li, Y.: Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes. In: Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2007), Newport, Rhode Island, October 7-10, pp. 159–168. ACM Press, New York (2007)CrossRefGoogle Scholar
  17. 17.
    Bouillon, M., Li, P., Anquetil, E., Richard, G.: Using Confusion Reject to Improve User and System Cross-Learning of Gesture Commands. In: Proc. ICDAR 2013 (to be appeared, 2013)Google Scholar
  18. 18.
    Li, P., Anquetil, E.: Graphical Gesture Commands’ learning with the help of Customizable Gesture Menus. In: Proc. IGS 2013 (to be appeared, 2013) Google Scholar
  19. 19.
    Li, P., Delaye, A., Anquetil, E.: Evaluation of Continuous Marking Menus for Learning Cursive Pen-based Commands. In: Proc. IGS 2011, pp. 217–220 (2011)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2013

Authors and Affiliations

  • PeiYu Li
    • 1
  • Manuel Bouillon
    • 1
  • Eric Anquetil
    • 1
  • Grégoire Richard
    • 1
  1. 1.IRISA/INSA RennesRennesFrance

Personalised recommendations