Journal on Multimodal User Interfaces

, Volume 4, Issue 3–4, pp 129–145 | Cite as

Exploiting on-the-fly interpretation to design technical documents in a mobile context

  • Sébastien MacéEmail author
  • Eric Anquetil
Original Paper


Pen-based interaction is well adapted for writing down information in a mobile context. However, there is a lack of software taking advantage of this interaction process to design technical documents in constrained environments. This is because sketch interpretation is a complex research problem and good performances are required to design industrial software. The first contribution of this article is to show how “on-the-fly” interpretation is a viable approach to design robust and efficient sketch interpretation systems. With this process, the system interprets each stroke after its drawing and produces a visual feedback to the user. The interaction with the user is then strongly taken into account to get pertinent information and better interpret the document. Moreover, on-the-fly interpretation opens a large panel of possibilities to design accurate documents, such as contextual help, complex layout reproduction, etc. Of course, such a process has several requirements to be usable in practice, for example to avoid disturbing the user with inconsistent feedback. The second contribution of this paper is a set of strategies, based on grammatical modeling and pattern recognition, to face these needs. We present two industrial pen-based software that are based on the principles we present in this article. One of them, dedicated to architectural floor-plan design, as not been presented previously.


Pen-based interaction Sketch interpretation On-the-fly interpretation Visual languages and grammars Fuzzy logic 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Almaksour A, Anquetil E, Quiniou S, Cheriet M (2010) Personalizable pen-based interface using lifelong learning. In: Proceedings of the international conference on frontiers in handwriting recognition (ICFHR’10) Google Scholar
  2. 2.
    Alvarado C, Davis R (2004) Sketchread: a multi-domain sketch recognition engine. In: Proceedings of the 17th annual ACM symposium on user interface software and technology (UIST’04), pp 23–32 CrossRefGoogle Scholar
  3. 3.
    Anstice J, Bell T, Cockburn A, Setchell M (1996) The design of a pen-based musical input system. In: Proceedings of the sixth Australian conference on computer-human interaction (OzChi’96), pp 260–267 CrossRefGoogle Scholar
  4. 4.
    Blostein D, Lank E, Rose A, Zanibbi R (2001) User interfaces for on-line diagram recognition. In: Selected papers from the fourth international workshop on graphics recognition algorithms and applications. LNCS, vol 2390. Springer, Berlin, pp 93–103 Google Scholar
  5. 5.
    Caetano A, Goulart N, Fonseca M, Jorge J (2002) Javasketchit: Issues in sketching the look of user interfaces. In: Proceedings of the AAAI spring symposium on sketch understanding (SSSU’02) Google Scholar
  6. 6.
    Chen Q, Grundy J, Hosking J (2003) An e-whiteboard application to support early design-stage sketching of uml diagrams. In: Proceedings of the 2003 IEEE symposium on human centric computing languages and environments, pp 219–226 CrossRefGoogle Scholar
  7. 7.
    Chok SS, Marriott K (2003) Automatic generation of intelligent diagram editors. ACM Trans Comput-Hum Interact 10(3):244–276 CrossRefGoogle Scholar
  8. 8.
    Costagliola G, Deufemia V, Risi M (2005) Sketch grammars: A formalism for describing and recognizing diagrammatic sketch languages. In: Proceedings of the eighth international conference on document analysis and recognition (ICDAR’05), Seoul, Korea, pp 1226–1231 CrossRefGoogle Scholar
  9. 9.
    Coyette A, Kieffer S, Vanderdonckt J (2007) Multi-fidelity prototyping of user interfaces. In: INTERACT 2007. LNCS, vol 4662. Springer, Berlin, pp 150–164. Part I Google Scholar
  10. 10.
    Donaldson AF, Williamson A (2005) Pen-based unput of uml activity diagrams for business process modelling. In: Proceedings of the first workshop on improving and assessing pen-based input techniques Google Scholar
  11. 11.
    Evodia (2010)
  12. 12.
    Feng G, Viard-Gaudin C, Sun Z (2009) On-line hand-drawn electric circuit diagram recognition using 2d dynamic programming. Pattern Recognit 42:3215–3223 zbMATHCrossRefGoogle Scholar
  13. 13.
    Forsberg A, Dieterich M, Zeleznik R (1998) The music notepad. In: ACM symposium on user interface software and technology, San Francisco, CA, USA, pp 203–210 Google Scholar
  14. 14.
    Fujiyama K, Iizuka K, Tanaka J (1999) Vic: Cmg input system using example figures. In: Proceedings of the international symposium on future software technology, Nanjing, China, pp 67–72 Google Scholar
  15. 15.
    Gennari L, Kara LB, Stahovich TF, Shimada K (2005) Combining geometry and domain knowledge to interpret hand-drawn diagrams. Comput Graph 29(4):547–562 CrossRefGoogle Scholar
  16. 16.
    Gross MD (1996) The electronic cocktail napkin—a computational environment for working with design diagrams. Des Stud 17:53–69 CrossRefGoogle Scholar
  17. 17.
    Hall A, Pomm C, Widmayer P (2007) A combinatorial approach to multi-domain sketch recognition. In: EUROGRAPHICS workshop on sketch-based interfaces and modelling (SBIM’07) Google Scholar
  18. 18.
    Hammond T, Davis R (2005) Ladder, a sketching language for user interface developers. Comput Graph 29(4):518–532 CrossRefGoogle Scholar
  19. 19.
    Jorge JA, Glinert EP (1995) Online parsing of visual languages using adjacency grammars. In: Proceedings of the eleventh international IEEE symposium on visual languages. IEEE Computer Society, Washington, pp 250–257 Google Scholar
  20. 20.
    Kara LB, Stahovich TF (2004) Hierarchical parsing and recognition of hand-sketched diagrams. In: Proceedings of the 17th annual ACM symposium on user interface software and technology (UIST’04), pp 13–22 CrossRefGoogle Scholar
  21. 21.
    Kim DH, Kim MJ (2006) A curvature estimation for pen input segmentation in sketch-based modeling. Comput Aided Des 36(3):238–248 CrossRefGoogle Scholar
  22. 22.
    Kurtoglu T, Stahovich TF (2002) Interpreting schematic sketches using physical reasoning. In: Proceedings of the AAAI spring symposium on sketch understanding (SSSU’02), Palo Alto, CA, USA, pp 78–85 Google Scholar
  23. 23.
    Landay JA, Myers BA (2001) Sketching interfaces: Toward more human interface design. Computer 34(3):56–64 CrossRefGoogle Scholar
  24. 24.
    Lank E, Thorley JS, Chen S, Blostein D (2001) On-line recognition of uml diagrams. In: Proceedings of the sixth international conference on document analysis and recognition (ICDAR’01), Seattle, USA, pp 356–360 CrossRefGoogle Scholar
  25. 25.
    Macé S, Anquetil E (2009) Eager interpretation of on-line hand-drawn structured documents: the Dali methodology. Pattern Recognit 42(12):3202–3214 zbMATHCrossRefGoogle Scholar
  26. 26.
    Macé S, Anquetil E, Bossis B (2007) Pen-based interaction for intuitive music composition and editing. In: Shen J, Shepherd J, Cui B, Liu L (eds) Intelligent music information systems: tools and methodologies, Idea Group, pp 261–288 Google Scholar
  27. 27.
    Mankoff J, Hudson SE, Abowd GD (2000) Oops: a toolkit supporting mediation techniques for resolving ambiguity in recognition-based interfaces. Comput Graph 24(6):819–834 CrossRefGoogle Scholar
  28. 28.
    Marriott K (1994) Constrain multiset grammars. In: Proceedings of the IEEE symposium on visual languages (VL’94), St Louis, MO, USA, pp 118–125 CrossRefGoogle Scholar
  29. 29.
    Marriott K, Meyer B, Wittenburg K (1998) A survey of visual language specification and recognition. In: Marriott K, Meyer B (eds) Theory of visual languages. Springer, Berlin CrossRefGoogle Scholar
  30. 30.
    Mas J, Sánchez G, Lladós J, Lamiroy B (2007) An incremental on-line parsing algorithm for recognizing sketching diagrams. In: Proceedings of the ninth international conference on document analysis and recognition (ICDAR’07), Curitiba, Brazil, pp 452–456 CrossRefGoogle Scholar
  31. 31.
    Mas J, Lladós J, Sánchez G, Jorge JA (2010) A syntactic approach based on distortion-tolerant adjacency grammars and a spatial-directed parser to interpret sketched diagrams. Pattern Recognit 43:4148–4164 zbMATHCrossRefGoogle Scholar
  32. 32.
    Miyao H, Maruyama M (2007) An online handwritten music symbol recognition system. Int J Doc Anal Recognit 9(1):49–58 CrossRefGoogle Scholar
  33. 33.
    Mouchère H, Anquetil E (2006) A unified strategy to deal with different natures of reject. In: Proceedings of the international conference on pattern recognition (ICPR’06), Hong Kong, China, pp 792–795 Google Scholar
  34. 34.
    Nakagawa M, Machii K, Kato N, Souya T (1993) Lazy recognition as a principle of pen interfaces. In: Proceedings of the conference companion on human factors in computing systems (INTERACT’93 and CHI’93), Amsterdam, The Netherlands, pp 89–90 CrossRefGoogle Scholar
  35. 35.
    Plimmer B, Apperley M (2003) Software to sketch interface designs. In: Proceedings of the ninth international conference on human-computer interaction (INTERACT’03), Zürich, Switzerland, pp 73–80 Google Scholar
  36. 36.
    Sezgin TM, Davis R (2007) Sketch interpretation using multiscale models of temporal patterns. IEEE Comput Graph Appl 27(1):28–37 CrossRefGoogle Scholar
  37. 37.
    Sezgin TM, Stahovich T, Davis R (2001) Sketch based interfaces: Early processing for sketch understanding. In: Proceedings of international conference on computer graphics and interactive techniques (ICCGIT’01), San Diego, California, USA, pp 37–44 Google Scholar
  38. 38.
    Shilman M, Pasula H, Russell S, Newton R (2002) Statistical visual language models for ink parsing. In: AAAI spring 2002 symposium on sketch understanding Google Scholar
  39. 39.
    Thimbleby W (2004) A novel pen-based calculator and its evaluation. In: Proceedings of the third Nordic conference on human-computer interaction, Tempere, Finland, pp 445–448 CrossRefGoogle Scholar
  40. 40.
    Valois JP, Cote M, Cheriet M (2001) Online recognition of sketched electrical diagrams. In: Proceedings of the sixth international conference on document analysis and recognition (ICDAR’01), Seattle, USA, pp 460–464 CrossRefGoogle Scholar
  41. 41.
    Zadeh LA (1994) Soft computing and fuzzy logic. IEEE Softw 11(6):48–56 CrossRefGoogle Scholar
  42. 42.
    Zhao R (1993) Incremental recognition in gesture-based and syntax-directed diagram editors. In: Proceedings of the INTERACT ’93 and CHI ’93 conference on human factors in computing systems (CHI’93), Amsterdam, The Netherlands, pp 95–100 CrossRefGoogle Scholar

Copyright information

© OpenInterface Association 2011

Authors and Affiliations

  1. 1.INSA–IRISA–UEBRennes CedexFrance

Personalised recommendations