Advertisement

Design Issues for Pen-Centric Interactive Maps

  • Louis Vuurpijl
  • Don Willems
  • Ralph Niels
  • Marcel van Gerven
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 281)

Abstract

Recent advances in interactive pen-aware systems, pattern recognition technologies, and human–computer interaction have provided new opportunities for pen-based communication between human users and intelligent computer systems. Using interactive maps, users can annotate pictorial or cartographic information by means of pen gestures and handwriting. Interactive maps may provide an efficient means of communication, in particular in the envisaged contexts of crisis management scenarios, which require robust and effective exchange of information. This information contains, e.g., the location of objects, the kind of incidents, or the indication of route alternatives. When considering human interactions in these contexts, various pen input modes are involved, like handwriting, drawing, and sketching. How to design the required technology for grasping the intentions of the user based on these pen inputs remains an elusive challenge, which is discussed in this chapter. Aspects like the design of a suitable set of pen gestures, data collection in the context of the envisaged scenarios, and the development of distinguishing features and pattern recognition technologies for robustly recognizing pen input from varying modes are described. These aspects are illustrated by presenting our recent results on the development of interactive maps within the framework of the ICIS project on crisis management systems.

Keywords

Support Vector Machine Gesture Recognition Dynamic Time Warping Crisis Management Multimodal Interface 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Homeland Security Working Group, Symbology Reference, version 2.20 (released September 14, 2005)Google Scholar
  2. 2.
    Agarawala, A., Balakrishnan, R.: Keepin’ it real: Pushing the desktop metaphor with physics, piles and the pen. In: CHI 2006 - the ACM Conf. on Human Factors in Computing Systems, pp. 1283–1292 (2006)Google Scholar
  3. 3.
    Alvarado, C., Davis, R.: SketchREAD: A multi-domain sketch recognition engine. In: Proc. of the 17th annual ACM symposium on user interface software and technology, Santa Fe, New Mexico, USA, pp. 23–32 (2004)Google Scholar
  4. 4.
    Benoit, C., Martin, J.-C., Pelachaud, C., Schomaker, L., Suhm, B.: Audio-visual and multimodal speech-based systems. In: Gibbon, D., Mertins, I., Moore, R. (eds.) Handbook of multimodal and spoken dialogue systems: Resources, terminology and product evaluation, pp. 102–203. Kluwer Academic Publishers, Dordrecht (2000)Google Scholar
  5. 5.
    Bhaskarabhatla, A.S., Madhvanath, S.: Experiences in collection of handwriting data for online handwriting recognition in Indic scripts. In: Proc. of the 4th Int. Conf. Linguistic Resources and Evaluation (LREC), CDROM (2004)Google Scholar
  6. 6.
    Bhattacharya, U.: Handwritten character databases of Indic scripts (2004), http://www.isical.ac.in/~ujjwal/download/database.html
  7. 7.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995)Google Scholar
  8. 8.
    Boves, L., Neumann, A., Vuurpijl, L., ten Bosch, L., Rossignol, S., Engel, R., Pfleger, N.: Multimodal interaction in architectural design applications. In: 8th ERCIM Workshop on User Interfaces for AI, Vienna, Austria (2004)Google Scholar
  9. 9.
    Bui, T.: Toward affective dialogue management using partially observable Markov decision processes. PhD thesis, University of Twente, Enschede (October 2008)Google Scholar
  10. 10.
    Buxton, W.: Chunking and phrasing and the design of human-computer dialogues. In: Proc. of the IFIP World Computer Congress, Dublin, Ireland, pp. 475–480 (1986)Google Scholar
  11. 11.
    Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vctor machines (2001), http://www.csie.ntu.edu.tw/~cjlin/libsvm
  12. 12.
    Cheriet, M., El Yacoubi, M., Fujisawa, H., Lopresti, D., Lorette, G.: Handwriting recognition research: Twenty years of achievement ...and beyond. Pattern Recognition 42(12), 3131–3135 (2009)CrossRefGoogle Scholar
  13. 13.
    Cohen, P.R., Johnston, M., McGee, D., Smith, I., Oviatt, S., Pittman, J., Chen, L., Clow, J.: Quickset: Multimodal interaction for simulation set-up and control. In: Proc. of the Fifth Applied Natural Language Processing meeting, Association for Computational Linguistics (1997)Google Scholar
  14. 14.
    Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: Multimodal interaction for distributed applications. In: Proc. of the Fifth ACM Int. Multimedia Conf., pp. 31–40 (1997)Google Scholar
  15. 15.
    Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: multimodal interaction for distributed applications. In: MULTIMEDIA 1997: Proc. of the fifth ACM international conference on Multimedia, pp. 31–40. ACM, New York (1997)CrossRefGoogle Scholar
  16. 16.
    Datcu, D., Rothkrantz, L.: A dialog action manager for automatic crisis management. In: Proc. of the 6th Int. Conf. on Information Systems for Crisis Response and Management (ISCRAM2008), pp. 384–393 (2008)Google Scholar
  17. 17.
    Datcu, D., Rothkrantz, L.: Semantic audio-visual data fusion for automatic emotion recognition. In: Euromedia 2008, April 2008, pp. 1–6. Eurosis, Ghent (2008)Google Scholar
  18. 18.
    den Os, E., Boves, L., Rossignol, S., ten Bosch, L., Vuurpijl, L.: Conversational agent or direct manipulation in human system interaction. Speech Communication 47, 194–207 (2005)CrossRefGoogle Scholar
  19. 19.
    Doyle, J., Bertolotto, M., Wilson, D.: A survey of multimodal interfaces for mobile mapping applications. In: Meng, L., Zipf, A., Winter, S. (eds.) Map-based Mobile Services. Lecture Notes in Geoinformation and Cartography, pp. 146–167. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Egger, M.: Find new meaning in your ink with tablet PC APIs in Windows Vista. MSDN Magazine, Microsoft Corporation (May 2006)Google Scholar
  21. 21.
    Engel, R., Pfleger, N.: Modality Fusion. Cognitive Technologies, pp. 223–235. Springer, Heidelberg (2006)Google Scholar
  22. 22.
    Fitrianie, S., Poppe, R., Bui, T.H., Chitu, A.G., Datcu, D., Dor, R., Hofs, D.H.W., Wiggers, P., Willems, D.J.M., Poel, M., Rothkrantz, L.J.M., Vuurpijl, L.G., Zwiers, J.: Multimodal human-computer interaction in crisis management environments. In: Burghardt, P., Van de Walle, B., Nieuwenhuis, C. (eds.) Proc. of the 4th Int. Conf. on Information Systems for Crisis Response and Management ISCRAM 2007, pp. 149–158 (2007)Google Scholar
  23. 23.
    Fitrianie, S., Rothkrantz, L.: Language-independent communication using icons on a PDA. In: Matoušek, V., Mautner, P., Pavelka, T. (eds.) TSD 2005. LNCS (LNAI), vol. 3658, pp. 404–411. Springer, Heidelberg (2005)Google Scholar
  24. 24.
    Fitrianie, S., Rothkrantz, L.: A visual communication language for crisis management. Int. Journal of Intelligent Control and Systems (Special Issue of Distributed Intelligent Systems) 12(2), 208–216 (2007)Google Scholar
  25. 25.
    Fitrianie, S., Tatomir, I., Rothkranz, L.: A context aware and user tailored multimodal information generation in a multimodal HCI framework. In: Euromedia 2008. Eurosis, Ghent (2008)Google Scholar
  26. 26.
    Fukunaga, K., Hostetler, L.: The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Transactions on Information Theory 21, 32–40 (1975)zbMATHCrossRefMathSciNetGoogle Scholar
  27. 27.
    Gennari, L., Kara, L.B., Stahovich, T.F., Shimada, K.: Combining geometry and domain knowledge to interpret hand-drawn diagrams. Computers and Graphics 29, 547–562 (2005)CrossRefGoogle Scholar
  28. 28.
    Guyon, I., Schomaker, L., Plamondon, R., Liberman, M., Janet, S.: UNIPEN project of on-line data exchange and recognizer benchmarks. In: Proc. ICPR 1994, October 1994, pp. 29–33 (1994)Google Scholar
  29. 29.
    Hammond, T., Davis, R.: Tahuti: A geometrical sketch recognition system for UML class diagrams. Papers from the 2002 AAAI Spring Symposium on Sketch Understanding, Stanford, California, USA, pp. 59–68. AAAI Press, Menlo Park (2002)Google Scholar
  30. 30.
    Hull, J.: A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(5), 550–554 (1994)CrossRefGoogle Scholar
  31. 31.
    Iivarinen, J., Peura, M., Särelä, J., Visa, A.: Comparison of combined shape descriptors for irregular objects. In: Clark, A.F. (ed.) 8th British Machine Vision Conf., BMVC 1997, Essex, UK, pp. 430–439 (1997)Google Scholar
  32. 32.
    Jaeger, S., Nakagawa, M.: Two on-line Japanese character databases in Unipen format. In: Proc. Intl. Conf. Document Analysis and Recognition, ICDAR 2001, pp. 566–570. IEEE Computer Society, Los Alamitos (2001)CrossRefGoogle Scholar
  33. 33.
    Jain, A., Duin, R., Mao, J.: Statistical pattern recognition: A review. IEEE Trans. PAMI 22(1), 4–37 (2000)Google Scholar
  34. 34.
    Kara, L.B., Stahovich, T.F.: An image-based trainable symbol recognizer for hand-drawn sketches. Computers and Graphics 29, 501–517 (2005)CrossRefGoogle Scholar
  35. 35.
    Kettebekov, S., Krahnstöver, N., Leas, M., Polat, E., Raju, H., Schapira, E., Sharma, R.: i2map: Crisis management using a multimodal interface. In: ARL Federate Laboratory 4th Annual Symposium, College Park, MD (March 2000)Google Scholar
  36. 36.
    Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Trans. PAMI 20(3) (1998)Google Scholar
  37. 37.
    Landay, J.A., Dannenberg, R.B.: Interactive sketching for the early stages of user interface design. In: CHI 1995 Computer Human Interaction, pp. 43–50 (1995)Google Scholar
  38. 38.
    LaViola Jr., J., Zeleznik, R.C.: A practical approach for writer-dependent symbol recognition using a writer-independent symbol recognizer. IEEE Transactions on pattern analysis and machine intelligence 29(11), 1917–1926 (2007)CrossRefGoogle Scholar
  39. 39.
    Lipscomb, J.: A trainable gesture recognizer. Pattern Recognition 24(9), 895–907 (1991)CrossRefGoogle Scholar
  40. 40.
    Long, C., Landay, J., Rowe, L., Michiels, J.: Visual similarity of pen gestures. In: CHI 2000: Proc. of the SIGCHI conference on Human factors in computing systems, pp. 360–367 (2000)Google Scholar
  41. 41.
    Marti, U., Bunke, H.: A full English sentence database for off-line handwriting recognition. In: Proc. of the 5th Int. Conf. on Document Analysis and Recognition (ICDAR 1999), Bangalore, India, pp. 705–708 (1999)Google Scholar
  42. 42.
    Miyao, H., Maruyama, M.: An online handwritten music score recognition system. In: 17th Int. Conf. on Pattern Recognition (ICPR 2004), vol. 1, pp. 461–464 (2004)Google Scholar
  43. 43.
    Montello, D.R.: Spatial cognition. In: Smelser, N.J., Baltes, P.B. (eds.) Int. Encyclopedia of the Social & Behavioral Sciences, pp. 14771–14775. Oxford Pergamon Press, Oxford (2001)Google Scholar
  44. 44.
    Nakagawa, M., Matsumoto, K.: Collection of on-line handwritten Japanese character pattern databases and their analysis. Int. Journal on Document Analysis and Recognition 7(1), 69–81 (2004)Google Scholar
  45. 45.
    Niels, R., Willems, D., Vuurpijl, L.: The NicIcon collection of handwritten icons. In: ICFHR8, the 11th Int. Conf. on Frontiers of Handwriting Recognition, Montreal, Canada, August 2008, pp. 296–301 (2008)Google Scholar
  46. 46.
    Niels, R., Vuurpijl, L., Schomaker, L.: Automatic allograph matching in forensic writer identification. Int. Journal of Pattern Recognition and Artificial Intelligence 21(1), 61–81 (2007)CrossRefGoogle Scholar
  47. 47.
    Oviatt, S.L.: Multimodal interfaces. In: Jacko, J., Sears, A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, ch. 14, pp. 286–304. Lawrence Erlbaum Assoc., Mahwah (2003)Google Scholar
  48. 48.
    Pechwitz, M., Maddouri, S., Märgner, V., Ellouze, N., Amiri, H.: Ifn/enit-database of handwritten Arabic words. In: 7th Colloque Int. Francophone sur l’Ecrit et le Document (CIFED 2002), Hammamet, Tunis, October 21-23, pp. 1–8 (2002)Google Scholar
  49. 49.
    Peura, M., Iivarinen, J.: Efficiency of simple shape descriptors. In: Arcelli, L.P., Cordella, C., Sanniti di Baja, G. (eds.) Advances in Visual Form Analysis, pp. 443–451. World Scientific, Singapore (1997)Google Scholar
  50. 50.
    Piramuthu, S.: Evaluating feature selection methods for learning in data mining applications. European Journal of Operational Research 156, 483–494 (2004)zbMATHCrossRefGoogle Scholar
  51. 51.
    Plamondon, R., Srihari, S.: On-line and off-line handwriting recognition: A comprehensive survey. IEEE PAMI 22(1), 63–84 (2000)Google Scholar
  52. 52.
    Poel, M., Poppe, R.W., Nijholt, A.: Meeting behavior detection in smart environments: Nonverbal cues that help to obtain natural interaction. In: Cohn, J., Huang, T.S., Pantic, M., Sebe, N. (eds.) Proc. of the IEEE Int. Conf. on Automatic face and Gesture Recognition (FGR 2008), Amsterdam, The Netherlands, September 2008, pp. 1–6. IEEE Computer Society Press, Los Alamitos (2008)CrossRefGoogle Scholar
  53. 53.
    Poppe, R.W.: Discriminative Vision-Based Recovery and Recognition of Human Motion. PhD thesis, University of Twente (April 2009)Google Scholar
  54. 54.
    Rubine, D.: Specifying gestures by example. Computer Graphics 25(4), 329–337 (1991)CrossRefGoogle Scholar
  55. 55.
    Schomaker, L.R.B.: From handwriting analysis to pen-computer applications. IEE Electronics Communication Engineering Journal 10(3), 93–102 (1998)CrossRefGoogle Scholar
  56. 56.
    Schomaker, L.R.B., Teulings, H.-L.: A handwriting recognition system based on properties of the human motor system. In: Proc. of the First Int. Workshop on Frontiers in Handwriting Recognition (IWFHR), Montreal, Canada, pp. 195–211 (1990)Google Scholar
  57. 57.
    Sharma, R., Yeasin, M., Krahnstöver, N., Rauschert, I., Cai, G., Brewer, I., Maceachren, A.M., Sengupta, K.: Speech-gesture driven multimodal interfaces for crisis management. In: Proc. of the IEEE, pp. 1327–1354 (2003)Google Scholar
  58. 58.
    The ICIS project. Interactive collaborative information systems ICIS (2004), http://www.icis.decis.nl/
  59. 59.
    Tian, F., Cheng, T., Wang, H., Dai, G.: Research on User-Centered Design and Recognition of Pen Gestures. In: Nishita, T., Peng, Q., Seidel, H.-P. (eds.) CGI 2006. LNCS, vol. 4035, pp. 312–323. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  60. 60.
    van Erp, M., Vuurpijl, L., Schomaker, L.: An overview and comparison of voting methods for pattern recognition. In: Proc. of IWFHR8, Niagara-on-the-Lake, Canada, August 2002, pp. 195–200 (2002)Google Scholar
  61. 61.
    Vapnik, V.: The nature of statistical learning theory. Springer, Berlin (1995)zbMATHGoogle Scholar
  62. 62.
    Viard-Gaudin, C., Lallican, P.M., Binter, P., Knerr, S.: The IRESTE On/Off (IRONOFF) dual handwriting database. In: Proc. Intl Conf. Document Analysis and Recognition, ICDAR 1999, Bangalore, India, pp. 455–458 (1999)Google Scholar
  63. 63.
    Vinciarelli, A.: A survey on off-line cursive script recognition. Pattern Recognition 35(7), 1433–1446 (2002)zbMATHCrossRefGoogle Scholar
  64. 64.
    Vuurpijl, L., Schomaker, L.: Finding structure in diversity: A hierarchical clustering method for the categorization of allographs in handwriting. In: Proc. ICDAR4, pp. 387–393. IEEE Computer Society, Los Alamitos (1997)Google Scholar
  65. 65.
    Wahlster, W. (ed.): SmartKom: Foundations of Multimodal Dialogue Systems. Springer, Berlin (2006)Google Scholar
  66. 66.
    Webb, A.R.: Feature selection and extraction. In: Statistical Pattern Recognition, 2nd edn., ch. 9, pp. 305–359. John Wiley & Sons Ltd., Chichester (2002)Google Scholar
  67. 67.
    Willems, D.M., Niels, R., van Gerven, M., Vuurpijl, L.: Iconic and multi-stroke gesture recognition. Pattern Recognition 42(12), 3303–3312 (2009)zbMATHCrossRefGoogle Scholar
  68. 68.
    Willems, D., Vuurpijl, L.: Pen gestures in online map and photograph annotation tasks. In: Proc.of the tenth Int. Workshop on Frontiers in Handwriting Recognition (IWFHR 2006), La Baule, France, October 2006, pp. 297–402 (2006)Google Scholar
  69. 69.
    Willems, D., Vuurpijl, L.: Designing interactive maps for crisis management. In: Proc. of the 4th Int. Conf. on Information Systems for Crisis Response and Management (ISCRAM 2007), pp. 159–166 (2007)Google Scholar
  70. 70.
    Willems, D.J.M.: Interactive Maps: using the pen in human-computer interaction. PhD thesis, Donders Institute for Brain, Cognition and Behavior, Radboud University Nijmegen, The Netherlands (in press)Google Scholar
  71. 71.
    Willems, D.J.M., Niels, R.: Definitions for features used in online pen gesture recognition. Technical report, NICI, Radboud University Nijmegen (2008)Google Scholar
  72. 72.
    Willems, D.J.M., Rossignol, S.Y.P., Vuurpijl, L.: Mode detection in online pen drawing and handwriting recognition. In: Proc. of the 8th Int. Conf. on Document Analysis and Recognition (ICDAR 2005), Seoul, Korea, pp. 31–35 (2005)Google Scholar
  73. 73.
    Willems, D.J.M., Vuurpijl, L.: A Bayesian network approach to mode detection for interactive maps. In: Proc. of the 9th Int. Conf. on Document Analysis and Recognition (ICDAR 2007), Curitiba, Brazil, pp. 869–873 (2007)Google Scholar
  74. 74.
    Willems, D.J.M., Vuurpijl, L.G.: Pen gestures in online map and photograph annotation tasks. In: Proc. of the Tenth Int. Workshop on Frontiers in Handwriting Recognition, pp. 397–402 (2006)Google Scholar
  75. 75.
    Zeleznik, R., van Dam, A., Li, C., Tenneson, D., Maloney, C., LaViola Jr., J.: Applications and issues in pen-centric computing. IEEE Multimedia 14(4), 14–21 (2008)Google Scholar
  76. 76.
    Zhang, L., Sun, Z.: An experimental comparison of machine learning for adaptive sketch recognition. Applied Mathematics and Computation 185(2), 1138–1148 (2007)zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Louis Vuurpijl
    • 1
  • Don Willems
    • 1
  • Ralph Niels
    • 1
  • Marcel van Gerven
    • 2
  1. 1.Donders Institute for Brain, Cognition and BehaviorRadboud University NijmegenNijmegenThe Netherlands
  2. 2.Institute for Computing and Information SciencesRadboud University NijmegenThe Netherlands

Personalised recommendations

Cite chapter

Buy options