Interactive Collaborative Information Systems pp 273-297 | Cite as
Design Issues for Pen-Centric Interactive Maps
Abstract
Recent advances in interactive pen-aware systems, pattern recognition technologies, and human–computer interaction have provided new opportunities for pen-based communication between human users and intelligent computer systems. Using interactive maps, users can annotate pictorial or cartographic information by means of pen gestures and handwriting. Interactive maps may provide an efficient means of communication, in particular in the envisaged contexts of crisis management scenarios, which require robust and effective exchange of information. This information contains, e.g., the location of objects, the kind of incidents, or the indication of route alternatives. When considering human interactions in these contexts, various pen input modes are involved, like handwriting, drawing, and sketching. How to design the required technology for grasping the intentions of the user based on these pen inputs remains an elusive challenge, which is discussed in this chapter. Aspects like the design of a suitable set of pen gestures, data collection in the context of the envisaged scenarios, and the development of distinguishing features and pattern recognition technologies for robustly recognizing pen input from varying modes are described. These aspects are illustrated by presenting our recent results on the development of interactive maps within the framework of the ICIS project on crisis management systems.
Keywords
Support Vector Machine Gesture Recognition Dynamic Time Warping Crisis Management Multimodal InterfacePreview
Unable to display preview. Download preview PDF.
References
- 1.Homeland Security Working Group, Symbology Reference, version 2.20 (released September 14, 2005)Google Scholar
- 2.Agarawala, A., Balakrishnan, R.: Keepin’ it real: Pushing the desktop metaphor with physics, piles and the pen. In: CHI 2006 - the ACM Conf. on Human Factors in Computing Systems, pp. 1283–1292 (2006)Google Scholar
- 3.Alvarado, C., Davis, R.: SketchREAD: A multi-domain sketch recognition engine. In: Proc. of the 17th annual ACM symposium on user interface software and technology, Santa Fe, New Mexico, USA, pp. 23–32 (2004)Google Scholar
- 4.Benoit, C., Martin, J.-C., Pelachaud, C., Schomaker, L., Suhm, B.: Audio-visual and multimodal speech-based systems. In: Gibbon, D., Mertins, I., Moore, R. (eds.) Handbook of multimodal and spoken dialogue systems: Resources, terminology and product evaluation, pp. 102–203. Kluwer Academic Publishers, Dordrecht (2000)Google Scholar
- 5.Bhaskarabhatla, A.S., Madhvanath, S.: Experiences in collection of handwriting data for online handwriting recognition in Indic scripts. In: Proc. of the 4th Int. Conf. Linguistic Resources and Evaluation (LREC), CDROM (2004)Google Scholar
- 6.Bhattacharya, U.: Handwritten character databases of Indic scripts (2004), http://www.isical.ac.in/~ujjwal/download/database.html
- 7.Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995)Google Scholar
- 8.Boves, L., Neumann, A., Vuurpijl, L., ten Bosch, L., Rossignol, S., Engel, R., Pfleger, N.: Multimodal interaction in architectural design applications. In: 8th ERCIM Workshop on User Interfaces for AI, Vienna, Austria (2004)Google Scholar
- 9.Bui, T.: Toward affective dialogue management using partially observable Markov decision processes. PhD thesis, University of Twente, Enschede (October 2008)Google Scholar
- 10.Buxton, W.: Chunking and phrasing and the design of human-computer dialogues. In: Proc. of the IFIP World Computer Congress, Dublin, Ireland, pp. 475–480 (1986)Google Scholar
- 11.Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vctor machines (2001), http://www.csie.ntu.edu.tw/~cjlin/libsvm
- 12.Cheriet, M., El Yacoubi, M., Fujisawa, H., Lopresti, D., Lorette, G.: Handwriting recognition research: Twenty years of achievement ...and beyond. Pattern Recognition 42(12), 3131–3135 (2009)CrossRefGoogle Scholar
- 13.Cohen, P.R., Johnston, M., McGee, D., Smith, I., Oviatt, S., Pittman, J., Chen, L., Clow, J.: Quickset: Multimodal interaction for simulation set-up and control. In: Proc. of the Fifth Applied Natural Language Processing meeting, Association for Computational Linguistics (1997)Google Scholar
- 14.Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: Multimodal interaction for distributed applications. In: Proc. of the Fifth ACM Int. Multimedia Conf., pp. 31–40 (1997)Google Scholar
- 15.Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: multimodal interaction for distributed applications. In: MULTIMEDIA 1997: Proc. of the fifth ACM international conference on Multimedia, pp. 31–40. ACM, New York (1997)CrossRefGoogle Scholar
- 16.Datcu, D., Rothkrantz, L.: A dialog action manager for automatic crisis management. In: Proc. of the 6th Int. Conf. on Information Systems for Crisis Response and Management (ISCRAM2008), pp. 384–393 (2008)Google Scholar
- 17.Datcu, D., Rothkrantz, L.: Semantic audio-visual data fusion for automatic emotion recognition. In: Euromedia 2008, April 2008, pp. 1–6. Eurosis, Ghent (2008)Google Scholar
- 18.den Os, E., Boves, L., Rossignol, S., ten Bosch, L., Vuurpijl, L.: Conversational agent or direct manipulation in human system interaction. Speech Communication 47, 194–207 (2005)CrossRefGoogle Scholar
- 19.Doyle, J., Bertolotto, M., Wilson, D.: A survey of multimodal interfaces for mobile mapping applications. In: Meng, L., Zipf, A., Winter, S. (eds.) Map-based Mobile Services. Lecture Notes in Geoinformation and Cartography, pp. 146–167. Springer, Heidelberg (2008)CrossRefGoogle Scholar
- 20.Egger, M.: Find new meaning in your ink with tablet PC APIs in Windows Vista. MSDN Magazine, Microsoft Corporation (May 2006)Google Scholar
- 21.Engel, R., Pfleger, N.: Modality Fusion. Cognitive Technologies, pp. 223–235. Springer, Heidelberg (2006)Google Scholar
- 22.Fitrianie, S., Poppe, R., Bui, T.H., Chitu, A.G., Datcu, D., Dor, R., Hofs, D.H.W., Wiggers, P., Willems, D.J.M., Poel, M., Rothkrantz, L.J.M., Vuurpijl, L.G., Zwiers, J.: Multimodal human-computer interaction in crisis management environments. In: Burghardt, P., Van de Walle, B., Nieuwenhuis, C. (eds.) Proc. of the 4th Int. Conf. on Information Systems for Crisis Response and Management ISCRAM 2007, pp. 149–158 (2007)Google Scholar
- 23.Fitrianie, S., Rothkrantz, L.: Language-independent communication using icons on a PDA. In: Matoušek, V., Mautner, P., Pavelka, T. (eds.) TSD 2005. LNCS (LNAI), vol. 3658, pp. 404–411. Springer, Heidelberg (2005)Google Scholar
- 24.Fitrianie, S., Rothkrantz, L.: A visual communication language for crisis management. Int. Journal of Intelligent Control and Systems (Special Issue of Distributed Intelligent Systems) 12(2), 208–216 (2007)Google Scholar
- 25.Fitrianie, S., Tatomir, I., Rothkranz, L.: A context aware and user tailored multimodal information generation in a multimodal HCI framework. In: Euromedia 2008. Eurosis, Ghent (2008)Google Scholar
- 26.Fukunaga, K., Hostetler, L.: The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Transactions on Information Theory 21, 32–40 (1975)zbMATHCrossRefMathSciNetGoogle Scholar
- 27.Gennari, L., Kara, L.B., Stahovich, T.F., Shimada, K.: Combining geometry and domain knowledge to interpret hand-drawn diagrams. Computers and Graphics 29, 547–562 (2005)CrossRefGoogle Scholar
- 28.Guyon, I., Schomaker, L., Plamondon, R., Liberman, M., Janet, S.: UNIPEN project of on-line data exchange and recognizer benchmarks. In: Proc. ICPR 1994, October 1994, pp. 29–33 (1994)Google Scholar
- 29.Hammond, T., Davis, R.: Tahuti: A geometrical sketch recognition system for UML class diagrams. Papers from the 2002 AAAI Spring Symposium on Sketch Understanding, Stanford, California, USA, pp. 59–68. AAAI Press, Menlo Park (2002)Google Scholar
- 30.Hull, J.: A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(5), 550–554 (1994)CrossRefGoogle Scholar
- 31.Iivarinen, J., Peura, M., Särelä, J., Visa, A.: Comparison of combined shape descriptors for irregular objects. In: Clark, A.F. (ed.) 8th British Machine Vision Conf., BMVC 1997, Essex, UK, pp. 430–439 (1997)Google Scholar
- 32.Jaeger, S., Nakagawa, M.: Two on-line Japanese character databases in Unipen format. In: Proc. Intl. Conf. Document Analysis and Recognition, ICDAR 2001, pp. 566–570. IEEE Computer Society, Los Alamitos (2001)CrossRefGoogle Scholar
- 33.Jain, A., Duin, R., Mao, J.: Statistical pattern recognition: A review. IEEE Trans. PAMI 22(1), 4–37 (2000)Google Scholar
- 34.Kara, L.B., Stahovich, T.F.: An image-based trainable symbol recognizer for hand-drawn sketches. Computers and Graphics 29, 501–517 (2005)CrossRefGoogle Scholar
- 35.Kettebekov, S., Krahnstöver, N., Leas, M., Polat, E., Raju, H., Schapira, E., Sharma, R.: i2map: Crisis management using a multimodal interface. In: ARL Federate Laboratory 4th Annual Symposium, College Park, MD (March 2000)Google Scholar
- 36.Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Trans. PAMI 20(3) (1998)Google Scholar
- 37.Landay, J.A., Dannenberg, R.B.: Interactive sketching for the early stages of user interface design. In: CHI 1995 Computer Human Interaction, pp. 43–50 (1995)Google Scholar
- 38.LaViola Jr., J., Zeleznik, R.C.: A practical approach for writer-dependent symbol recognition using a writer-independent symbol recognizer. IEEE Transactions on pattern analysis and machine intelligence 29(11), 1917–1926 (2007)CrossRefGoogle Scholar
- 39.Lipscomb, J.: A trainable gesture recognizer. Pattern Recognition 24(9), 895–907 (1991)CrossRefGoogle Scholar
- 40.Long, C., Landay, J., Rowe, L., Michiels, J.: Visual similarity of pen gestures. In: CHI 2000: Proc. of the SIGCHI conference on Human factors in computing systems, pp. 360–367 (2000)Google Scholar
- 41.Marti, U., Bunke, H.: A full English sentence database for off-line handwriting recognition. In: Proc. of the 5th Int. Conf. on Document Analysis and Recognition (ICDAR 1999), Bangalore, India, pp. 705–708 (1999)Google Scholar
- 42.Miyao, H., Maruyama, M.: An online handwritten music score recognition system. In: 17th Int. Conf. on Pattern Recognition (ICPR 2004), vol. 1, pp. 461–464 (2004)Google Scholar
- 43.Montello, D.R.: Spatial cognition. In: Smelser, N.J., Baltes, P.B. (eds.) Int. Encyclopedia of the Social & Behavioral Sciences, pp. 14771–14775. Oxford Pergamon Press, Oxford (2001)Google Scholar
- 44.Nakagawa, M., Matsumoto, K.: Collection of on-line handwritten Japanese character pattern databases and their analysis. Int. Journal on Document Analysis and Recognition 7(1), 69–81 (2004)Google Scholar
- 45.Niels, R., Willems, D., Vuurpijl, L.: The NicIcon collection of handwritten icons. In: ICFHR8, the 11th Int. Conf. on Frontiers of Handwriting Recognition, Montreal, Canada, August 2008, pp. 296–301 (2008)Google Scholar
- 46.Niels, R., Vuurpijl, L., Schomaker, L.: Automatic allograph matching in forensic writer identification. Int. Journal of Pattern Recognition and Artificial Intelligence 21(1), 61–81 (2007)CrossRefGoogle Scholar
- 47.Oviatt, S.L.: Multimodal interfaces. In: Jacko, J., Sears, A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, ch. 14, pp. 286–304. Lawrence Erlbaum Assoc., Mahwah (2003)Google Scholar
- 48.Pechwitz, M., Maddouri, S., Märgner, V., Ellouze, N., Amiri, H.: Ifn/enit-database of handwritten Arabic words. In: 7th Colloque Int. Francophone sur l’Ecrit et le Document (CIFED 2002), Hammamet, Tunis, October 21-23, pp. 1–8 (2002)Google Scholar
- 49.Peura, M., Iivarinen, J.: Efficiency of simple shape descriptors. In: Arcelli, L.P., Cordella, C., Sanniti di Baja, G. (eds.) Advances in Visual Form Analysis, pp. 443–451. World Scientific, Singapore (1997)Google Scholar
- 50.Piramuthu, S.: Evaluating feature selection methods for learning in data mining applications. European Journal of Operational Research 156, 483–494 (2004)zbMATHCrossRefGoogle Scholar
- 51.Plamondon, R., Srihari, S.: On-line and off-line handwriting recognition: A comprehensive survey. IEEE PAMI 22(1), 63–84 (2000)Google Scholar
- 52.Poel, M., Poppe, R.W., Nijholt, A.: Meeting behavior detection in smart environments: Nonverbal cues that help to obtain natural interaction. In: Cohn, J., Huang, T.S., Pantic, M., Sebe, N. (eds.) Proc. of the IEEE Int. Conf. on Automatic face and Gesture Recognition (FGR 2008), Amsterdam, The Netherlands, September 2008, pp. 1–6. IEEE Computer Society Press, Los Alamitos (2008)CrossRefGoogle Scholar
- 53.Poppe, R.W.: Discriminative Vision-Based Recovery and Recognition of Human Motion. PhD thesis, University of Twente (April 2009)Google Scholar
- 54.Rubine, D.: Specifying gestures by example. Computer Graphics 25(4), 329–337 (1991)CrossRefGoogle Scholar
- 55.Schomaker, L.R.B.: From handwriting analysis to pen-computer applications. IEE Electronics Communication Engineering Journal 10(3), 93–102 (1998)CrossRefGoogle Scholar
- 56.Schomaker, L.R.B., Teulings, H.-L.: A handwriting recognition system based on properties of the human motor system. In: Proc. of the First Int. Workshop on Frontiers in Handwriting Recognition (IWFHR), Montreal, Canada, pp. 195–211 (1990)Google Scholar
- 57.Sharma, R., Yeasin, M., Krahnstöver, N., Rauschert, I., Cai, G., Brewer, I., Maceachren, A.M., Sengupta, K.: Speech-gesture driven multimodal interfaces for crisis management. In: Proc. of the IEEE, pp. 1327–1354 (2003)Google Scholar
- 58.The ICIS project. Interactive collaborative information systems ICIS (2004), http://www.icis.decis.nl/
- 59.Tian, F., Cheng, T., Wang, H., Dai, G.: Research on User-Centered Design and Recognition of Pen Gestures. In: Nishita, T., Peng, Q., Seidel, H.-P. (eds.) CGI 2006. LNCS, vol. 4035, pp. 312–323. Springer, Heidelberg (2006)CrossRefGoogle Scholar
- 60.van Erp, M., Vuurpijl, L., Schomaker, L.: An overview and comparison of voting methods for pattern recognition. In: Proc. of IWFHR8, Niagara-on-the-Lake, Canada, August 2002, pp. 195–200 (2002)Google Scholar
- 61.Vapnik, V.: The nature of statistical learning theory. Springer, Berlin (1995)zbMATHGoogle Scholar
- 62.Viard-Gaudin, C., Lallican, P.M., Binter, P., Knerr, S.: The IRESTE On/Off (IRONOFF) dual handwriting database. In: Proc. Intl Conf. Document Analysis and Recognition, ICDAR 1999, Bangalore, India, pp. 455–458 (1999)Google Scholar
- 63.Vinciarelli, A.: A survey on off-line cursive script recognition. Pattern Recognition 35(7), 1433–1446 (2002)zbMATHCrossRefGoogle Scholar
- 64.Vuurpijl, L., Schomaker, L.: Finding structure in diversity: A hierarchical clustering method for the categorization of allographs in handwriting. In: Proc. ICDAR4, pp. 387–393. IEEE Computer Society, Los Alamitos (1997)Google Scholar
- 65.Wahlster, W. (ed.): SmartKom: Foundations of Multimodal Dialogue Systems. Springer, Berlin (2006)Google Scholar
- 66.Webb, A.R.: Feature selection and extraction. In: Statistical Pattern Recognition, 2nd edn., ch. 9, pp. 305–359. John Wiley & Sons Ltd., Chichester (2002)Google Scholar
- 67.Willems, D.M., Niels, R., van Gerven, M., Vuurpijl, L.: Iconic and multi-stroke gesture recognition. Pattern Recognition 42(12), 3303–3312 (2009)zbMATHCrossRefGoogle Scholar
- 68.Willems, D., Vuurpijl, L.: Pen gestures in online map and photograph annotation tasks. In: Proc.of the tenth Int. Workshop on Frontiers in Handwriting Recognition (IWFHR 2006), La Baule, France, October 2006, pp. 297–402 (2006)Google Scholar
- 69.Willems, D., Vuurpijl, L.: Designing interactive maps for crisis management. In: Proc. of the 4th Int. Conf. on Information Systems for Crisis Response and Management (ISCRAM 2007), pp. 159–166 (2007)Google Scholar
- 70.Willems, D.J.M.: Interactive Maps: using the pen in human-computer interaction. PhD thesis, Donders Institute for Brain, Cognition and Behavior, Radboud University Nijmegen, The Netherlands (in press)Google Scholar
- 71.Willems, D.J.M., Niels, R.: Definitions for features used in online pen gesture recognition. Technical report, NICI, Radboud University Nijmegen (2008)Google Scholar
- 72.Willems, D.J.M., Rossignol, S.Y.P., Vuurpijl, L.: Mode detection in online pen drawing and handwriting recognition. In: Proc. of the 8th Int. Conf. on Document Analysis and Recognition (ICDAR 2005), Seoul, Korea, pp. 31–35 (2005)Google Scholar
- 73.Willems, D.J.M., Vuurpijl, L.: A Bayesian network approach to mode detection for interactive maps. In: Proc. of the 9th Int. Conf. on Document Analysis and Recognition (ICDAR 2007), Curitiba, Brazil, pp. 869–873 (2007)Google Scholar
- 74.Willems, D.J.M., Vuurpijl, L.G.: Pen gestures in online map and photograph annotation tasks. In: Proc. of the Tenth Int. Workshop on Frontiers in Handwriting Recognition, pp. 397–402 (2006)Google Scholar
- 75.Zeleznik, R., van Dam, A., Li, C., Tenneson, D., Maloney, C., LaViola Jr., J.: Applications and issues in pen-centric computing. IEEE Multimedia 14(4), 14–21 (2008)Google Scholar
- 76.Zhang, L., Sun, Z.: An experimental comparison of machine learning for adaptive sketch recognition. Applied Mathematics and Computation 185(2), 1138–1148 (2007)zbMATHCrossRefGoogle Scholar