Abstract
To bridge the gap between paper and electronic forms of documents, computers must be able to recognize and generate diagrams as well as text. Diagrams used in society are expressed in a variety of notations, which we call natural visual languages. Examples include notations used for mathematics, music, engineering drawings and architecture. These visual languages do not have fixed, formal definitions, but evolve through use in society. This paper examines the use of graph transformation in processing natural visual languages, describing the difficult problems in this domain, existing graph transformation work in this area, and competing methods. Many problems have not been adequately addressed by any technique. The use of graph transformation is appropriate, since the representation and manipulation of spatial and logical relationships is central to the computation.
This research is supported by Canada’s Natural Sciences and Engineering Research Council.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
F. Arefi, C. Hughes, D. Workman, “Automatically Generating Visual Syntax-Directed Editors,” Communications of the ACM, 33(3), March 1990, pp. 349–360.
S. Baumann, “A Simplified Attributed Graph Grammar for High-Level Music Recognition,” Proc. Third Intl. Conf. on Document Analysis and Recognition, Montreal, Canada, Aug. 1995, pp. 1080–1083.
D. Blostein, “General Diagram-Recognition Methodologies,” in Graphics Recognition — Methods and Applications, Eds. R. Kasturi, K. Tombre, LNCS Vol. 1072, Springer Verlag, 1996, pp. 106–122.
D. Blostein, “Application of Graph Rewriting to Document Image Analysis,” Proc. Theory and Application of Graph Transformations — TAGT’98, Paderborn, Germany, Nov. 1998, pp. 16–23.
D. Blostein, A. Schürr, “Computing with Graphs and Graph Transformation,” Software Practice and Experience, 29(3), 1999, pp. 197–217.
H. Bunke, “Attributed Programmed Graph Grammars and Their Application to Schematic Diagram Interpretation,” IEEE Trans. Pattern Analysis and Machine Intelligence 4(6), Nov. 1982, pp. 574–582.
P. Chou, “Recognition of Equations Using a Two-Dimensional Stochastic Context-Free Grammar,” Proc. SPIE Visual Communications and Image Processing IV, Philadelphia PA, Nov. 1989, pp.852–863.
J. Cordy, C. Halpern-Hamu, E. Promislow, “TXL: A Rapid Prototyping System for Programming Language Dialects”, Computer Languages, 16(1), Jan. 1991, pp. 97–107.
D. Dori, A. Pnueli, “The Grammar of Dimensions in Machine Drawings,” Computer Vision, Graphics and Image Processing, 42, pp. 1–18, 1988.
H. Fahmy, D. Blostein, “A Graph Grammar Programming Style for Recognition of Music Notation,” Machine Vision and Applications, 6(2), 1993, pp. 83–99.
H. Fahmy, D. Blostein, “A Graph-Rewriting Paradigm for Discrete Relaxation: Application to Sheet-Music Recognition,” Intl. Journal of Pattern Recognition and Artificial Intelligence, 12(6), Sept. 1998, pp. 763–799.
H. Göttler, “Diagram Editors = Graphs + Attributes + Graph Grammars,” Intl. Journal of Man-Machine Studies, 37(4), Oct. 1992, pp. 481–502.
A. Grbavec, D. Blostein, “Mathematics Recognition Using Graph Rewriting,” Third Intl. Conf. Document Analysis and Recognition, Montreal, Canada, Aug. 1995, pp. 417–421.
L. Haken, D. Blostein, “A New Algorithm for Horizontal Spacing of Printed Music,” Intl. Computer Music Conf, Banff, Sept. 1995, pp. 118–119.
International Journal on Document Analysis and Recognition, Springer Verlag.
D. Isenor, S. Zaky, “Fingerprint Identification Using Graph Matching,” Pattern Recognition, 19(2), 1986, pp. 113–122.
D. Knuth, “Mathematical Typography,” Bulletin of the American Mathematical Society, 1(2), March 1979.
G. Kopec, P. Chou, “Document Image Decoding Using Markov Source Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, 16(6), June 1994, pp. 602–617.
S. Lavirotte, L. Pottier, “Optical Formula Recognition,” Fourth Intl. Conf. on Document Analysis and Recognition, Ulm, Germany, Aug. 1997, pp. 357–361.
G. Lohse, K. Biolsi, N. Walker, H. Ruetter, “A Classification of Visual Representations,” Communications of the ACM, 37(4), Dec. 1994, pp. 36–49.
K. Marriott, B. Meyer, K. Wittenburg, “A Survey of Visual Language Specification and Recognition,” in Visual Language Theory, Eds. K. Marriott, B. Meyer, Springer Verlag 1998, pp. 5–85.
B. Messmer, H. Bunke, “Clustering and Error-Correcting Matching of Graphs for Learning and Recognition of Symbols in Engineering Drawings,” DAS 96 IAPR Workshop on Document Analysis Systems, Malvern PA, Oct. 1996, pp. 282–296.
M. Minas, “Automatically Generating Environments for Dynamic Diagram Languages,” Proc. 1998 IEEE Symposium on Visual Languages, Halifax, Canada, Sept. 1998, pp. 70–71.
L. O’Gorman, R. Kasturi, Document Image Analysis, IEEE Computer Society Press, 1995.
F. Parmentier, A. Belaïd, “Logical Structure Recognition of Scientific Bibliographic References,” Fourth Intl. Conf. on Document Analysis and Recognition, Ulm, Germany, Aug. 1997, pp. 1072–1076.
B. Pasternak, “Processing Imprecise and Structural Distorted Line Drawings by an Adaptable Drawing Interpretation Kernel,” Proc. IAPR Workshop on Document Analysis Systems, Kaiserslautern, Germany, Oct. 1994, pp. 349–363.
M. Petre, “Why Looking Isn’t Always Seeing: Readership Skills and Graphical Programming,” Communications of the ACM, 38(6), June 1995, pp. 33–44.
A. Pies, Repräsentation und Verarbeitung von Musikalischem Wissen — Eine Attributierte Programmierte Graph-Grammatik zur Erkennung Gedruckter Partituren, Diplomarbeit, DFKI Kaiserslautern, Fachbereich Informatik, Aug. 1994.
B. Poirier, M. Dagenais, “An Interactive System to Extract Structured Text from a Geometrical Representation,” Fourth Intl. Conf. on Document Analysis and Recognition, Ulm, Germany, Aug. 1997, pp. 342–346.
L. Protsko, P. Sorenson, J. Tremblay, D. Schaefer, “Towards the Automatic Generation of Software Diagrams,” IEEE Trans. Software Engineering, 17(1), Jan. 1991, pp. 10–21.
Proc. Annual Symposia on Document Analysis and Information Retrieval, Las Vegas, 1992, 1993, 1994, 1995, 1996, 1997, sponsored by the University of Nevada.
Proc. IAPR Workshops on Document Analysis Systems, Kaiserslautern, Germany, Oct. 1994; Malvern PA, Oct. 1996; Nagano, Japan, Nov. 1998.
Proc. IAPR Workshops on Graphics Recognition, Pennsylvania, USA 1995; Nancy, France, 1997; Jaipur, India, 1999. http://graphics.basit.com/iapr-tcl0/GREC99/
Proc. Intl. Confs. on Document Analysis and Recognition, France, 1991; Japan 1993; Canada 1995; Germany 1997; Japan 1999; sponsored by IAPR and IEEE.
M. A. Rahgozar, R. Cooperman, “A Graph-based Table Recognition System,” Document Recognition III, SPIE Proceedings Series, Vol. 2660, San Jose, California, Jan. 1996, pp. 192–203.
G. Read, Music Notation: A Manual of Modern Practice (Second Edition), Taplinger Publishing, New York, NY, 1979.
J. Serrano, “The Use of Semantic Constraints on Diagram Editors,” Proc. 11th IEEE Symposium on Visual Languages, Darmstadt, Germany, Sept. 1995, pp. 211–216.
A. Smithies, K. Novins, J. Arvo, “A Handwriting-Based Equation Editor,” Proc Graphics Interface’ 99, Kingston, Canada, June, 1999, pp. 84–91.
R. Tamassia, G. Battista, C. Batini, “Automatic Graph Drawing and Readability of Diagrams,” IEEE Trans. Systems, Man and Cybernetics, 18(1), Jan. 1988, pp. 61–79.
R. Wilson, E. Hancock, “Structural Matching by Discrete Relaxation,” IEEE Trans. Pattern Analysis and Machine Intelligence, 19(6), June 1997, pp. 634–648.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Blostein, D. (2000). Defining the Syntax and Semantics of Natural Visual Languages. In: Nagl, M., Schürr, A., Münch, M. (eds) Applications of Graph Transformations with Industrial Relevance. AGTIVE 1999. Lecture Notes in Computer Science, vol 1779. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45104-8_16
Download citation
DOI: https://doi.org/10.1007/3-540-45104-8_16
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67658-4
Online ISBN: 978-3-540-45104-4
eBook Packages: Springer Book Archive