Intelligent Diagrammatic Interfaces: State of the Art

  • Bernd Meyer
  • Kim Marriott
  • Gerard Allwein

Abstract

One important practical role for diagrammatic reasoning is to provide the foundation for intelligent diagrammatic environments (IDEs), in which structured diagrams are used as a means of natural visual communication between the human user and the system. The present chapter provides a characterisation of IDEs and gives an overview of the state of the art. Rather than attempting an exhaustive survey, we discuss the subtasks that have to be addressed in a general IDE framework and show how they are handled in representative systems from the following IDE application areas:
  • Document understanding

  • Diagramming support

  • Diagrammatic query languages for spatial information systems

  • Diagrammatic theorem provers

Keywords

Brittleness Tated Prefix Editing Kato 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Ah-Soon, C. (1998). A constraint network for symbol detection in architectural drawings. In K. Tombre and A. Chhabra (Eds), Graphics Recognition: Algorithms and systems, Vol. 1389 of LNCS. Berlin: Springer, pp. 80–90.Google Scholar
  2. 2.
    Baird, H. (1990). Industrial applications. In H. Bunke and A. Sanfeliu (Eds), Syntactic and structural pattern recognition: Theory and applications. World Scientific, pp. 369–380.Google Scholar
  3. 3.
    Baker, S., Ireland, A. and Smaill, A. (1992). On the use of the constructive omega rule within automated deduction. In A. Voronkov (Ed.), Logic programming and automated reasoning. LNAI 624. Berlin: Springer, pp. 214–225.CrossRefGoogle Scholar
  4. 4.
    Barker-Plummer, D. and Bailin, S. (1992). Proving the diamond lemma with the GROVER theorem proving system. In AAAI symposium on reasoning with diagrammatic representations.Google Scholar
  5. 5.
    Barwise, J. and Etchemendy, J. (1996). Visual information and valid reasoning. In G. Allwein and J. Barwise (Eds), Logical reasoning with diagrams. Oxford: Oxford University Press, pp. 3–26.Google Scholar
  6. 6.
    Borning, A., Marriott, K., Stuckey, P. and Xiao, Y. (1997). Solving linear arithmetic constraints for user interface applications. In UIST’97: ACM symposium on user interface software and technology.Google Scholar
  7. 7.
    Brachman, R.J. and Schmolze, J.G. (1985). An overview of KL-ONE knowledge representation systems. Cognitive Science August:171–216.Google Scholar
  8. 8.
    Catarci, T. Costabile, M.F., Levialdi, S. and Batini, C. (1997). Visual query systems for databases: A survey. Journal of Visual Languages and ComputingGoogle Scholar
  9. 9.
    Chok, S. and Marriott, K. (1995). Automatic construction of user interfaces from constraint multiset grammars. In IEEE symposium on visual languages. Los Alamitos, CA: IEEE Computer Society Press, pp. 242–250.Google Scholar
  10. 10.
    Chok, S. and Marriott, K. (1998a). Automatic construction of intelligent diagram editors. In Proceedings of the 11th ACM symposium on user interface software and technology, pp. 185–194.Google Scholar
  11. 11.
    Chok, S. and Marriott, K. (1998b). Automatic construction of intelligent diagram editors. In Proceedings of the 11th ACM symposium on user interface software and technology.Google Scholar
  12. 12.
    Clarke, B. (1981). A calculus of individuals based on connection. Notre Dame Journal of Formal Logic 23(3):204–218.CrossRefGoogle Scholar
  13. 13.
    Clarke, B. (1985). Individuals and points. Notre Dame Journal of Formal Logic 26(1):61–75.MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Clementini, E., DiFelice, P. and van Oosterom, P. (1993). A small set of formal topological relationships suitable for end-user interaction. In D. Abel and B. Ooi (Eds), International symposium on advances in spatial databases (SSD’93), Singapore, June, pp. 277–295.Google Scholar
  15. 15.
    Cohn, A.G. (1997). Qualitative spatial representation and reasoning techniques. In Künstliche Intelligenz 1997 (KI’97). LNAI 1303. Springer, pp. 1–30.Google Scholar
  16. 16.
    Costagliola, G., De Lucia, A., Orefice, S. and Tortora, G. (1998). Positional grammars: Theory and practice in a visual language for interface modelling. In K. Marriott and B. Meyer (Eds), Visual language theory. New York: Springer, pp. 171–191.CrossRefGoogle Scholar
  17. 17.
    Dinesh, T. and Usküdarli, S. (1998). Input and output for specified visual languages. In K. Marriott and B. Meyer (Eds), Visual language theory. New York: Springer, pp. 325–351.CrossRefGoogle Scholar
  18. 18.
    Egenhofer, M. (1996). Spatial-query-by-sketch. In IEEE sypmosium on visual languages, pp. 170–177.Google Scholar
  19. 19.
    Egenhofer, M. and Franzosa, R. (1991). Point-set topological spatial relations. International Journal of Geographical Information Systems 5(2):161–174.CrossRefGoogle Scholar
  20. 20.
    Faloutsos, C., Barber, R., Flickner, M., Hafner, J., Niblack, W., Petrovic, D. and Equitz, W. (1994). Efficient and effective querying by image content. Journal of Intelligent Information Systems 3:231–262.CrossRefGoogle Scholar
  21. 21.
    Fumas, G. (1990). Formal models for imaginal deduction. In Twelfth annual conference of the Cognitive Science Society. Hillsdale, NJ: Erlbaum, pp. 662–669.Google Scholar
  22. 22.
    Fumas, G. (1991). New graphical reasoning models for understanding graphical interfaces. In Proceedings of the ACM conference on human factors in computing systems (CHI). New York: ACM Press, pp. 71–78.Google Scholar
  23. 23.
    Fumas, G. (1992). Reasoning with diagrams only. In AAAI spring symposium on reasoning with diagrammatic representations, Stanford.Google Scholar
  24. 24.
    Futrelle, R. and Nikolakis, N. (1995). Efficient analysis of complex diagrams using constraint-based parsing. In Proceedings of the third international conference on document analysis and recognition (ICDAR), pp. 782–790.Google Scholar
  25. 25.
    Gelernter, H. (1963). Realization of a geometry theorem-proving machine. In E. Feigenbaum and J. Feldman (Eds), Computers and thought. New York: McGraw-Hill, pp. 134–152..Google Scholar
  26. 26.
    Gooday, J. and Cohn, A. (1996). Using spatial logic to describe visual programming languages. Artificial Intelligence Review 10:171–186.CrossRefGoogle Scholar
  27. 27.
    Gross, M. (1994a). Recognizing and interpreting diagrams in design. In T. Catarci, M. Costabile, S. Levialdi and G. Santucci (Eds), Advanced visual interfaces (AVI). New York: ACM Press.Google Scholar
  28. 28.
    Gross, M. (1994b). Stretch-a-sketch: A dynamic diagrammer. In Proceedings of the 1994 IEEE symposium on visual languages. Los Alamitos, CA: IEEE Computer Society Press, pp. 232–238:Google Scholar
  29. 29.
    Gross, M. and Do, E.-L. (1996). Ambiguous intentions: a paper-like interface for creative design. In ACM 9th symposium on user interface software and technology (UIST). ACM Press, pp. 183–192.Google Scholar
  30. 30.
    Haarslev, V. (1998). A fully formalized theory for describing visual notations. In K. Marriott and B. Meyer (Eds), Visual language theory. New York: Springer, pp. 261–192g.Google Scholar
  31. 31.
    Haarslev, V. and Wessel, M. (1997). Querying GIS with animated spatial sketches. In IEEE sypmosium on visual languages, Capri, Italy, pp. 201–208.Google Scholar
  32. 32.
    Helm, R. and Marriott, K. (1986). Declarative graphics. In Proceedings of the 3rd international conference on logic programming, Vol. 225 of LNCS. New York: Springer, pp. 513–527.Google Scholar
  33. 33.
    Helm, R., Marriott, K., Huynh, T. and Vlissides, J. (1995). An object-oriented architecture for constraint-based graphical editing. In C. Laffra, E. Blake, V. de Mey and X. Pintado (Eds), Object-oriented programming for graphics. Berlin: Springer, pp. 217–238.CrossRefGoogle Scholar
  34. 34.
    Jamnik, M., Bundy, A. and Green, I. (1997). Automation of diagrammatic proofs in mathematics. In B. Kokinov (Ed.), Perspectives on cognitive science, Vol. 3. New Bulgarian University, pp. 168–175.Google Scholar
  35. 35.
    Karima, M., Sadhal, K. and McNeil, T. (1995). From paper drawings to computer aided design. IEEE Computer Graphics and Applications February:24–39.Google Scholar
  36. 36.
    Kasturi, R., Raman, R., Chennubthotla, C. and O’Gorman, L. (1992). An overview of techniques for graphics recognition. In H. Baird, H. Bunke, and K. Yamamoto (Eds), Structured document image analysis. Berlin: Springer, pp. 285–324.CrossRefGoogle Scholar
  37. 37.
    Kato, T. (1992). Cognitive view mechanism for multimedia information systems. In R. Cooper (Ed.), International workshop on interfaces to database systems. LNCS. New York: Springer.Google Scholar
  38. 38.
    Lakin, F. (1986). Spatial parsing for visual languages. In S.-K. Chang, T. Ichikawa and P.A. Ligomenides (Eds), Visual languages. New York: Plenum Press, pp. 35–85.CrossRefGoogle Scholar
  39. 39.
    Lakin, F. (1987). Visual grammars for visual languages. In AAAI-87, 7th national conference on AI, pp. 683–688.Google Scholar
  40. 40.
    Marriott, K. (1994). Constraint multiset grammars. In IEEE symposium on visual languages. Los Alamitos, CA: IEEE Computer Society Press, pp. 118–125.Google Scholar
  41. 41.
    Marriott, K., Meyer, B. and Wittenburg, K. (1998). A survey of visual language specification and recognition. In K. Marriott and B. Meyer (Eds), Visual language theory. New York: Springer, pp. 5–85.CrossRefGoogle Scholar
  42. 42.
    Meyer, B. (1992). Beyond icons: Towards new metaphors for visual query languages for spatial information systems. In R. Cooper (Ed.), Interfaces to database systems. Glasgow: Springer, pp. 113–135.Google Scholar
  43. 43.
    Meyer, B. (1994). Pictorial deduction in spatial information systems. In International IEEE workshop on visual languages (VL’94), St Louis, MO, October, pp. 23–30.Google Scholar
  44. 44.
    Meyer, B. (1997). Formalization of visual mathematical notations. In M. Anderson (Ed.), AAAI symposium on diagrammatic reasoning (DR-II), Boston, MA, November. AAAI Technical Report FS-97–02, AAAI Press, pp. 97–02.Google Scholar
  45. 45.
    Meyer, B. (1999). Constraint diagram reasoning. In Principles and practice of constraint programming: CP’99, Alexandria, VA, October. Berlin: Springer.Google Scholar
  46. 46.
    Meyer, B. (2000). A constraint-based framework for diagrammatic reasoning. In Applied Artificial Intelligence (special issue on constraint handling rules) 14.Google Scholar
  47. 47.
    Meyer, B. and Marriott, K. (1997). Specifying diagram animation with rewrite systems. In International workshop on theory of visual languages (TVL’97), Capri, Italy, September, pp. 85–96.Google Scholar
  48. 48.
    Meyer, B. and Zweckstetter, H. (1998). Interpretation of visual notations in the recopla editor generator. In FRVDR’98: AAAI fall symposium on formalizing reasoning with visual and diagrammatic representations, Orlando, FL, October. AAAI TR FS-98–04.Google Scholar
  49. 49.
    Meyer, B., Zweckstetter, H., Mandel, L. and Gassmann, Z. (1999). Automatic construction of intelligent diagrammatic environments. In HCI’99: 8th international conference on human-computer interaction, Munich, Germany, August. Hillsdale, NJ: Erlbaum.Google Scholar
  50. 50.
    Minas, M. and Viehstaedt, G. (1995). Diagen: A generator for diagram editors providing direct manipulation and execution of diagrams. In IEEE workshop on visual languages. Los Alamitos, CA: IEEE Computer Society Press, pp. 203–210.Google Scholar
  51. 51.
    Nagy, G., Seth, S. and Viswanathan, M. (1992). A prototype document image analysis system for technical journals. IEEE Computer 25(7):10–22.CrossRefGoogle Scholar
  52. 52.
    Rozenberg, G. (Ed.)(1997). Handbook of graph grammars and computing by graph transformation. Singapore, World Scientific.Google Scholar
  53. 53.
    Schneider, M. (1999). Uncertainty management for spatial data in databases: Fuzzy spatial data types. In International symposium on advances in spatial databases (SSD’99), Hong Kong. LNCS 1651, pp. 330–351. LNCS 1651.Google Scholar
  54. 54.
    Weitzman, L. and Wittenburg, K. (1993). Relational grammars for interactive design. In IEEE symposium on visual languages, pp. 4–11.Google Scholar
  55. 55.
    Wessel, M. and Haarslev, V. (1998). VISCO: Bringing visual spatial querying to reality. In IEEE sypmosium on visual languages, Halifax, Canada, pp. 170–177.Google Scholar
  56. 56.
    Wittenburg, K. and Weitzman, L. (1990). Visual grammars and incremental parsing for interface languages. In IEEE symposium on visual languages. Los Alamitos, CA: IEEE Computer Society Press, pp. 111–118.Google Scholar
  57. 57.
    Yokokura, N. and Watanabe, T. (1998). Layout-based approach for extracting constructive elements of bar-charts. In K. Tombre, and A. Chhabra (Eds),Google Scholar

Copyright information

© Springer-Verlag London 2002

Authors and Affiliations

  • Bernd Meyer
  • Kim Marriott
  • Gerard Allwein

There are no affiliations available

Personalised recommendations