Visual Language Theory: Towards a Human-Computer Interaction Perspective

  • N. Hari Narayanan
  • Roland Hübscher

Abstract

The main reason for using visual languages is that they are often far more convenient to the user than traditional textual languages. Therefore, visual languages intended for use by both computers and humans ought to be designed and analyzed not only from the perspective of computational resource requirements, but also from the perspective of languages that are cognitively usable and useful. Theoretical and practical research on visual languages needs to take into account the full context of a coupled human-computer system in which the visual language facilitates interactions between the computational and the cognitive parts. This implies that theoretical analyses ought to address issues of comprehension, reasoning, and interaction in the cognitive realm as well as issues of visual program parsing, execution, and feedback in the computational realm. The human aspect is crucial to visual languages, and therefore we advocate a correspondingly broadened scope of inquiry for visual language research. In this chapter we describe aspects of human use of visual languages that ought to be important considerations in visual language research and design, and summarize research from related fields such as software visualization and diagrammatic reasoning that addresses these issues. A framework consistent with the broadened scope of visual language research is proposed and used to categorize and discuss several formalizations and implemented systems. In the course of showing how a sample of current work fits into this framework, open issues and fruitful directions for future research are also identified.

Keywords

Dial Metaphor Comic Strip Compro Furnas 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    H. Abelson, G.J. Sussman, and J. Sussman.Structure and Interpretation of Computer Programs.MIT Press, Cambridge, 2nd edition, 1996.MATHGoogle Scholar
  2. [2]
    M. Andries, G. Engels, and J. Rekers. How to represent a visual specification. Inthis volume. Google Scholar
  3. [3]
    A. Antonietti. Why does mental visualization facilitate problem solving? In R.H. Logie and M. Denis, editorsMental Images in Human Cognitionpages 211–227. Elsevier Science Publishers, Amsterdam, 1991.CrossRefGoogle Scholar
  4. [4]
    R. Arnheim.Visual Thinking.University of California Press, Berkeley, 1969.Google Scholar
  5. [5]
    A. Badre and J. Allen. Graphic language representation and programming behavior. In S. Salvendy and M.J. Smith, editorsDesigning and Using Human-Computer Interfaces and Knowledge Based Systemspages 59–65. Elsevier Science Publishers, Amsterdam, 1989.Google Scholar
  6. [6]
    J. Barwise and J. Etchemendy. Heterogeneous logic. In J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editorsDiagrammatic Reasoning: Cognitive and Computational Perspectivespages 211–234. AAAI Press, Menlo Park and MIT Press, Cambridge, 1995.Google Scholar
  7. [7]
    J. Bertin.Semiology of Graphics.1967. English translation by W.J. Berg, University of Wisconsin Press, Madison, 1983.Google Scholar
  8. [8]
    A.F. Blackwell. Metacognitive theories of visual programming: What do we think we are doing? InIEEE Workshop on Visual Languagespages 240–246. IEEE Computer Society Press, Los Alamitos, 1996.Google Scholar
  9. [9]
    P. Bottoni, M.F. Costabile, S. Levialdi, and P. Mussio. Formalizing visual languages. InIEEE Workshop on Visual Languagespages 45–52. IEEE Computer Society Press, Los Alamitos, 1995.Google Scholar
  10. [10]
    P. Bottoni, M.F. Costabile, S. Levialdi, and P. Mussio. Formalizing visual languages. In IEEE Workshop on Visual Languages, pages 45–52. IEEE Computer Society Press, Los Alamitos, 1995 Google Scholar
  11. [11]
    P. Bottoni, M.F. Costabile, S. Levialdi, and P. Mussio. Specification of visual languages as means for interaction. Inthis volume. Google Scholar
  12. [12]
    F.P. Brooks. No silver bullet: Essence and accidents of software engineering.IEEE Computer20(4):10–19, 1987.MathSciNetCrossRefGoogle Scholar
  13. [13]
    M.H. Brown and R. Sedgewick. Techniques for algorithm animation.IEEE Software2(1):28–38, 1985.CrossRefGoogle Scholar
  14. [14]
    M.D. Byrne, R. Catrambone, and J.T. Stasko. Do algorithm animations aid learning? Technical Report GIT-GVU-96–18, Graphics, Visualization and Usability Center, Georgia Institute of Technology, Atlanta, USA, August 1996.Google Scholar
  15. [15]
    CACM. Special section on educational technologyCommunications of the ACM39(4), April, 1996.Google Scholar
  16. [16]
    S.-K. Chang Visual languages: A tutorial and survey.IEEE Software4:29–39, 1987.CrossRefGoogle Scholar
  17. [17]
    T.B. Dinesh and S. Üsküdarli. Input and output for specified visual languages. Inthis volume. Google Scholar
  18. [18]
    S. Douglas, C. Hundhausen, and D. McKeown. Toward empirically-based software visualization languages. InProc. IEEE Symposium on Visual Languagespages 342–349. IEEE Computer Society Press, Los Alamitos, 1995.Google Scholar
  19. [19]
    Y. Engelhardt, J. Bruin, T. Janssen, and R. Scha. The visual grammar of information graphics. In N.H. Narayanan and J. Damski, editorsProc. AID’96 Workshop on Visual Representation Reasoning and Interaction in DesignKey Center for Design Computing, University of Sydney, 1996.Google Scholar
  20. [20]
    M. Erwig and B. Meyer. Heterogeneous visual languages: Integrating visual and textual programming. InProc. IEEE Symposium on Visual Languagespages 318–325. IEEE Computer Society Press, Los Alamitos, 1995.Google Scholar
  21. [21]
    M.R. Frank and J.D. Foley. A pure reasoning engine for programming by example. Technical Report GIT-GVU-94–11, Graphics, Visualization and Usability Center, Georgia Institute of Technology, Atlanta, USA, April 1994.Google Scholar
  22. [22]
    E. Freeman, D. Gelernter, and S. Jagannathan. In search of a simple visual vocabulary. InProc. IEEE Symposium on Visual Languagespages 302–309. IEEE Computer Society Press, Los Alamitos, 1995.Google Scholar
  23. [23]
    G.W. Fumas. New graphical reasoning models for understanding graphical interfaces. InProc. Human Factors in Computing Systems Conference (CHI’91)pages 71–78. ACM Press, New York, 1991.Google Scholar
  24. [24]
    G.W. Fumas. Reasoning with diagrams only. In N.H. Narayanan, editorProc. AAAI Spring Symposium on Reasoning with Diagrammatic Representationspages 118–123, Technical Report SS-92–02, AAAI Press, Menlo Park, March 1992.Google Scholar
  25. [25]
    J.S. Gero and M. Yan. Shape emergence by symbolic reasoning.Environment and Planning B: Planning and Design21:191–218, 1994.CrossRefGoogle Scholar
  26. [26]
    J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editors.Diagrammatic Reasoning: Cognitive and Computational Perspectives.AAAI Press, Menlo Park, MIT Press, Cambridge, 1995.Google Scholar
  27. [27]
    E.P. Glinert. Nontextual programming environments. In S.-K. Chang, editorPrinciples of Visual Programming Systemspages 144230. Prentice-Hall, New York, 1990.Google Scholar
  28. [28]
    V. Goel.Sketches of Thought.MIT Press, Cambridge, 1995.Google Scholar
  29. [29]
    E.H. Gombrich.Art and Illusion: A Study in the Psychology of Pictorial Representations.Phaidon, London, 1968.Google Scholar
  30. [30]
    J.M. Gooday and A.G. Cohn. Visual language syntax and semantics: A spatial logic approach. In K. Marriott and B. Meyer, editorsInternational Workshop on Theory of Visual LanguagesGubbio, Italy, 1996. Monash University.Google Scholar
  31. [31]
    N. Goodman.Languages of Art: An Approach to a Theory of Symbols.Hackett Publishing Company, Indianapolis, 1976.Google Scholar
  32. [32]
    T.R.G. Green. Cognitive dimensions of notations. In R. Winder and A. Sutcliffe, editorsPeople and ComputersV, Cambridge University Press, 1990.Google Scholar
  33. [33]
    T.R.G. Green and M. Petre. When visual programs are harder to read than textual programs. In G.C. van der Veer, M.J. Tauber, S. Bagnarola, and M. Antavolits, editorsHuman-Computer Interaction: Tasks and Organization Proc. 6th European Conference on Cognitive Ergonomicspages 167–180, 1992.Google Scholar
  34. [34]
    T.R.G. Green and M. Petre. Usability analysis of visual programming environments: A cognitive dimensions framework.Journal of Visual Languages and Computing7(2):131–174, 1996.CrossRefGoogle Scholar
  35. [35]
    T.R.G. Green, M. Petre, and R.K.E. Bellamy. Comprehensibility of visual and textual programs: A test of superlativism against the match-mismatch conjecture. In J. Koenemann-Belliveau, T.G. Mo-her, and S.P. Robertson, editorsProc. Fourth Workshop on Empirical Studies of Programmers.Ablex Publishers, Greenwich, 1992.Google Scholar
  36. [36]
    M. Gross. The fat pencil, the cocktail napkin, and the slide library. In A. Harfmann and M. Fraser, editorsProc. ACADIA 94pages 103–113, 1994.Google Scholar
  37. [37]
    C.A. Gurr. On the isomorphism, or lack of it, of representations. Inthis volume. Google Scholar
  38. [38]
    V. Haarslev. A fully formalized theory for describing visual notations. Inthis volume. Google Scholar
  39. [39]
    E. Hammer. Logic and visual information. InStudies in Logic Language e.4 Computation.CSLI Press, Stanford University, Palo Alto, 1995.MATHGoogle Scholar
  40. [40]
    M. Hegarty. Mental animation: Inferring motion from static displays of mechanical systems.Journal of Experimental Psychology: Learning Memory e4 Cognition18(5):1084–1102, 1992.CrossRefGoogle Scholar
  41. [41]
    M. Hegarty and M.A. Just. Constructing mental models of machines from text and diagrams.Journal of Memory and Language32:717742, 1993.Google Scholar
  42. [42]
    D. Hix and H.R. Hutson.Developing User Interfaces: Ensuring Usability Through Product E4 Process.John Wiley&Sons, Inc., New York, 1993.Google Scholar
  43. [43]
    R. Hübscher. Rewriting interaction. InProc. Human Factors in Computing Systems Conference (CHI’95)pages 105–106. ACM Press, New York, 1995.Google Scholar
  44. [44]
    R. Hübscher. Composing complex behavior from simple visual descriptions. InProc. IEEE Symposium on Visual Languagespages 88–94, IEEE Computer Society Press, Los Alamitos, 1996.CrossRefGoogle Scholar
  45. [45] R. Hübscher. Visual constraint rules.Journal of Visual Languages and ComputingforthcomingGoogle Scholar
  46. [46]
    J. Huttenlocher. Constructing spatial images: A strategy in reasoning.Psychological Review75(6):550–560, 1968.CrossRefGoogle Scholar
  47. [47]
    S.H. Joseph and T.P. Pridmore. Knowledge-directed interpretation of mechanical engineering drawings.IEEE Trans. on Pattern Analysis and Machine Intelligence14(9):928–940, 1992.CrossRefGoogle Scholar
  48. [48]
    K.M. Kahn and V.A. Saraswat. Complete visualizations of concurrent programs and their executions. InProc. IEEE Symposium on Visual Languagespages 7–14. IEEE Computer Society Press, Los Alamitos, 1990.Google Scholar
  49. [49]
    K.R. Koedinger. Emergent properties and structural constraints: Advantages of diagrammatic representations for reasoning and learning. In N.H. Narayanan, editorProc. AAAI Spring Symposium on Reasoning with Diagrammatic Representationspages 154–169, Technical Report SS-92–02, AAAI Press, Menlo Park, March 1992.Google Scholar
  50. [50]
    J. Larkin. Display based problem solving. In D. Klahr and K. Kotovsky, editorsComplex Information Processing.Lawrence Erlbaum Publishers, Hillsdale, 1989.Google Scholar
  51. [51]
    J.H. Larkin and H.A. Simon. Why a diagram is (sometimes) worth ten thousand words.Cognitive Science11:65–99, 1987.CrossRefGoogle Scholar
  52. [52]
    G.I. Lohse, K. Biolsi, N. Walker, and H.H. Rueler. A classification of visual representations.Communications of the ACM37(12):36–49, 1994.CrossRefGoogle Scholar
  53. [53]
    R.K. Lowe. Constructing a mental representation from an abstract technical diagram.Learning and Instruction3:157–179, 1993.CrossRefGoogle Scholar
  54. [54]
    R.K. Lowe. Selectivity in diagrams. Reading beyond the lines.Educational Psychology14:467–491, 1994.CrossRefGoogle Scholar
  55. [55]
    D.E. Mahling and D.L. Fisher. The cognitive engineering of visual languages. InProc. IEEE Symposium on Visual Languagespages 22–28. IEEE Computer Society Press, Los Alamitos, 1990.Google Scholar
  56. [56]
    K. Marriott and B. Meyer. The CCMG visual language hierarchy. Inthis volume. Google Scholar
  57. [57]
    K. Marriott, B. Meyer, and K. Wittenburg. A survey of visual language specification and recognition. Inthis volume. Google Scholar
  58. [58]
    P. McCorduck.Aaron’s Code.Freeman, San Francisco, 1991.Google Scholar
  59. [59]
    T. Menzies. Frameworks for assessing visual languages. Technical Report TR 95–35, Department of Software Development, Monash University, Australia, 1995.Google Scholar
  60. [60]
    B. Meyer. Pictures depicting pictures: On the specification of visual languages by visual grammars. InProc. IEEE Symposium on Visual Languagespages 41–47. IEEE Computer Society Press, Los Alamitos, 1992.Google Scholar
  61. [61]
    B. Myers. Visual programming, programming by example and program visualization: A taxonomy. InProc. Human Factors in Computing Systems Conference (CHI’86)pages 59–66. ACM Press, New York, 1986.Google Scholar
  62. [62]
    N.H. Narayanan, editor.Proc. AAAI Spring Symposium on Reasoning with Diagrammatic RepresentationsTechnical Report SS-92–02, AAAI Press, Menlo Park, March 1992.Google Scholar
  63. [63]
    N.H. Narayanan and M. Hegarty. On designing comprehensible interactive hypermedia manuals.International Journal of Human Computer Studiesforthcoming.Google Scholar
  64. [64]
    N.H. Narayanan, M. Suwa, and H. Motoda. How things appear to work: Predicting behaviors from device diagrams InProc. 12th National Conference on Artificial Intelligencepages 1161–1167. AAAI Press, Menlo Park, 1994.Google Scholar
  65. [65]
    N.H. Narayanan, M. Suwa, and H. Motoda. A study of diagrammatic reasoning from verbal and gestural protocols. InProc. 16th Annual Conference of the Cognitive Science Societypages 652–657. Lawrence Erlbaum Publishers, Hillsdale, 1994.Google Scholar
  66. [66]
    N.H. Narayanan, M. Suwa, and H. Motoda. Behavior hypothesis from schematic diagrams. In J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editorsDiagrammatic Reasoning: Cognitive and Computational Perspectivespages 501–534. AAAI Press, Menlo Park and MIT Press, Cambridge, 1995.Google Scholar
  67. [67]
    N.H. Narayanan, M. Suwa, and H. Motoda. Diagram-based problem solving: The case of an impossible problem. InProc. 17th Annual Conference of the Cognitive Science Societypages 206–211. Lawrence Erlbaum Publishers, Hillsdale, 1995.Google Scholar
  68. [68]
    A. Newell and H.A. Simon.Human Problem Solving.Prentice-Hall, Englewood Cliffs, 1972.Google Scholar
  69. [69]
    W.M. Newman and M.G. Lamming.Interactive System Design.Addison-Wesley, Wokingham, 1995.Google Scholar
  70. [70]
    J.V. Nickerson. Visual programming. Limits of graphic representation. InProc. IEEE Symposium on Visual Languagespages 178–179. IEEE Computer Society Press, Los Alamitos, 1994.CrossRefGoogle Scholar
  71. [71]
    D.A. Norman. Cognitive engineering. In D.A. Norman and S.W. Draper, editorsUser Centered System Designpages 31–65. Lawrence Erlbaum Publishers, Hillsdale, 1986.Google Scholar
  72. [72]
    D.A. Norman.The Psychology of Everyday Things.Basic Books, New York, 1988.Google Scholar
  73. [73]
    G.S. Novak and W.C. Bulko. Diagrams and text as computer input.Journal of Visual Languages and Computing4:161–175, 1993.CrossRefGoogle Scholar
  74. [74]
    M. Petre. Why looking isn’t always seeing: Readership skills and graphical programmingCommunications of the ACM38:33–44, 1995.CrossRefGoogle Scholar
  75. [75]
    M. Petre, A.F. Blackwell, and T.R.G. Green. Cognitive questions in software visualization. In J. Stasko, J. Domingue, B. Price, and M. Brown, editorsSoftware Visualization: Programming as a Multi-Media Experience.MIT Press, Cambridge, 1997.Google Scholar
  76. [76]
    B.A. Price, R.M. Baecker, and I.S. Small. A principled taxonomy of software visualization.Journal of Visual Languages and Computing4(3):211–266, 1993.CrossRefGoogle Scholar
  77. [77]
    J. Puigsegur, J. Agusti, and D. Robertson. A visual programming language. InProc. IEEE Symposium on Visual Languagespages 214–215. IEEE Computer Society Press, Los Alamitos, 1996.CrossRefGoogle Scholar
  78. [78]
    D.R. Raymond. Characterizing visual languages. InProc. IEEE Symposium on Visual Languagespages 176–182. IEEE Computer Society Press, Los Alamitos, 1991.Google Scholar
  79. [79]
    A. Repenning. Bending the rules: Steps toward semantically enriched graphical rewrite rules. InProc. IEEE Symposium on Visual Languagespages 226–233. IEEE Computer Society Press, Los Alamitos, 1995.Google Scholar
  80. [80]
    A. Repenning and T. Sumner. Agentsheets: A medium for creating domain-oriented visual languages.IEEE Computer28:17–25, 1995.CrossRefGoogle Scholar
  81. [81]
    M. Resnick. Beyond the centralized mindset.Journal of the Learning Sciences5(1):1–22, 1996.CrossRefGoogle Scholar
  82. [82]
    G.G. Robertson, S.K. Card, and J.D. Mackinlay. Information visualization using 3D interactive animation.Communications of the ACM36(4):57–71, 1993.CrossRefGoogle Scholar
  83. [83]
    F. Saint-Martin.Semiotics of Visual Language.Indiana University Press, Bloomington, 1990.Google Scholar
  84. [84]
    D.L. Schwartz and J.B. Black. Analog imagery in mental model reasoning: Depictive models.Cognitive Psychology30:254–219, 1996.CrossRefGoogle Scholar
  85. [85]
    T. Selker and L. Koved. Elements of visual language. InProc. IEEE Symposium on Visual Languagespages 38–44. IEEE Computer Society Press, Los Alamitos, 1988.Google Scholar
  86. [86]
    R.N. Shepard and L.A. Cooper.Mental Images and Their Transformations.MIT Press, Cambridge, 1986.Google Scholar
  87. [87]
    S.-J. Shin.The Logical Status of Diagrams.Cambridge University Press, Cambridge, 1994.MATHGoogle Scholar
  88. [88]
    N.C. Shu. Visual programming languages: A perspective and a dimensional analysis. In S.-K. Chang, T. Ichikawa, and P.A. Ligomenides, editorsVisual Languagespages 11–34. Plenum Publishing Corporation, New York, 1986.CrossRefGoogle Scholar
  89. [89]
    A. Sinha and I. Vessey. Cognitive fit in recursion and iteration: An empirical study.IEEE Trans. on Software EngineeringSE-10(5):386–379, 1992.Google Scholar
  90. [90]
    D.C. Smith, A. Cypher, and J. Spohrer. KidSim: Programming agents without a programming language.Communications of the ACM37:54–68, 1994.CrossRefGoogle Scholar
  91. [91]
    R.B. Smith. The alternate reality kit: An animated environment for creating interactive simulations. InProc. IEEE Symposium on Visual Languagespages 99–106. IEEE Computer Society Press, Los Alamitos, 1986.Google Scholar
  92. [92]
    S. Steinman and K. Carver.Visual programming with Prograph CPX.Manning Publications/Prentice Hall, Englewood Cliffs, 1995.Google Scholar
  93. [93]
    K. Stenning, R. Cox, and J. Oberlander. Contrasting the cognitive effects of graphical and sentential logic teaching. Reasoning, representation and individual differences.Language and Cognitive Processes10(3/4):333–354, 1995.CrossRefGoogle Scholar
  94. [94]
    K. Stenning and J. Oberlander. A cognitive theory of graphical and linguistic reasoning: Logic and implementation.Cognitive Science19:97–140, 1995.CrossRefGoogle Scholar
  95. [95]
    S. Tessler, Y. Iwasaki, and K. Law. Qualitative structural analysis using diagrammatic reasoning. In J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editorsDiagrammatic Reasoning: Cognitive and Computational Perspectivespages 711–730. AAAI Press, Menlo Park and MIT Press, Cambridge, 1995.Google Scholar
  96. [96]
    E.R. Tufte.The Visual Display of Quantitative Information.Graphics Press, Cheshire, 1983.Google Scholar
  97. [97]
    E.R. Tufte.Envisioning Information.Graphics Press, Cheshire, 1990.Google Scholar
  98. [98]
    E.R. Tufte.Visual Explanations.Graphics Press, Cheshire, 1997.MATHGoogle Scholar
  99. [99]
    B. Tversky. Cognitive origins of graphic productions. In F.T. Marchese, editorUnderstanding Images: Finding Meaning in Digital Imagerypages 29–53. Springer-Verlag, New York, 1995.Google Scholar
  100. [100]
    D. Wang, J.R. Lee, and H. Zeevat. Reasoning with diagrammatic representations. In J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editorsDiagrammatic Reasoning: Cognitive and Computational Perspectivespages 339–396. AAAI Press, Menlo Park and MIT Press, Cambridge, 1995.Google Scholar
  101. [101]
    D. Wang and H. Zeevat. A syntax directed approach to picture semantics. Inthis volume. Google Scholar
  102. [102]
    K. Wittenburg and L. Weitzman. Relational grammars: Theory and practice in a visual language interface for process modeling. Inthis volume. Google Scholar

Copyright information

© Springer Science+Business Media New York 1998

Authors and Affiliations

  • N. Hari Narayanan
  • Roland Hübscher

There are no affiliations available

Personalised recommendations