Why Are Analogue Graphics and Natural Language Both Needed in HCI?
The combined use of language and analogue graphics for the expression of information probably is as old as language itself. The paper addresses the question why we need both the expressions of natural language and analogue graphics for the representation of information. It is argued that analogue graphics and natural language have the complementary expressive virtues of specificity and focus, respectively. Their corresponding lack of focus and specificity, respectively, explain why (a) both have developed a number of mechanisms for coping with these deficiencies and (b) why their combination may often have superior expressive power. Since specificity follows from the analogue character of analogue graphics rather than from their graphic character, analogue sound and touch representations are analysed to explore whether results from the analysis of analogue graphics and their complementarity with natural language can be transferred to other analogue modalities of expression. The paper exemplifies the comparatively new field of Modality Theory.
KeywordsNatural Language Linguistic Expression External Representation Analogue Representation Indexical Reference
Unable to display preview. Download preview PDF.
- N.O. Bernsen. Matching Information and Interface Modalities. An Example Study. Technical Report Deliverable 2.1.1, Esprit Basic Research project GRACE, 1993.Google Scholar
- N.O. Bernsen. Modality Theory: Supporting multimodal interface design. In Proceedings from the ERCIM Workshop on Multimodal Human-Computer Interaction, Nancy, November 1993.Google Scholar
- N.O. Bernsen. A research agenda for modality theory. In R. Cox, M. Petre, P. Brna, and J. Lee, editors, Proceedings of the Workshop on Graphical Representations, Reasoning and Communication, pages 43–46, Edinburgh, August 1993. World Conference on Artificial Intelligence in Education.Google Scholar
- N.O. Bernsen and B. Svane. Communication failure and mental models. In Proceedings of the 13th Scandinavian Conference on Linguistics, 1992.Google Scholar
- E. Hovy and Y. Arens. When is a picture worth a thousand words? Allocation of modalities in multimedia communication. In Paper presented at the AAAI Symposium on Human-Computer Interfaces, Stanford, 1990.Google Scholar
- E. Klein and L.A. Pineda. Semantics and graphical information. In D. Diaper, editor, Human-Computer Interaction — INTERACT’90, Amsterdam, 1990. Elsevier.Google Scholar
- J. Lee and H. Zeevat. Integrating natural language and graphics in dialogue. In D. Diaper, editor, Human-Computer Interaction — INTERACT’90, Amsterdam, 1990. Elsevier.Google Scholar