Abstract
In the late Eighties the Natural Language Processing community began appreciating the role of multimodality in interactive systems. Intelligent multimodal systems are systems that integrate natural language (generally so far keyboard-based input, shortly also voice) with other media such as gestures in input or graphics in output. The perspective of what can be called visible interactive communication is discussed and considered as a possible new modality of natural language, after the spoken and the written ones. This should not be confused with the type of hyper-media that are now being developed. There, basically, the interface space is finite, even if one dimension may be added. Here the infinite creativity of human language is potentially preserved as the fundamental communication instrument.
This is a revised version of an invited talk that was delivered at the 10th European Conference on Artificial Intelligence in Vienna, and published in the Proceedings (B. Neuman, ed.), John Wiley & Sons, 1992, pp. 853–862.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Allgayer, J., Harbusch, K., Kobsa, A., Reddig, C, Reithinger, N. & Schmauks, D. (1989). XTRA: A Natural-Language Access System to Expert Systems. Intl. J. on Man-Machine Studies 31: 161–195.
Appelt, D. E. (1985). Planning English Sentences. Cambridge University Press.
Arens, Y., Miller, L., Shapiro, S. C. & Sondheimer, N. K. (1988). Automatic Construction of User Interface Displays. In Proceedings of The Seventh AAAI Conference. St. Paul, Minnesota.
Arens, Y. & Hovy, E. (1990). How to Describe What? Towards a Theory of Modality Utilization. In Proceedings of The Twelfth Cognitive Science Conference. Cambridge, MA.
Barrett, E. (1989). Textual Intervention, Collaboration, and the Online Environment. In Barrett, E. (ed.) The Society of Text. MIT Press.
Bush, V. (1945). As We May Think. Atlantic Monthly 7: 101–108.
Carenini, G., Pianesi, F., Ponzi, M. & Stock, O. (1993). Natural Language Generation and Hypertext Access. Applied Artificial Intelligence 7: 135–164.
Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C. N., Sullivan, J. W., Gargan Jr, R. A., Schlossberg, J. L. & Tyler, S. W. (1989). Synergetic Use of Direct Manipulation and Natural Language. In Proceedings of The CHI’89, 227–233. Austin, Texas.
Conklin, J. (1987). Hypertext: An Introduction and Survey. IEEE Computer 20(9).
Dahlbäck, N. & Jönsson, A. (1989). Empirical Studies of Discourse Representations for Natural Language Interfaces. In Proceedings of The Fourth Conf. European Chapter of the ACL, 291–307. Association for Computational Linguistics.
Dale, R. (1992). Visible Language: Multimodal Constraints in Information Presentation. In Dale, R., Hovy, E., Rosner, D. & Stock, O. (eds.) Aspects of Automated Language Generation, 281–283. Springer.
Feiner, S. & McKeown, K. (1990). Coordinating Text and Graphics in Explanation Generation. In Proceedings of The AAAI-90, 442–449. Boston, MA.
Franconi, E. (1990). The YAK Manual: Yet Another KRAPFEN. IRST Manual 9003-01. IRST. Trento, Italy.
Grosz, B. J. & Sidner, C. L. (1986). Attention, Intentions and the Structure of Discourse. Computational Linguistics 12(3): 175–204.
Halasz, F. G. (1988). Reflections on NoteCards, Seven Issues for the Next Generation of Hypermedia Systems. Communications of the ACM 31(7).
Hollan, J., Rich, E., Hill, W., Wroblenski, D., Wilker, W., Wittenburg, K. & Grudin, J. (1988). An Introduction to Hits: Human Interface Tool Suite. MCC, Tech. Rep. ACA-HI-406-88. Austin, Texas.
Hovy, E. (1988). Generating Natural Language under Pragmatic Constraints. Lawrence Erlbaum Associates.
Hovy, E., & Arens, Y. (1991). Automatic Generation of Formatted Text. In Proceedings of The Ninth AAAI Conference, 92–97. Anaheim, CA.
Kass, R. & Finin, T. (1988). Modeling the User in Natural Language Systems. Computational Linguistics 14(3): 5–22.
Kobsa, A. & Wahlster, W. (eds.) (1989). User Models in Dialog Systems. Springer.
Kurohashi, S., Nagao, M., Sato, S. & Murakami, M. (1992). A Method of Automatic Hypertext Construction from an Encyclopedic Dictionary of a Specific Field. In Proceedings of The Third Conference on Applied Natural Language Processing, 239–240. Trento, Italy: Association for Computational Linguistics.
Lay, K. Y., Malone, T. W. & Yu, K. C. (1988). Object Lens: A “Spreadsheet” for Cooperative Work. ACM Transaction on Office Information Systems 6: 332–353.
Lavelli, A., Magnini, B. & Strapparava, C. (1992). An Approach to Multilevel Semantics for Applied Systems. In Proceedings of The Third Conference on Applied Natural Language Processing, 17–24. Trento, Italy: Association for Computational Linguistics.
Lavelli, A. & Stock, O. (1990). When Something is Missing: Ellipsis, Coordination and the Chart. In Proceedings of The Thirteenth International Conference on Computational Linguistics, COLING-90, 184–189. Helsinki, Finland: Association for Computational Linguistics.
Lemke, A. & Fischer, G. A. (1990). Cooperative Problem Solving System for User Interface Design. In Proceedings of The Eighth AAAI Conference, 479–484. Boston, MA.
MacLaughlin, D. M. & Shaked, V. (1989). Natural Language Text Generation in Semi-Automated Forces. BBN Report 7092.
Mackinlay, J. D. (1986). Automatic Design of Graphical Presentations. Ph.D. dissertation, Stanford University.
Maybury, M. T. (1991). Planning Multimedia Explanations Using Communicative Acts. In Proceedings of The Ninth AAAI Conference, 61–66. Anaheim, CA.
Maybury, M. T. (ed.) (1993). Intelligent Multimedia Interfaces. AAAI Press/MIT Press: Menlo Park, CA/Cambridge, MA.
Moore, J. D. & Swartout, W. (1990). Pointing: A way toward explanation dialogue. In Proceedings of The Eighth AAAI Conference, 457–464. Boston, MA.
Nelson, T. H. (1981). Literary Machines. Swarthmore, PA 19801: Nelson, T., P.O. Box 128.
Oviatt, S. L. & Cohen, P. R. (1989). The Effects of Interaction on Spoken Discourse. In Proceedings of The Twenty-Seventh Meeting of the ACL, 126–134. Vancouver, Canada: Association for Computational Linguistics.
Oviatt, S. L. & Cohen, P. R. (1991). discourse Structure and Performances Efficiency in Interactive and Noninteractive Spoken Modalities. Computer Speech and Language 5(4): 297–326.
Paris, C. L. (1987). Combining discourse Strategies to Generate Descriptions to Users Along a Naive/Expert Spectrum. In Proceedings of The IJCAI-87, 626–632. Milan, Italy.
Pianesi, F. (1993). Head Driven Bottom Up Generation and Government and Binding: A Unified Perspective. In Horacek, H. & Zock, M. (eds.) New Concepts in Natural Language Generation, 187–214. Printer: London.
Reiter, E., Mellish, C. & Levine, J. (1992). Automatic Generation of On-Line Documentation in the IDAS Project. In Proceedings of The Third Conference on Applied natural Language Processing, 64–71. Trento, Italy: Association for Computational Linguistics.
Samek-Lodovici, V. & Strapparava, C. (1990). Identifying Noun Phrase References, the Topic Module of the ALFRESCO System. In Proceedings of The ECAI-90, 573–578. Stockholm, Sweden.
Schmauks, D. (1987). Natural and Simulated Pointing. In Proceedings of The Third Conference of the European Chapter of the ACL, 179–183. Association for Computational Linguistics.
Slack, J. & Conati, C. (to appear). Modeling Interest: Exploration of an Information Space. To appear in Acta Phsychologica on Cognitive Ergonomics.
Stefik, M., Foster, G., Bobrow, D., Kahn, K., Lanning, S. & Suchman, L. (1987). Beyond the chalkboard: Using computers to support collaboration and problem solving in meetings. Communication of the ACM 30(1): 32–47.
Stock, O. (1989). Parsing with Flexibility, Dynamic Strategies and Idioms in Mind. Computational Linguistics 15(1): 1–18.
Stock, O. (1991). Natural Language and Exploration of an Information Space: the ALFRESCO Interactive System. In Proceedings of The IJCAI-91, 972–978. Sydney, Australia: Morgan Kaufmann.
Strapparava, C. (1991). From Scopings to Interpretation: The Semantic Interpretation within the ALFRESCO System. In Ardizzone, Gaglio & Sorbello (eds.) Trends in Artificial Intelligence, Proceedings of The Second Congress of the Italian Association for Artificial Intelligence, 281–290. Springer-Verlag.
Stringa, L. (1990). An Integrated Approach to Artificial Intelligence. IRST-Technical Report 9012–11. IRST: Trento, Italy.
Suchman, L. A. (1987). Plans and Situated Action. Cambridge University Press.
Wahlster, W. (1988). User and Discourse Models for Multimodal Communication. In Sullivan, J. W. & Tyler, S. W. (eds.) Architectures for Intelligent Interfaces: Elements and Prototypes. Addison-Wesley.
Wahlster, W., André E., Bandyopadhyay, S., Graf, W. & Rist, T. (1992). WIP: The Coordinated Generation of Multimodal Presentations from a Common Representation. In Slack, J., Ortony, A. & Stock, O. (eds.) Communication from Artificial Intelligence Perspective; Theoretical and Applied Issues, 121–143. Springer Verlag.
Weiser, M. (1991). The Computer for the 21st Century. Scientific American 265(3): 66.
Zancanaro, M., Stock, O. & Strapparava, C. (1993). Dialogue Cohesion Sharing and Adjusting in an Enhanced Multimodal Environment. In Proceedings of The IJCA1–93, 1230–1236. Chambery, France: Morgan Kaufmann.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1995 Springer Science+Business Media Dordrecht
About this chapter
Cite this chapter
Stock, O. (1995). A Third Modality of Natural Language?. In: Mc Kevitt, P. (eds) Integration of Natural Language and Vision Processing. Springer, Dordrecht. https://doi.org/10.1007/978-94-011-0445-6_7
Download citation
DOI: https://doi.org/10.1007/978-94-011-0445-6_7
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-010-4199-7
Online ISBN: 978-94-011-0445-6
eBook Packages: Springer Book Archive