A Third Modality of Natural Language?



In the late Eighties the Natural Language Processing community began appreciating the role of multimodality in interactive systems. Intelligent multimodal systems are systems that integrate natural language (generally so far keyboard-based input, shortly also voice) with other media such as gestures in input or graphics in output. The perspective of what can be called visible interactive communication is discussed and considered as a possible new modality of natural language, after the spoken and the written ones. This should not be confused with the type of hyper-media that are now being developed. There, basically, the interface space is finite, even if one dimension may be added. Here the infinite creativity of human language is potentially preserved as the fundamental communication instrument.

Key words

natural language multimedia human-computer interaction communication modalities dialogue generation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Allgayer, J., Harbusch, K., Kobsa, A., Reddig, C, Reithinger, N. & Schmauks, D. (1989). XTRA: A Natural-Language Access System to Expert Systems. Intl. J. on Man-Machine Studies 31: 161–195.CrossRefGoogle Scholar
  2. Appelt, D. E. (1985). Planning English Sentences. Cambridge University Press.CrossRefGoogle Scholar
  3. Arens, Y., Miller, L., Shapiro, S. C. & Sondheimer, N. K. (1988). Automatic Construction of User Interface Displays. In Proceedings of The Seventh AAAI Conference. St. Paul, Minnesota.Google Scholar
  4. Arens, Y. & Hovy, E. (1990). How to Describe What? Towards a Theory of Modality Utilization. In Proceedings of The Twelfth Cognitive Science Conference. Cambridge, MA.Google Scholar
  5. Barrett, E. (1989). Textual Intervention, Collaboration, and the Online Environment. In Barrett, E. (ed.) The Society of Text. MIT Press.Google Scholar
  6. Bush, V. (1945). As We May Think. Atlantic Monthly 7: 101–108.Google Scholar
  7. Carenini, G., Pianesi, F., Ponzi, M. & Stock, O. (1993). Natural Language Generation and Hypertext Access. Applied Artificial Intelligence 7: 135–164.CrossRefGoogle Scholar
  8. Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C. N., Sullivan, J. W., Gargan Jr, R. A., Schlossberg, J. L. & Tyler, S. W. (1989). Synergetic Use of Direct Manipulation and Natural Language. In Proceedings of The CHI’89, 227–233. Austin, Texas.Google Scholar
  9. Conklin, J. (1987). Hypertext: An Introduction and Survey. IEEE Computer 20(9).Google Scholar
  10. Dahlbäck, N. & Jönsson, A. (1989). Empirical Studies of Discourse Representations for Natural Language Interfaces. In Proceedings of The Fourth Conf. European Chapter of the ACL, 291–307. Association for Computational Linguistics.Google Scholar
  11. Dale, R. (1992). Visible Language: Multimodal Constraints in Information Presentation. In Dale, R., Hovy, E., Rosner, D. & Stock, O. (eds.) Aspects of Automated Language Generation, 281–283. Springer.CrossRefGoogle Scholar
  12. Feiner, S. & McKeown, K. (1990). Coordinating Text and Graphics in Explanation Generation. In Proceedings of The AAAI-90, 442–449. Boston, MA.Google Scholar
  13. Franconi, E. (1990). The YAK Manual: Yet Another KRAPFEN. IRST Manual 9003-01. IRST. Trento, Italy.Google Scholar
  14. Grosz, B. J. & Sidner, C. L. (1986). Attention, Intentions and the Structure of Discourse. Computational Linguistics 12(3): 175–204.Google Scholar
  15. Halasz, F. G. (1988). Reflections on NoteCards, Seven Issues for the Next Generation of Hypermedia Systems. Communications of the ACM 31(7).Google Scholar
  16. Hollan, J., Rich, E., Hill, W., Wroblenski, D., Wilker, W., Wittenburg, K. & Grudin, J. (1988). An Introduction to Hits: Human Interface Tool Suite. MCC, Tech. Rep. ACA-HI-406-88. Austin, Texas.Google Scholar
  17. Hovy, E. (1988). Generating Natural Language under Pragmatic Constraints. Lawrence Erlbaum Associates.Google Scholar
  18. Hovy, E., & Arens, Y. (1991). Automatic Generation of Formatted Text. In Proceedings of The Ninth AAAI Conference, 92–97. Anaheim, CA.Google Scholar
  19. Kass, R. & Finin, T. (1988). Modeling the User in Natural Language Systems. Computational Linguistics 14(3): 5–22.Google Scholar
  20. Kobsa, A. & Wahlster, W. (eds.) (1989). User Models in Dialog Systems. Springer.zbMATHGoogle Scholar
  21. Kurohashi, S., Nagao, M., Sato, S. & Murakami, M. (1992). A Method of Automatic Hypertext Construction from an Encyclopedic Dictionary of a Specific Field. In Proceedings of The Third Conference on Applied Natural Language Processing, 239–240. Trento, Italy: Association for Computational Linguistics.Google Scholar
  22. Lay, K. Y., Malone, T. W. & Yu, K. C. (1988). Object Lens: A “Spreadsheet” for Cooperative Work. ACM Transaction on Office Information Systems 6: 332–353.CrossRefGoogle Scholar
  23. Lavelli, A., Magnini, B. & Strapparava, C. (1992). An Approach to Multilevel Semantics for Applied Systems. In Proceedings of The Third Conference on Applied Natural Language Processing, 17–24. Trento, Italy: Association for Computational Linguistics.Google Scholar
  24. Lavelli, A. & Stock, O. (1990). When Something is Missing: Ellipsis, Coordination and the Chart. In Proceedings of The Thirteenth International Conference on Computational Linguistics, COLING-90, 184–189. Helsinki, Finland: Association for Computational Linguistics.Google Scholar
  25. Lemke, A. & Fischer, G. A. (1990). Cooperative Problem Solving System for User Interface Design. In Proceedings of The Eighth AAAI Conference, 479–484. Boston, MA.Google Scholar
  26. MacLaughlin, D. M. & Shaked, V. (1989). Natural Language Text Generation in Semi-Automated Forces. BBN Report 7092.Google Scholar
  27. Mackinlay, J. D. (1986). Automatic Design of Graphical Presentations. Ph.D. dissertation, Stanford University.Google Scholar
  28. Maybury, M. T. (1991). Planning Multimedia Explanations Using Communicative Acts. In Proceedings of The Ninth AAAI Conference, 61–66. Anaheim, CA.Google Scholar
  29. Maybury, M. T. (ed.) (1993). Intelligent Multimedia Interfaces. AAAI Press/MIT Press: Menlo Park, CA/Cambridge, MA.Google Scholar
  30. Moore, J. D. & Swartout, W. (1990). Pointing: A way toward explanation dialogue. In Proceedings of The Eighth AAAI Conference, 457–464. Boston, MA.Google Scholar
  31. Nelson, T. H. (1981). Literary Machines. Swarthmore, PA 19801: Nelson, T., P.O. Box 128.Google Scholar
  32. Oviatt, S. L. & Cohen, P. R. (1989). The Effects of Interaction on Spoken Discourse. In Proceedings of The Twenty-Seventh Meeting of the ACL, 126–134. Vancouver, Canada: Association for Computational Linguistics.Google Scholar
  33. Oviatt, S. L. & Cohen, P. R. (1991). discourse Structure and Performances Efficiency in Interactive and Noninteractive Spoken Modalities. Computer Speech and Language 5(4): 297–326.CrossRefGoogle Scholar
  34. Paris, C. L. (1987). Combining discourse Strategies to Generate Descriptions to Users Along a Naive/Expert Spectrum. In Proceedings of The IJCAI-87, 626–632. Milan, Italy.Google Scholar
  35. Pianesi, F. (1993). Head Driven Bottom Up Generation and Government and Binding: A Unified Perspective. In Horacek, H. & Zock, M. (eds.) New Concepts in Natural Language Generation, 187–214. Printer: London.Google Scholar
  36. Reiter, E., Mellish, C. & Levine, J. (1992). Automatic Generation of On-Line Documentation in the IDAS Project. In Proceedings of The Third Conference on Applied natural Language Processing, 64–71. Trento, Italy: Association for Computational Linguistics.Google Scholar
  37. Samek-Lodovici, V. & Strapparava, C. (1990). Identifying Noun Phrase References, the Topic Module of the ALFRESCO System. In Proceedings of The ECAI-90, 573–578. Stockholm, Sweden.Google Scholar
  38. Schmauks, D. (1987). Natural and Simulated Pointing. In Proceedings of The Third Conference of the European Chapter of the ACL, 179–183. Association for Computational Linguistics.Google Scholar
  39. Slack, J. & Conati, C. (to appear). Modeling Interest: Exploration of an Information Space. To appear in Acta Phsychologica on Cognitive Ergonomics.Google Scholar
  40. Stefik, M., Foster, G., Bobrow, D., Kahn, K., Lanning, S. & Suchman, L. (1987). Beyond the chalkboard: Using computers to support collaboration and problem solving in meetings. Communication of the ACM 30(1): 32–47.CrossRefGoogle Scholar
  41. Stock, O. (1989). Parsing with Flexibility, Dynamic Strategies and Idioms in Mind. Computational Linguistics 15(1): 1–18.MathSciNetGoogle Scholar
  42. Stock, O. (1991). Natural Language and Exploration of an Information Space: the ALFRESCO Interactive System. In Proceedings of The IJCAI-91, 972–978. Sydney, Australia: Morgan Kaufmann.Google Scholar
  43. Strapparava, C. (1991). From Scopings to Interpretation: The Semantic Interpretation within the ALFRESCO System. In Ardizzone, Gaglio & Sorbello (eds.) Trends in Artificial Intelligence, Proceedings of The Second Congress of the Italian Association for Artificial Intelligence, 281–290. Springer-Verlag.Google Scholar
  44. Stringa, L. (1990). An Integrated Approach to Artificial Intelligence. IRST-Technical Report 9012–11. IRST: Trento, Italy.Google Scholar
  45. Suchman, L. A. (1987). Plans and Situated Action. Cambridge University Press.Google Scholar
  46. Wahlster, W. (1988). User and Discourse Models for Multimodal Communication. In Sullivan, J. W. & Tyler, S. W. (eds.) Architectures for Intelligent Interfaces: Elements and Prototypes. Addison-Wesley.Google Scholar
  47. Wahlster, W., André E., Bandyopadhyay, S., Graf, W. & Rist, T. (1992). WIP: The Coordinated Generation of Multimodal Presentations from a Common Representation. In Slack, J., Ortony, A. & Stock, O. (eds.) Communication from Artificial Intelligence Perspective; Theoretical and Applied Issues, 121–143. Springer Verlag.CrossRefGoogle Scholar
  48. Weiser, M. (1991). The Computer for the 21st Century. Scientific American 265(3): 66.CrossRefGoogle Scholar
  49. Zancanaro, M., Stock, O. & Strapparava, C. (1993). Dialogue Cohesion Sharing and Adjusting in an Enhanced Multimodal Environment. In Proceedings of The IJCA1–93, 1230–1236. Chambery, France: Morgan Kaufmann.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1995

Authors and Affiliations

  1. 1.IRST — Istituto per la Ricerca Scientifica e TecnologicaPovo — TrentoItaly

Personalised recommendations