The Design of a Model-Based Multimedia Interaction Manager



We describe here the conceptual design of Cicero, an application-independent human-computer interaction manager that performs run-time media coordination and allocation, so as to adapt dynamically to the presentation context; knows what it is presenting, so as to maintain coherent extended human-machine dialogues; and is plug-in compatible with host information resources such as “briefing associate” workstations, expert systems, databases, etc., as well as with multiple media such as natural language, graphics, etc. The system design calls for two linked reactive planners that coordinate the actions of the system’s media and information sources. To enable presentational flexibility, the capabilities of each medium and the nature of the contents of each information source are semantically modeled as Virtual Devices — abstract descriptions of device I/O capabilities — and abstract information types respectively in a single uniform knowledge representation framework. These models facilitate extensibility by supporting the specification of new interaction behaviors and the inclusion of new media and information sources.

Key words

human-computer interaction management presentation planning information integration discourse structure information-to-medium allocation Virtual Devices 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Agre, P. E. & Chapman, D. (1987). Pengi: An Implementation of a Theory of Activity. Proceedings of The 6th AAAI, 196–201. Seattle, WA.Google Scholar
  2. Arens, Y. & Hovy, E. (1990). How to Describe What? Towards a Theory of Modality Utilization. Proceedings of the 12th Annual Conference of the Cognitive Science Society, 487–494. Cambridge, MA.Google Scholar
  3. Arens, Y., Miller, L., Shapiro, S. C. & Sondheimer, N. K. (1988). Automatic Construction of User-Interface Displays. Proceedings of The 7th AAAI Conference, 808–813. St. Paul, MN.Google Scholar
  4. Arens, Y., Miller, L., & Sondheimer, N. K. (1991). Presentation Design Using an Integrated Knowledge Base. In Sullivan, Joseph, W. and Tyler, Sherman W. (eds.) Intelligent User Interfaces, 241–258. Addison-Wesley: Reading.Google Scholar
  5. Arens, Y., Hovy, E. H. & Trimble, J. (1992). An Automatic Presentation Builder. Unpublished ms, USC Information Sciences Institute.Google Scholar
  6. Arens, Y., Hovy, E. H. & Vossers, M. (1993). On the Knowledge Underlying Multimedia Presentations. In Mark Maybury (ed.) Intelligent Multimedia Interfaces, 280–306. AAAI Press.Google Scholar
  7. Arens, Y., Hovy, E. H. & Van Mulken, S. (1993). Structure and Rules in Automated Multimedia Presentation Planning. Proceedings of IJCAI-93, 231–235. Chambéry, France.Google Scholar
  8. Arens, Y., Chee, C. Y., Hsu, C-N. & Knoblock, C. A. (1993). Retrieving and Integrating Data from Multiple Information Sources. International Journal of Intelligent and Cooperative Information Systems 2(2): 127–158.CrossRefGoogle Scholar
  9. Burger, J. & Marshall, R. (1991). AIMI: An Intelligent Multimedia Interface. Proceedings of The 9th AAAI Conference, 23–28. Anaheim, CA.Google Scholar
  10. Erman, L. D., Lark, J. S. & Hayes-Roth, F. (1987). ABE: An Environment for Engineering Intelligent Systems. Technical Report TTR-ISE-87–106, Teknowledge Inc., Palo Alto, CA.Google Scholar
  11. Feiner, S. & McKeown, K. R. (1990). Coordinating Text and Graphics in Explanation Generation. Proceedings of The 8th AAAI, 442–449.Google Scholar
  12. Feiner, S. (1991). An Architecture for Knowledge-Based Graphical Interfaces. ACM/SIGCHI Workshop on Architectures for Intelligent Interfaces: Elements and Prototypes. In Sullivan, Joseph W. & Tyler, Sherman W. (eds.) Intelligent User Interfaces, 259–279. Addison-Wesley: Reading.Google Scholar
  13. Foley, J., Kim, W., Kovacevic, S. & Murray, K. (1991). UIDE — An Intelligent User Interface Design Environment. In Sullivan, J. & Tyler, S. (eds.) Archietectures for Intelligent User Interfaces: Elements and Prototypes, 339–384. Addison-Wesley: Reading.Google Scholar
  14. Hovy, E. H. (1988). Planning Coherent Multisentential Text. Proceedings of The 26th Annual Meeting of the Association for Computational Linguistics, 163–169. Buffalo, N.Y.Google Scholar
  15. Hovy, E. H. & Arens, Y. (1990). When is a Picture Worth a Thousand Words? — Allocation of Modalities in Multimedia Communication. Presented at the AAAI Symposium on Human-Computer Interaction, Stanford University.Google Scholar
  16. Hovy, E. H. & Arens, Y. (1991). Automatic Generation of Formatted Text. Proceedings of The 10th AAAI, 92–97. Anaheim, CA.Google Scholar
  17. Hovy, E. H. & Arens, Y. (1993). The Planning Paradigm Required for Automated Multimedia Presentation Planning. In Human-Computer Collaboration: Reconciling Theory, Synthesizing Practice. Papers from the 1993 Fall Symposium Series, AAAI Technical Report FS-93–05.Google Scholar
  18. Hovy, E. H., Lavid, J., Maier, E., Mittal, V. & Paris, C. L. (1992). Employing Knowledge Resources in a New Text Planning Architecture. In Dale, R., Hovy, E., Rösner, D. & Stock, O. (eds.) Aspects of Automated Natural Language Generation, 57–72. Heidelberg: Springer Verlag Lecture Notes in AI number 587.CrossRefGoogle Scholar
  19. Intelligent User Interfaces. Arens Y., Feiner, S., Foley, J., Hovy, E., John, B., Neches, R., Pausch, R., Schorr, H. & Swartout, W. (1991). USC/ISI Research Report No. ISI/RR-91-288, September 30, 1991.Google Scholar
  20. Luo, P., Szekely, P. & Neches, R. (1993). Management of Interface Design in Humanoid. Proceedings of INTERCHI’93, 107–114. Amsterdam, The Netherlands.Google Scholar
  21. MacGregor, R. (1988). A Deductive Pattern Matcher. Proceedings of AAAI-88, 403–408. St. Paul, MN.Google Scholar
  22. Mackinlay, J. (1986). Automatic Design of Graphical Presentations. Ph.D. dissertation, Stanford University.Google Scholar
  23. Mackinlay, J. (1986). Automating the Design of Graphical Presentation of Relational Information. ACM Transactions on Graphics Special Issue on User Interface Software Part I. 5(2): 110–141.Google Scholar
  24. Mann, W. C. & Thompson, S. A. (1988). Rhetorical Structure Theory: Toward a Functional Theory of Text Organization. Text 8(3): 243–281.Google Scholar
  25. Neal, J. G., Shapiro, S. C, Thielman, J. R., Lammens, J. M., Funke, D. J., Byoun, J. S., Paul, R., Dobes, Z., Glanowski, S. & Summers, M. S. (1990). Intelligent Multi-Media Integrated Interface Project. SUNY Buffalo and RADC Technical Report TR-90-128.Google Scholar
  26. Neches, R., Foley, J., Szekely, P., Sukaviriya, P., Luo, P., Kovacevic, S. & Hudson, S. (1993). Knowledgeable Development Environments Using Shared Design Models. Proceedings of the International Workshop on Intelligent User Interfaces, 183–190. Orlando, FL.Google Scholar
  27. Olsen, D. (1986). MIKE: The Menu Interaction Kontrol Environment. ACM Transactions on Graphics. 17(3): 43–50.MathSciNetGoogle Scholar
  28. The Penman Project. (1988). The Penman Primer, User Guide, and Reference Manual. Unpublished USC/ISI documentation.Google Scholar
  29. Schank, R. & Abelson, R. (1977). Scripts, Plans, Goals and Understanding: An Inquiry into Human Knowledge Structures. Hillsdale: Lawrence Erlbaum Associates.zbMATHGoogle Scholar
  30. Szekely, P., Luo, P. & Neches, R. (1993). Beyond Interface Builders: Model-Based Interface Design. Proceedings of INTERCHI’93, 383–390. Amsterdam, the Netherlands.Google Scholar
  31. Vossers, M. (1991). Automatic Generation of Formatted Text and Line Drawings. Master’s thesis, University of Nijmegen, The Netherlands.Google Scholar
  32. Wahlster, W., Andre, E., Bandyopadhyay, S., Graf, W. & Rist, T. (1991). WIP: The Coordinated Generation of Multimodal Presentations from a Common Representation. In Ortony, A., Slack, J. & Stock, O. (eds.), Computational Theories of Communication and their Applications, 190–213. Springer Verlag, Berlin.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1995

Authors and Affiliations

  1. 1.Information Sciences InstituteUniversity of Southern CaliforniaMarina del ReyUSA

Personalised recommendations