Temporal Codes within a Typology of Cooperation Between Modalities



The Human-Machine Communication Department of LIMSI (LIMSI Report 1994), a laboratory of the French National Scientific Research Agency (CNRS), conducts research in the field of Pattern Recognition, Artificial Intelligence and Pattern Generation. Several approaches are currently investigated there for studying interactions between modalities: studies about the cognitive processes involved in human intermodal tasks (Denis and Cocude 1992; Gryl 1994; Daniel et al. 1994), image and language interactions for learning (Bordeaux 1993), human factors (Castaing and True 1993), symbolic approaches for designing multimodal human-computer interfaces (Bellik and Teil 1993; Barès et al. 1992), spatial reasoning (Ligozat 1992; Briffault 1992).

Key words

multimodality transfer equivalence specialisation redundancy complementarity Guided Propagation networks binding through synchrony 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. AAAI-NLP-VP (1994). Working notes of the workshop on the Integration of Natural Language and Vision Processing. Chair P. Mc Kevitt. Twelfth National Conf. on Artificial Intelligence, august, 2nd/3rd, Seattle.Google Scholar
  2. Arnold, M. (1993). Linguistic and Spatial Representation of a Scene: An Experiment in Image Synthesis. Sémiotiques 4: 7–29. In French.Google Scholar
  3. Barès, M., Nèel, F., Teil, D. & Martin, J.-C. (1992). Conceptual Approach to Man-Machine Interaction: Application to Pilote’s Assistance. In Proceedings of The Interface Between Real and Virtual Worlds Conference, 571–590. Montpellier, France. In French.Google Scholar
  4. Bellik, Y. & Teil, D. (1993). A Multimodal Dialogue Controller for Multimodal User Interface Management System Application: A Multimodal Window Manager. Adjunct proceedings of INTERCHI’93, 93–94. ACM Press: Amsterdam.Google Scholar
  5. Béroule D. (1990). Guided Propagation: Current State of Theory and Application. In Soulié, F. Fogelman & Hérault, J. (eds.) Neurocomputing, Vol. F 68, 241–260. NATO ASI Series; Berlin Heidelberg: Springer-Verlag.Google Scholar
  6. Béroule, D. (1988). The Never-Ending Learning. In Eckmiller, R. & Malsburg, C.v.d. (eds.) Neural Computers, 219–230. NATO ASI Series vil F41, Berlin: Springer-Verlag.Google Scholar
  7. Béroule, D. (1989). The Adaptive, Dynamic and Associative Memory Model: A Possible Future Tool for Vocal Human-Computer Communication. In Taylor, M., Néel, F. & Bouwhuis, D. G. (eds.) The Structure of Multimodal Dialogue, 189–202. Elsevier Science Publisher B.V.: North HollandGoogle Scholar
  8. Béroule, D., Von Hoe, R. & Ruellan, H. (1994). A Guided Propagation Model of Reading, Instituut voor Perceptie Onderzoek (IPO), Annual Progress Report NB0228, Eindhoven, The Netherlands.Google Scholar
  9. Bordeaux, F. (1993). Building Categories from Visual Experiences. In Proceedings of The IEEE International Conference on Systems, Man and Cybernetics. October, Le Touquet, France.Google Scholar
  10. Briffault, X. (1992). Computer Modelisation of the Expression of Localisation in Natural Language. Ph.D. diss. Paris VI University. 255 pp. In French.Google Scholar
  11. Castaing, M. F. & True-Martini, D. (1993). Constraints in the Recording of Speech Data Bases. In Proceedings of The Internat. Symp. on Spoken Dialogue, 29–31. Tokyo, Japon.Google Scholar
  12. Daniel, M. -P., Carité, L. & Denis, M. (1994). Modes of Linearization in the Description of Spatial Configurations. In Portugali, J. (ed.) The Construction of Cognitive Maps. Kluwer: Dordrecht, The Netherlands.Google Scholar
  13. Denis, M. & Cocude, M. (1992). Structural Properties of Visual Images Constructed form Poorly or Well-structured Verbal Descriptions. Memory and Cognition 20: 497–506.CrossRefGoogle Scholar
  14. Escande, P., Béroule, D. & Blanchet, P. (1991). Speech Recognition Experiments with Guided Propagation. Proc. of IJCNN’91, Singapour.Google Scholar
  15. Faure, C. & Arnold, M. (1993). Man-Machine Interaction and Minimum Principle. In Proceedings of The IHM’93, 3–8 Lyon. In French.Google Scholar
  16. GDR-PRC-CHM (1993). Activity Report About Multimodal Human-Computer Interfaces, CNRS-MESR, Coutaz, J. & Caelen, J. In French. In 1994: C. Faure is at the head of the CHM Pôle, F. Poirier at the head of the multimodal working group.Google Scholar
  17. Grumbach, A. (1994). Artificial Cognition. Addison-Wesley France. In French.Google Scholar
  18. Gryl, A. (1994). Cognitive Approach of Route Descriptions. In Pre-proceedings of The Fith European Workshop on Imagery and Cognition (EWIC), 6–8 Saarbrücken, Germany, Univ. of the Saarland.Google Scholar
  19. Huls, C, Bos, E. & Dijkstra, A. (1994). Talking Pictures. An Empirical Study into the Usefulness of Natural Language Output in a Graphical Interface. Working notes of the AAAI-94 workshop on the Integration of Natural Language and Vision Processing, 83–90. Chair P. McKevitt. Twelfth National Conf. on Artificial Intelligence, Seattle, Washington.Google Scholar
  20. Hummel, J. E. & Biederman, I. (1990). Dynamic Binding: A Basis for the Representation of Shape by Neural Networks. 12th Annual Meeting of the Cognitive Science Society. Cambridge, MA.Google Scholar
  21. Images and Languages. (1993). Proceedings of The Interdisciplinary Workshop “Image and Language: Multimodality and Cognitive Modelisation”. M. Denis and M. Carfantan (eds.) CNRS, Paris, 1993. In French.Google Scholar
  22. Jackendoff, R. (1987). On Beyond Zebra: The Relation of Linguistic and Visual Information. Cognition 26(2): 89–114.CrossRefGoogle Scholar
  23. Lainé, A. & Béroule, D. (1994). Towards Active Perception for Automatic Speech Recognition. In Proceedings of The NSI’94 Chamonix, 23–26. In French.Google Scholar
  24. Ligozat, G. (1992). Strategies for Route Description: An Interdisciplinary Approach. ECAI’92 workshop W19, Spatial Concepts: Connecting Cognitive Theories with Formal Representations. Vienna, Austria.Google Scholar
  25. LIMSI, Scientific Report, Human-Machine Communication Department, Orsay, 1994.Google Scholar
  26. Martin, J. C. (1994). A Study Based on Several Types of Coorperation between Modalities and Binding through Synchrony. Working notes of the AAAI-94 workshop on the Integration of Natural Language and Vision Processing 181–184. Chair P. Mc Kevitt. Twelfth National Conf. on Artificial Intelligence, Seattle, Washington.Google Scholar
  27. Martin, J. C. & Béroule, D. (1993). Types and Goals of Co-Operation Between Modalities. Proceedings of The IHM’93, 17–22. Lyon. In French.Google Scholar
  28. Nenov, V. I. & Dyer, M. G., (1988). DETE: Connectionist/Symbolic Model of Visual and Verbal Association. In Proceedings of The Connexionnist Models Summer School 1988. CMU.Google Scholar
  29. Nigay, L. & Coutaz, J. (1993). A Design Space For Multimodal Systems: Concurrent Processing and Data Fusion, conf. Proceedings of The INTERCHI’93, 172–178. ACM Press: Amsterdam.Google Scholar
  30. Nioche, C, Tassin, J-P. & Béroule, D. (1994). Towards a Functional Modelisation of Neuro-modulation. In Proceedings of The NSI’94, 231–234. Chamonix, In French.Google Scholar
  31. Novak, G. S. & Bulko, W. C. (1990). Understanding Natural Language with Diagrams. In Proceedings of The 8th National Conf. on Artificial Intelligence Vol. 1, 465–470, AAAI Press/MIT Press.Google Scholar
  32. ÓNualláin, S. & Smith, A. G. (1994). Artificial Intelligence Review 8(2–3): 113–122 (this volume).CrossRefGoogle Scholar
  33. Robert-Ribes, J., Schwartz, J. L. & Escudier, P. (1994). Artificial Intelligence Review 9 (this volume).Google Scholar
  34. Roques, M. (1994). Dynamic Grammatical Representations in Guided Propagation Networks. In Lecture Notes in Artificial Intelligence 862, Carrasco, R. C., Oncina, J. (eds.) Grammatical Inference and Applications, 189–202. Second International Colloqium, ICGI-94, Alicante, Spain.Google Scholar
  35. Shastri, L. & Ajjanagadde, V. (1993). From Simple Associations to Systematic Reasoning: A Connectionist Representation of Rules, Variables and Dynamic Bindings Using Temporal Synchrony. Behavioral and Brain Sciences 16: 417–494.CrossRefGoogle Scholar
  36. von der Malsburg, C. & Schneider, W. (1986). A Neural Cocktail-Party Processor. Biol. Cybern. 54: 29–40.CrossRefGoogle Scholar
  37. Westerlund, P., Béroule, D. & Roques, M. (1994). Experiments of Robust Parsing Using a Guided Propagation Network. In Proceedings of The International Conference on New Methods in Language Processing (NEMLAP), Manchester.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1995

Authors and Affiliations

  1. 1.LIMSI (Laboratoire d’ Informatique et de Mécanique pour les Sciences de I’Ingénieur)Orsay CedexFrance
  2. 2.ENST (Ecole Nationale Supérieure des Télécommunications)Paris Cedex 13France

Personalised recommendations