Emotion-Oriented Systems

Part of the series Cognitive Technologies pp 349-367


Coordinating the Generation of Signs in Multiple Modalities in an Affective Agent

  • Jean-Claude MartinAffiliated withComputer Sciences Laboratory for Mechanics and Engineering Sciences (LIMSI) Email author 
  • , Laurence DevillersAffiliated withLIMSI-CNRS
  • , Amaryllis RaouzaiouAffiliated withNational Technical University of Athens
  • , George CaridakisAffiliated withImage, Video and Multimedia Systems Lab, National Technical University of Athens
  • , Zsófia RuttkayAffiliated withUniversity of Twente
  • , Catherine PelachaudAffiliated withCNRS-LTCI, TELECOM ParisTech
  • , Maurizio ManciniAffiliated withInfoMus Lab, Universitá di Genova
  • , Radek NiewiadomskiAffiliated withTelecom ParisTech
  • , Hannes PirkerAffiliated withAustrian Research Institute for Artificial Intelligence
    • , Brigitte KrennAffiliated withAustrian Research Institute for Artificial Intelligence
    • , Isabella PoggiAffiliated withUniversity of Rome 3
    • , Emanuela Magno CaldognettoAffiliated withInstitute of Cognitive Sciences and Technologies
    • , Federica CavicchioAffiliated withCIMeC Università di Trento
    • , Giorgio MerolaAffiliated withUniversità Roma Tre
    • , Alejandra García RojasAffiliated withEcole Polytechnique Fédérale de Lausanne
    • , Frédéric VexoAffiliated withEcole Polytechnique Fédérale de Lausanne
    • , Daniel ThalmannAffiliated withEcole Polytechnique Fédérale de Lausanne
    • , Arjan EggesAffiliated withUniversiteit Utrecht
    • , Nadia Magnenat-ThalmannAffiliated withUniversity of Geneva

* Final gross prices may vary according to local VAT.

Get Access


In order to be believable, embodied conversational agents (ECAs) must show expression of emotions in a consistent and natural looking way across modalities. The ECA has to be able to display coordinated signs of emotion during realistic emotional behaviour. Such a capability requires one to study and represent emotions and coordination of modalities during non-basic realistic human behaviour, to define languages for representing such behaviours to be displayed by the ECA, to have access to mono-modal representations such as gesture repositories. This chapter is concerned about coordinating the generation of signs in multiple modalities in such an affective agent. Designers of an affective agent need to know how it should coordinate its facial expression, speech, gestures and other modalities in view of showing emotion. This synchronisation of modalities is a main feature of emotions.