Cognitive processes, such as inference, categorization, and memory, are not independent from their physical instantiations. Individuals’ choices, perception, and actions emerge and are dynamically affected and enhanced by the interaction between sensory-motor systems and the inhabited environment that includes the organizational, cultural, and physical context). This interplay brings up instantiations of cognitive behavioural systems.

The above considerations led to the idea to dedicate a special issue of Cognitive Computation to this topic, with the aims to initiate a discussion on what has been achieved to-date and look at future perspectives.

The volume originated in the European Science Foundation (ESF) Research Network COST Action 2102: Cross Modal Analysis of Verbal and Nonverbal Communication (www.cost.esf.org/domains_actions/ict/Actions/2102) funded by the European Cooperation in Science and Technology (COST, www.cost.esf.org). COST is an European instrument funded by the ESF supporting cooperation among scientists and researchers across Europe, allowing the coordination of nationally-funded research on a European level, increasing the mobility of Early Stage Researchers and fostering the establishment of scientific excellence in nine key domains, including Information Communication Technology (ICT)—of which COST Action 2102 is part of.

The main objective of COST 2102 was to develop an advanced acoustical, perceptual and psychological analysis of verbal and nonverbal communication signals originating in spontaneous body-to-body interaction, in order to identify algorithms and automatic procedures capable of automatically perceiving and distinguishing human emotional states. Researchers involved in COST 2102 are convinced emotions drive our daily life and that research in Human-Machine Interaction cannot continue to neglect this aspect of human behaviour in developing future communication systems. Several key aspects were considered, such as the integration of the developed algorithms and procedures for application in telecommunications, and for the recognition of emotional states, gestures, speech and facial expressions—in anticipation of the implementation of intelligent avatars and interactive dialogue systems that could be exploited to improve user access to future telecommunication services.

The Action profited from two former COST Actions (COST 277 and COST 278) that identified new appropriate mathematical models and algorithms to drive the implementation of the next generation of telecommunication services such as remote health monitoring systems, interactive dialogue systems, and intelligent avatars.

During its 4 year lifespan, COST 2102 initiated its research work programme with the intent to underline and exploit the amount of information conveyed by multimodal signals. It then moved on to address actions that are derived from them, on how these actions are organized in a realistic and socially believable context, on how to combine and build up knowledge through verbal and non verbal signals enacted in an environment and in a context, and on the processing issues—which are also an important consideration owing to the fact that since this research approach is computationally complex, the amount of data to be treated may thus be considered algorithmically infeasible. Therefore algorithms were developed for gaining enactive knowledge through data processing that accounted for natural and intuitive approaches—based more on heuristics and experiences rather than on symbols—as well as on the discovery of new processing possibilities for data analysis, coordination of the data flow through synchronization, and temporal organization and optimization of the extracted features.

Finally, towards the end of its running period, COST 2102 focused on Cognitive Behavioural Systems and to this end, an International Conference was held in Dresden, from the 21st to the 26th of February 2011 (www.ias.et.tu-dresden.de/ias/index.php?id=503), jointly sponsored by the 2nd European Network for the Advancement of Artificial Cognitive Systems,

Interaction and Robotics (EUCogII, www.eucognition.com), the European Network of Excellence on Social Signal Processing (SSPnet, www.sspnet.org), and by COST2102. The objectives of this international conference were to discover more natural and intuitive approaches for modelling, and uncovering the wealth of information conveyed by humans during interaction for developing realistic and socially believable agents, and developing new cognitive systems and models of cognitive processes. The fundamental questions that were addressed included:

  • How human factors and behavioural patterns can be modelled in order to bring machine intelligence close to human expectations?

  • Do we already have a computational paradigm or do we need to learn more from signals and data?

  • How much trustful, credible and satisfactory an emotionally coloured multimodal systems will appear to the end user?

  • What role will the physical instantiation and the appearance play in the interaction?

The above topics were discussed at the Dresden conference and established the basis for a new research network (that is planned to be implemented by a new follow-on COST Action proposal), and are also in part, covered in this volume through the contributions of several authors.

The contributions in this volume cover different topics that are all of importance in the context of modelling and implementation of Cognitive Behavioural Systems. In particular, the papers by Sadeghipour & Kopp on embodied gesture processing, by Bourbakis et al. on meta-features for understanding people’s emotional behaviour, by Kröger et al. on movements and holds in ASL, and by Poggi et al. on multimodal aspects of agreements cover characteristics related to the analysis, modeling and processing of interactional behavioural features. Finally the papers by Grassi et al. on sentic web, and by Loizou & Laouris on machine learning technologies are examples of applications in the field.

Clearly, not all the research facets of Cognitive Behavioural Systems have been discussed in this special issue, but we hope to trigger a debate in this emerging field and raise the interest of the readers of Cognitive Computation.

Finally, we are deeply indebted to the contributors for making this special issue a scientifically stimulating compilation of new and original ideas and to the anonymous referees for their rigorous and invaluable scientific reviews.