In the first study of this special issue, Giritlioglu et al. introduce a new audio-visual dataset called SIAP Self-presentation and Induced Behavior Archive for Personality Analysis and use deep learning methods to analyze Big Five personality traits. 60 subjects present themselves in an interview-like setting, and additional videos are collected from induced behaviors. Different modalities are tested and evaluated using both the SIAP and ChaLearn LAP First Impressions datasets, while the impact of the induced behavior for personality analysis is thoroughly investigated.

In another deep learning-based study, PLAAN (Pain Level Assessment with Anomaly-detection based Network), Li, Gosh, and Joshi aim to overcome the limitations stemming from the missing expert-labeled data in pain estimation by using long short term memory - dynamic neural network (LSTM-DNN) models. The EmoPain Challenge dataset is used to evaluate pain estimation outcomes and protective behavior estimation tasks from body movements during the study. While the pain estimation performance increases with LSTM-DNN networks, the protective behavior results (i.e., hesitation, guarding, stiffness, bracing, support) require further refinement with anomaly detection. Anomaly detection improves the outcomes significantly, and the study provides a detailed comparative analysis of different networks for the pain estimation task.

The third paper of the issue proposes board games as a fertile ground for studying social signals and group interactions, and contributes a new dataset to the literature, called MUMBAI, collected from four-person interactions during board games. Schimmel et al. have annotated the dataset with emotional moments, as well as personality and game experience questionnaires filled by participating subjects. The dataset and the accompanying code for affective evaluation baselines are made publicly available for researchers.

In another game study, Surer et al. focus on the usability of a scenario-based game generator framework created by game developers for the Chemical, Biological, Radiological, Nuclear, and Explosives (CBRNe) domain. This comparative study paves the way for comparison in personal computer (PC) environments and virtual reality (VR) environments. The scenarios of the PC- and VR-based games originated from several joint exercises of the EU H2020-funded European Network Of CBRN TraIning CEnters (eNOTICE) project, conducted physically in Nimes (France) and Brussels (Belgium) in 2018. The results show that the scenario-based game generator has a vast potential to automatize the structured game generation processes.

Game-based studies do not focus only on interaction within human players, but also with robots and other embodied agents. Ince et al. developed a drum-playing game for multimodal human-robot interaction using audio-visual cues. The player can play drums with a humanoid robot in different scenarios, where several parameters such as presence and embodiment could be evaluated. The proposed system has been tested with human participants via both objective and subjective measurements. This study aims to bring the lessons learned further to design robots and avatar assistants for children with special needs.

Taken together, these studies use state of the art approaches to human behavior analysis and apply them in innovative domains, illustrating the potential of multimodal analysis in a wide range of applications, and contributing several new databases and tools for the research community in the process.

We want to thank all the authors who submitted manuscripts for this special issue, as well as our reviewers, for their valuable suggestions and rigorous evaluations of the papers.