Seniors’ Sensing of Agents’ Personality from Facial Expressions

Several research studies have shown that emotions and facial emotional expressions play an important role in everyday life [7, 11, 12]. Indeed, there is a wide field of research dealing with questions concerning emotional facial expressions as a signal that mediates social interaction. Interactional exchanges constitute the main source of emotional appraisals and emotions are indispensable for starting, maintaining, modifying, strengthening or breaking relationships with others [2]. In this context, facial emotional expressions exemplify the visible component of emotions, providing information on the emotional states of the interacting person. To this aim, facial emotional expressions have been subjected to numerous studies and over time they have become the pivotal topic of many researches, some of which focused on the visual appearance of virtual agents in order to investigate the types of reactions such may trigger in individuals. The Semaine project¹, for example, aimed at the implementation of artificial conversational agents allowing for “emotionally colored interactions between a person and a machine”. To achieve this, the agents had to exploit mostly non-verbal facial expression behaviors in order to elicit similar behavior in human interlocutors [1]. The Empathic project² attempts to expand this work by “developing causal models of [agent] coach-user interactional exchanges that engage elders in emotionally believable interactions […]”. Doing so, Empathic is age contextualized and will account for social and cultural variations in Western Europe, collecting data in Spain, France, Norway, and Italy. It will follow a user-centered design approach so as to establish trusting human-machine relationships, take appropriate actions to provide long-term care, and detect emotional states and negative moods, such as depression. To achieve this, we will build upon previous work [e.g. 4, 5, 6], yet aim at developing solutions which are more efficient and more effective (e.g. [3]).

A first pilot study focused on eliciting initial guidelines for designing such user-centered and adaptable interfaces. The study explored user’s requirements and expectations with respect to their age and familiarity with technology. The goal was to (1) assess seniors’ preferences in initiating conversations with an agent visually expressing emotional behavior; (2) summon elder’s emotional responses arising from such non-verbal agent behavior; and (3) measure their interest in these interactions.

The research sample consisted of 45 healthy seniors (20 females), aged 65+ years (AVG = 70.28 years; SD = ±5.52). They were recruited in Campania, a region in the south of Italy. Participation was voluntary and approved by the ethical committee of the Dept. of Psych. at Univ. della Campania Luigi Vanvitelli (Code No. 25/2017).

We used a tripartite questionnaire to collect user feedback. Section 1 focused on participants’ socio-demographic information, their degree of technology experience, and their used technological devices with respective ratings on perceived usage difficulty. Section 2 established users’ preferences regarding the interaction with each of the proposed agents. This section was clustered in four subsections, each consisting of 7 items, investigating the practicality (PQ), pleasure feelings (HQI, and HQS), and attractiveness (ATT) experienced while watching the video-clips [8]. Finally, Sect. 3 collected feedback on 10 agent characteristics rated on 5-point Likert scales ranging from 1 = strongly agree to 5 = strongly disagree (note: 3 = I don’t know). Section 1 was completed first. Then participants watched the 4 videos, each time followed by a separate Sect. 2, before they eventually completed Sect. 3.

Our results show that, 57.8% of the participants regularly use at least one of the proposed technological devices (i.e. smartphone, tablet, laptop), 26.7% use one from time to time, 2.2% had attempted to use one, and 13.3% never used one. The most popular device seems to be the smartphone (68.9% regular usage), followed by the tablet (22.2% regular usage) and laptop (17.8% regular usage). The smartphone appears to be also the device with the highest usability, rated as easy to use by 75.6% of the participants. Asked about their acceptance for the shown avatars, 71.1% liked Francesca, Serena was accepted by 66.7% and Gerardo by 46.7%. Pasquale was the least favored, accepted by only 22.2% of the participants.

Analyzing the Pragmatic Qualities (PQ), Hedonic Qualities of Identification (HQI) and Stimulation (HQS), and the attractiveness (ATT) of the agents, a 2(gender) × 4(PQ, HQS, HQI, ATT) ANOVA was applied (α = .01). No interactions were found between gender and avatars. Significant differences were, however, found among avatars for PQ (F(1, 43) = 40.04, p < .001), HQI (F(1, 43) = 42.32, p < .001), HQS (F(1, 43) = 47.84, p < .001), and ATT (F(1, 43) = 47.51, p < .001). Bonferroni post hoc tests revealed that these significant differences were always between Pasquale and Francesca (with p < .001 for PQ, HQI, HQS, and ATT), between Pasquale and Serena (with p < .001, for PQ, HQI, HQS, and ATT), between Gerardo and Francesca (with p < .001, for PQ, HQI, HQS, and ATT), and between Gerardo and Serena (with p < .001 for PQ, and ATT, and p = .002 for HQI). No significant HQS differences were found between Gerardo and Pasquale, Francesca and Serena, and Gerardo and Serena.

To summarize the preference scores obtained in each cluster of Sect. 2 (PQ, HQI, HQS, ATT), we classified them in Strong/High (7–13 points), Moderate (14-20 points), Mild (21–27 points), and None (28–35 points). According to this arrangement, 64.4% of the participants considered Francesca moderately pragmatic (PQ) (Serena = 53.3%, Gerardo = 42.2%, Pasquale = 24.5%). They also found her to be moderately mind-pleasing (HQI) (66.5%; Serena = 51.1%, Gerardo = 35.6%, Pasquale = 22.2%), moderately stimulating (HQS) (75.5%; Serena = 66.6%, Gerardo = 40.0%, Pasquale = 20.0%), and moderately attractive (ATT) (64.5%; Serena = 55.6%, Gerardo = 44.4%, Pasquale = 31.1%).

Our study showed that a significant number of seniors use at least one technological device, where the smartphone seems to be preferred and perceived as most usable. Furthermore, it was shown that generally seniors show a positive attitude towards interactions with an artificial avatar, and that they favor avatars which express a positive personality. That is, regarding pragmatic, hedonic, and attractiveness features our avatars Francesca and Serena did consistently score higher than Pasquale and Gerardo. Although participants were not informed about an avatar’s personality, they were able to perceive relevant facial expressions, suggesting that they have preferences for positive facial dynamics. Future work requires a deeper investigation of this capability as those avatars showing positive facial dynamics were females, indicating a potential gender influence on the processing of emotional facial expressions [9]. Also, the effect of voice has to be explored, as for now the utilized avatars were moving their lips yet audio output was deliberately muted.