Multimodal Integration of Dynamic Audio–Visual Cues in the Communication of Agreement and Disagreement

Abstract

Recent research has stressed the importance of using multimodal and dynamic features to investigate the role of nonverbal behavior in social perception. This paper examines the influence of low-level visual and auditory cues on the communication of agreement, disagreement, and the ability to convince others. In addition, we investigate whether the judgment of these attitudes depends on ratings of socio-emotional dimensions such as dominance, arousal, and valence. The material we used consisted of audio–video excerpts that represent statements of agreement and disagreement, as well as neutral utterances taken from political discussions. Each excerpt was rated on a number of dimensions: agreement, disagreement, dominance, valence, arousal, and convincing power in three rating conditions: audio-only, video-only, and audio–video. We extracted low-level dynamic visual features using optical flow. Auditory features consisted of pitch measurements, vocal intensity, and articulation rate. Results show that judges were able to distinguish statements of disagreement from agreement and neutral utterances on the basis of nonverbal cues alone, in particular, when both auditory and visual information were present. Visual features were more influential when presented along with auditory features. Perceivers mainly used changes in pitch and the maximum speed of vertical movements to infer agreement and disagreement, and targets appeared more convincing when they showed consistent and rapid movements on the vertical plane. The effect of nonverbal features on ratings of agreement and disagreement was completely mediated by ratings of dominance, valence, and arousal, indicating that the impact of low-level audio–visual features on the perception of agreement and disagreement depends on the perception of fundamental socio-emotional dimensions.

Appendices

Appendix 1: Definitions of Socio-Emotional Dimensions

Participants were given some time prior to the study to become familiar with the definitions of the dimensions under study. They also had the definitions along with them in case they needed it during the rating session. Definitions:

• Agreement: an attitude that reflects harmony or accordance in opinion or feeling.

• Disagreement: an attitude that reflects a lack of consensus or approval about an opinion or feeling.

• Dominance: a disposition or a tendency to exert influence and control over other individuals and life events in general.

• Submissiveness: a disposition or a tendency to be influenced and controlled by other individuals or by events.

• Positive–Negative: refers to the pleasant or unpleasant character of the attitude expressed in the excerpt.

• Emotional arousal: refers to the degree of physical and physiological activation associated with the emotional state of the individual.

• Convincing power: refers to the extent to which the statement can provoke a change of opinion, feeling, or attitude in an interlocutor⁶.

Appendix 2: Method for the Estimation of Optical Flow

Optical flow estimation techniques rely on the fact that the local image appearance around a particular location does not change drastically between two consecutive frames. The local image appearance is generally described by the pixel intensity values around the point under consideration or by the image gradient around this point. Most optical flow estimation techniques find the optical flow by minimizing a function that comprises two terms: (1) a term that measures how well the second image is reconstructed by transforming the first image using the flow field and (2) a term that promotes the selection of “simple” optical flow fields. In particular, the functions generally have the form of a minimization over the flow field U (Horn and Schunck 1981):

$${ \hbox{min} }D\,(I_{0} , \, I_{1} , \, U) +\varvec{\lambda}R(U).$$

Herein, D (I ₀, I ₁, U) is an error function that measures the difference between image I ₁ and image I ₀ transformed by flow field U; for instance, a sum of squared errors may be employed. The function R (U) is a regularizer that promotes smoothness of the optical flow field U, i.e. that penalizes solutions in which neighbouring locations have drastically different flow estimates. The scalar λ is a free parameter that weighs the regularizer against the error function.

In this study, we employed an optical flow estimation method that uses a sum of absolute differences as error function (Werlberger et al. 2010):

$$D\,\left( {I_{0} , \, I_{1} , \, U} \right) = \iint \, |I_{0} \left( {x,y} \right) - I_{1} \left( {U_{x} \left( {x, \, y} \right), \, U_{y} \left( {x, \, y} \right)} \right)|dxdy,$$

where the integrals are summing the errors over the entire image. In the equation, the optical flow at location (x, y) is represented by the x-component U _x of the optical flow vector and by the y-component U _y of the optical flow vector. This decomposition of the optical flow vector is illustrated in the panel B of Fig. 2.

The regularizer is implemented by a function that measures the (weighted) total variation of the optical flow field:

$$R\left( U \right) = {\iint \iint }W\left( {x, \, y, \, x^{\prime }, \, y^{\prime }} \right)\left[ {|U_{x} \left( {x, \, y} \right) - U_{x} \left( {x^{\prime }, \, y^{\prime }} \right)\left| {_{\epsilon }} + \right|U_{y} \left( {x^{\prime }, \, y^{\prime }} \right) - U_{y} \left( {x^{\prime }, \, y^{\prime }} \right)|_{\epsilon } } \right]dx \, dy \, dx^{\prime } \, dy^{\prime },$$

where again, the integrals are over the entire image. In the above equation, $\epsilon$ is a small constant and |q|$\epsilon$ denotes the so-called Huber loss (Hube 1964):

$$\left| q \right|_{\smallint }\,= q^{2} /2\smallint {\text{if }}\left| q \right| \le \smallint ;\left| q \right| - \smallint /2{\text{ otherwise}} .$$

In the above equation, the weights W(x, y, x′, y′) are computed based on the spatial distance between points (x, y) and (x′, y′) and based on the pixel intensity differences between those points. Specifically, the weights are large for nearby points with a similar color, but small for points that are far away or that have a dissimilar color (much like an anisotropic filter). As a result, the regularizer promotes optical flow fields that are locally smooth, but smoothness is not promoted across edges in the image (as edges suggest a different object with a potentially different optical flow).

Hence, the advantage of the optical flow estimator we used is that it promotes smoothness, whilst allowing for discontinuities in the optical flow field near edges in the image. The optical flow estimator outlined above is presently one of the best-performing algorithms on a widely used benchmark data set for evaluation of optical flow algorithms (Werlberger et al. 2010; Baker et al. 2011).