Non-determinism in the Uptake of Gestural Information

Abstract

It is well established that gestures and speech form an integrated system of communication; gestures that match the meaning of the speech they accompany favor the listener’s discourse comprehension, whereas mismatching gestures whose meaning conveys information contradicting that conveyed by speech, impair comprehension. A less investigated issue is whether or not the uptake of gestural information is a deterministic process. In line with recent studies in the literature, we purport that the process may be modulated by certain factors. In particular, we investigate the role of unrelated gestures whose meaning, which is irrelevant to the speech they accompany, could be neglected. The results of four experiments led us to conclude that unrelated gestures are not processed, and that the uptake of gestural information is a non-deterministic process.

Appendices

Appendix 1

Appendix 1A: The Funfair Discourse Used in Experiments 1 and 2 (Semantic Units are Separated by Slashes)

It was there, at the funfair, it was there that I found her,/ and it was at the funfair that I lost her./ It was a vast funfair./ A funfair with shooting-ranges and candy floss stalls/ and Japanese bagatelle tables, stalls with bottles of champagne/ and showmen’s booths and roundabouts./ And the roundabouts turned and creaked/ and the candy floss scented the air/ and the rifles shot./ I was shooting at the target./ I can shoot at the target very well and I am proud of it./ No, wait a moment, I am wrong!/ I did not meet her at the shooting-range./ I met her at the candy floss stall. Yes, it was at the candy floss stall that I found her./ The candy floss scented the air,/ and she was eating it/ and she blew on her candy/ and I was all covered with white powder./ She started laughing/ and I asked her: “What’s your name?”/ And she shouted to me: “I’ll tell you later”./

Appendix 1B: Examples of Co-speech Gestures Produced by the Animated Agent in Both the Gesture Condition and the Unrelated Condition (Experiments 1 and 2—Funfair Discourse)

The left and the central columns present the co-occurrence of speech and gestures in the gesture condition, whereas the right and the central columns present the co-occurrence of speech and gestures in the unrelated conditions. Please note that the beginning of each condition is highlighted in bold.

Underscoring of semantic units indicates the duration of the gesture with the corresponding number.

Semantic units gesture condition	Computer animated agent’s gestures	Semantic units unrelated condition
BEGINNING OF GESTURE CONDITION FICTION
It was there, at the funfair, it was there that I found her,	Hands still, resting on thighs
and it was at the funfair that I lost her	Hands still, resting on thighs
It was a vast funfair 1	(1) Raises both hands simultaneously to a position at the level of the navel, palms down From here, the hands are raised to head height in a continuous movement, curving outwards as if describing a circle, with palms facing forward towards the hypothetical interlocutor. Both hands return to chest level	The candy floss scented the air,
		and she was eating it, and she blew on her candy 1
A funfair with shooting-ranges 2 and candy floss stalls,3	(2) The right hand closes into a fist, and the left hand remains open with palm facing to the right; then both hands open and assume a position as if to point to the agent’s left; the left hand is slightly higher than the right; the left arm is almost fully straightened, while the right one is more bent. Then both hands move down to the thighs (3) The right hand is rapidly raised to stomach level, outside the line of the body; the hand is half open as if to indicate a direction; it is then quickly moved back to rest on the thigh	And I was all covered with white powder 2
		She started laughing 3
and Japanese bagatelle tables 4 stalls with bottles of champagne, 5	(4) The right hand is raised from the right leg and moves to one side of the body (further from the body than in the previous movement), almost to stomach height. The hand is half open, with the fingers towards a hypothetical interlocutor, as if to indicate a direction; the arm is partly bent	and I asked her: “What’s your name?” 4
	(5) Just before the right hand comes to rest on the leg, the left hand is raised from the other leg and moves forward; the thumb and index finger are extended and pointing, while the other fingers are partly bent and the palm is almost hidden by the fingers. The hand then returns directly to the left leg	And she shouted to me, “I’ll5 tell you later”
and showmen’s booths 6 and roundabouts 7	(6) Both hands are raised simultaneously from the legs. The right hand is open with the palm towards the right leg, away from the body, with the thumb pointed upwards; the left hand is open and moves away from the body, with the palm still facing the thigh, and the index finger and thumb straight. Both hands then move back to the thighs (7) The left hand is raised immediately from the left leg and moved forward and to the right, showing the palm with fingers open, to chest height (the elbow is bent at 90°), and remains in that position for 2 s; the right hand is open and remains still at hip height, with the palm towards the thigh	BEGINNING OF UNRELATED CONDITION FICTION
		( The actor is silent ) 6 It was there, at the funfair7
And the roundabouts turned and creaked, 8	(8) Both hands start to move from left to right, held forwards at chest height; the right hand continues the movement gradually downwards to hip level. Then both hands are moved back towards the legs but do not come to rest on them. The left hand, open and palm facing downwards, slides downwards and outwards from the left leg, with the arm extended. In a continuous movement, the hand moves back upwards in a circular fashion and comes back to rest on the left thigh	it was there 8 that I found her,
and the candy floss scented the air, 9	(9) Both hands are moved away from the legs up to the mouth area, in a cupped position 2–3 cm apart with the palms towards the mouth and fingers slightly bent, moving rapidly backwards and forwards for around 2 s	and it was at the funfair that I lost her 9
and the rifles shot 10	(10) Then the open right hand moves to the chest area, while the left hand moves down to hip level. The left hand, open and palm upwards, is raised in a circular movement, and then from the hip area it is moved up to the chest, near to the right hand	It was a vast funfair 10
I was shooting at the target 11	(11) Both hands are at chest height, with palms facing each other; both hands are moved towards the left. The left hand moves away from the body, while the right remains at stomach level. The ring and little fingers of the left hand close, while the other digits are straight; at the same time, the middle, ring and little fingers of the right hand close, while the thumb and index finger remain straight and extended	A funfair 11 with shooting-ranges 12 and candy floss stalls, 13
I can shoot at the target very well 12 and I am proud of it 13	(12) From that position, the hands are raised momentarily to shoulder height, with index fingers pointing upwards, and then immediately lowered to their initial position. The right hand is open, with the palm towards the stomach and the arm bent; the left hand is at the front, with thumb and index finger straight, while the other fingers and the right arm are slightly bent (13) The hands are brought up to cover the face for 2 s; the hands are open, with the palms towards the face
No, no,14 wait a moment15 …	(14) The hands remain in this position for 2 s and then slowly descend to hip level. When the hands reach stomach level, the right hand moves towards the left, while the left hand opens and moves forwards, with the fingers almost fully extended. Then the hands move directly towards the thighs, but do not actually come to rest on them (15) The left hand is raised quickly up to the top of the chest, with the palm towards the right and fingers open. The right hand is raised slightly, up to hip level, with the palm still facing downwards. The fingers are semi-extended. Both hands move rhythmically at around hip height and are brought slightly outwards	and Japanese bagatelle tables 14 stalls with bottles of champagne, 15
I am wrong!	The hands stay still near the legs for 7 s	and showmen’s booths and roundabouts

Appendix 1C: Examples of Sentences Used for the Recognition Task of Experiment 1

Literal :

I can shoot at the target very well

Paraphrases :

I am really able to shoot at the target

Wrong :

I do not know how to shoot at the target

Literal :

I shot, the egg popped up. I turned aside, she wasn’t there any more

Paraphrases :

When I shot at the egg she disappeared

Wrong :

While I was going to shoot to the egg, I turned aside and she wasn’t there any more

Appendix 2

Appendix 2A: The Color Discourse Used in Experiments 3 and 4 (Semantic Units are Separated by Slashes)

It’s beyond dispute/ that colors carry strong expressive components./ Some attempts have been made to describe the specific expressive characters of the various colors/ and to draw some general conclusions from the symbolic use the different cultures have made of them./ There is a very widespread belief that the expression of colors is based on association./ Therefore, red should be considered exciting/ because it reminds us of the connotations of fire, blood and revolution./ Green evokes the restorative thought of nature,/ and blue is refreshing like water./ The theory of association, however, is not more interesting or prolific in this field than in others./ In addition, the effects of colors are too direct and spontaneous/ to be simply the result of an interpretation given through knowledge./ On the other hand, no hypothesis has been advanced so far on the kind of physiological process/ that could help to explain the influence of colors on the organism./ The need to discuss the form makes us feel on more solid ground, though,/ as we can compare the expression of specific patterns/with that or more general properties/ such as spatial orientation, balance or the geometrical characteristics of the outlines.

Appendix 2B: Examples of Co-speech Gestures Produced by the Animated Agent in Both the Gesture Condition and the Unrelated Condition (Experiments 3 and 4—Color Discourse)

The left and the central columns present the co-occurrence of speech and gestures in the gesture condition, whereas the right and the central columns present the co-occurrence of speech and gestures in the unrelated conditions. Please note that the beginning of each condition is highlighted in bold. Underscoring of semantic units indicates the duration of the gesture with the corresponding number.

NB: if the numbering of the gestures corresponding to the semantic units is not perfectly sequential, this is likely to be due to changes in the sentence structure in the translation from the Italian. We have therefore preferred to maintain the speech-gesture correspondence rather than the sequential gesture numbering.

Semantic units gesture condition	Computer animated agent’s gestures	Semantic units unrelated condition
BEGINNING OF GESTURE CONDITION FICTION
It’s beyond dispute that colors carry strong expressive components 1	(1) The agent’s hands are resting on his legs at knee height. He raises both hands simultaneously to the level of the navel. From here, the hands open outwards with palms facing slightly forwards, towards the hypothetical interlocutor. The hands are then brought swiftly back to the thighs	There is a very widespread belief that the expression of colors is based on association 1
Some attempts have been made to describe the specific expressive characteristics of the various colors 2	(2) The left hand is raised from the left leg up to the mouth area, half open and with the palm towards the right. The hand moves down to chest level in a continuous movement and then assumes an up-down undulating motion. Initially, the fingers are all partly closed, then the index finger gradually extends almost completely; the palm of the hand faces towards the interlocutor (first at mouth level and then at chest height) At the same time, the open right hand is raised from the right leg to a position at the level of the navel, with the palm facing to the left and fingers half extended	Therefore, red should be considered exciting, 2
and to draw some general conclusions from the symbolic use the different cultures have made of them. 3	(3) From the chest, both hands move down onto the legs and closer to each other (to about 2–3 cm apart), with palms facing each other, fingers extended and thumbs pointing upwards. The right hand is raised from the right leg up to chest level, with fingers open, while the left hand rises at the same time in line with the right hand, first with fingers half closed and then extended. The right hand is then brought back onto the right leg, with palm facing to the left and thumb raised, while the left hand moves in small circles, with fingers semi-extended. Finally, the left hand moves back onto the left leg, with the palm facing to the right and fingers extended	because it reminds us of the connotations of fire, blood and revolution 3
There is a very widespread6 belief4 that the expression of colors is based on association5	(4) The left hand is raised from the left leg up to neck height, with the index finger pointing upwards and the other fingers half closed, whereas the right hand remains open at hip level, with the palm facing to the left (5) The left hand moves down from the mouth to shoulder height, then in an up-down waving motion, first with the hand open and then with fingers half closed, with the palm facing downwards and to the right (6) The right hand is raised from the right thigh up to the same height as the left hand; both palms are parallel to the thighs. The hands move forwards, and both palms turn towards the agent; the arms remain partly bent	Green 4 evokes the restorative thought of nature 5,
		and blue 6 is refreshing like water 7
Therefore, red 7 should be considered exciting, 8	7) Both hands are brought up to chest height, first closer together then further apart again (8) With both hands open at chest level, the right hand moves closer to then further from the left hand
because it reminds us of the connotations of fire 8, blood 9 and revolution 10	(8) From the chest, both hands are moved to the shoulder area, with palms facing each other and fingers extended. They are then quickly lowered to waist level, with fingers closed. The fingers immediately reopen and the right hand is raised up to the right shoulder; the fingers open and the index finger points upwards above the shoulder, with the palm turned slight to the left. The left hand stops at chest height, facing forwards with the palm open (9) The left hand moves up and down repeatedly, while the right hand remains at shoulder level (10) Both hands are moved to chest level; the fingers are open and the hands extended forwards with palms facing the interlocutor. The index fingers appear to point upwards	The theory of association, however, 8 is not more interesting or prolific 9 in this field 10 than in others
Green 11 evokes the restorative thought of nature 12,	(11) The left hand is lowered slightly and turned, in an open position, so that the palm faces upward, with the thumb and index finger more extended than the other digits. It then turns again, so as to finish with the palm facing the right hand (12) Both hands are turned towards the agent and slightly folded (i.e., with the fingers of the right hand covering those of the left). Both hands then move outwards, with a turn of the arms, until the right hand is outside the line of the body, while the left hand remains slightly closer to the body. The hands return to chest level	In addition,11 the effects of colors are too direct and 12 spontaneous
and blue 13 is refreshing like water 14	(13) The right hand moves down from the chest onto the right thigh, while the left hand is raised to shoulder level (14) The left hand is moved repeatedly up and down, moving the fingers at the same time	to be simply the result 13 of an interpretation given 14 through knowledge
The theory of association, however, is not more interesting 15 or prolific 16 in this field than in others	(15) The left hand moves down from the shoulder to the chest and the palm is turned towards the interlocutor, with fingers half closed. Simultaneously, the right hand is raised from the right leg up to the same height as the left hand, with the palm facing towards it (16) Both hands close and are then moved from the chest to rest on the legs	BEGINNING OF UNRELATED CONDITION FICTION It’s beyond dispute 15 that colors carry strong expressive components 16
In addition 17, the effects of colors are too direct and spontaneous 18	(17) The hands are raised to hip level, with palms facing each other and fingers open (18) The right hand remains at hip level, open and with the palm facing left, and moves outwards away from the body. The left hand is raised to chest height, outside the line of the body, and is pushed quickly forwards, with fingers open, and then pulled back equally quickly towards the chest (19) The right hand is raised from the right leg by about 5 cm, with the palm towards the leg, while the left hand remains at chest level	Some attempts have been made 17 to describe the specific 18 expressive characteristics of the various colors 19
to be simply the result 19 of an interpretation given through knowledge 20	(20) The right hand moves to around stomach level, away from the body; the hand is open with the index finger almost fully extended, the other fingers half closed, and the palm facing left. It is then brought back to rest on the leg, closing almost into a fist. At the same time, the left hand also moves slightly towards the side of the body, then returns to its previous position and then moves towards the side again	and to draw some general conclusions from the symbolic use the different cultures have made 20 of them

Appendix 2C: Examples of Sentences Used for the Recognition Task of Experiment 3

Literal :

Green evokes the restorative thought of nature

Paraphrases :

The color evoking the restorative thought of nature is green

Wrong :

Green evokes the sense of tiredness of nature

Literal :

Blue is refreshing like water

Paraphrases :

Blue color gives a feeling of freshness like water

Wrong :

Blue color is fresh like sea