Expressivity in Interactive Speech Synthesis; Some Paralinguistic and Nonlinguistic Issues of Speech Prosody for Conversational Dialogue Systems

Abstract

This chapter explores the role of prosody in expressive speech synthesis and goes beyond present technology to consider the interrelated multimodal aspects of interactive spoken dialogue systems for human–machine or human–human interaction. The chapter stresses that social aspects of spoken dialogue are now ripe to be considered in the design of interactive systems and shows how three modalities can be combined—utterance content, speech expressivity, and facial or bodily gestures—to express social factors and manage the interaction. Linguistic prosody has been well described in the literature but the social aspects of managing a spoken dialogue remain as a next-step for speech synthesis research. This chapter shows how voice quality functions socially as well as linguistically, and describes an application of speech synthesis in a robot dialogue system that makes complementary use of visual information and peaking-style variation.

Appendix

Full text of the Herme utterances as used in the Science Gallery experiment:

Each line represents a new utterance to be synthesised separately, with a variablelength pause, and some groups of lines (starting with a dash) form a mini subdialogue section—we experimented by varying the timing of the pauses, i.e. utterance onsets, according to the nature of each response from the visitors to the exhibition—Herme’s chat partners. Where possible, the content of any response was ignored as the wizard’s task was simply to trigger the next utterance and the automatic version had no content processing module.

- hello? hi

hello

hi

- my name is hur-mi.

h e r m e

hur-mi

what’s your name?

- how old are you?

really

I’m nearly seven weeks old

- do you have an i d number

i need an i d number to talk to you

i d numbers are on your right

thank you

- are you from dublin?

really

I’m from the Speech Communication Lab here in Tee See Dee

- tell me about you

really

owe

- tell me something else

owe

really

- why are you here today?

really

why

- do you like the exhibition

really

why?

i like your hair

- do you know any good jokes?

tell me a funny joke

ha ha haha ha

- tell me a knock knock joke

who’s there

who?

who

ha ha haha ha

- I know a joke

what’s yellow and goes through walls

a ghost banana

ha ha hehe he.

ho hoho ho ho

- thanks for your help

goodbye, see you later

goodbye

Note how these utterance chunks typically group into triads (sets of three) and how they maintain the initiative of the conversation throughout. Note also some spelling hacks (e.g., using ‘owe’ to ensure that the synthesiser correctly pronounced ‘oh!’). The repeated use of ‘oh’ and ‘really’ (sometimes with ‘why’) with various punctuation served to keep the conversation interactive and was key to Herme’s supposed conversational abilities.