Embodied meaning: a systemic functional perspective on paralanguage
- 2.9k Downloads
This paper develops a framework for analysing paralanguage, initially inspired by systemic functional linguistic (hereafter SFL) research on early child language development. A distinction is drawn between non-semiotic behaviour (somasis) and meaning (semiosis), and within semiosis between language and paralanguage (using the term paralanguage to refer to semiosis dependent on language and realised through both sound quality and body language, the latter including facial expression, gesture, posture and movement). Within paralanguage a distinction is drawn between sonovergent resources in sync with or in tune with the prosodic phonology of spoken language, and semovergent resources supporting the ideational, interpersonal and textual meaning resources of spoken language’s content plane. The paper closes with a brief discussion of the intermodal relations among language, paralanguage and other modalities of communication.
In this paper we introduce a social semiotic framework for analysing paralanguage. Our approach was inspired by Chris Cléirigh’s contributions1 to New South Wales Youth Justice Conferencing research consolidated in Zappavigna and Martin (2018) and Martin and Zappavigna (2018).2 Cléirigh’s work drew on Matthiessen’s synopses (Matthiessen 2004; Matthiessen 2007; Matthiessen 2009) of Systemic Functional Linguistic (hereafter SFL) research on early child language development. Following Matthiessen (2009) we use the term paralanguage to refer to gestural resources arranged along what McNeill (1992) christened as ‘Kendon’s Continuum’ (gesticulation, pantomime and emblems), along with the vocalisations outlined in van Leeuwen (1999) not usually included in linguistic descriptions of the segmental and prosodic phonology of spoken language (timbre, tempo, tension, pitch range etc.). In this paper however we will consider only gestural systems.
SFL metafunctions (ideational, interpersonal and textual meaning)
Kendon (2004: 158–159)
Enacting social relations
Pragmatic (modal, interactive, performative)
Managing information flow
There are a number of reasons why our SFL interpretation of paralanguage is timely. One has to do with the explosion of SFL inspired work on modalities other than language triggered by Kress and van Leeuwen’s (1996, 2006) monographs, which focused on single static images. As reviewed in Martinec (2005), O’Halloran et al. n.d.(in press) and Taylor (2017) this work has now been extended to the study of diagrams, Powerpoint slides, webpages, comics, picturebooks, moving images, sound and music, architecture, sculpture, toys and behaviour. Since so many texts involve one or more of these modalities, it is advantageous when studying inter-modal relations to be able to draw on descriptions informed by the same theoretical principles. The concept of metafunction introduced above for example allow us to compare like with like as far as convergence and divergence of meaning across modalities is concerned (Painter et al. 2013). Paralanguage is so closely coordinated with spoken language and so regularly implicated in inter-modal texts of several kinds that the utility of a common metalanguage is clear.
Alongside theoretical integration, SFL is particularly well-suited to the study of paralanguage in a number of ways. One is that it provides a linguistically informed model of prosodic phonology (Halliday 1967, 1970; Halliday and Greaves 2008; Smith and Greaves 2015; van Leeuwen 1992; Martinec 2002) which can be used to make explicit the coordination of rhythm and intonation in spoken language with beats and strokes in gesture. This has facilitated Martinec’s development of Kendon’s early work (Kendon 1972) in this area, taking into account later work by Tuite (1993). We will in fact suggest that SFL’s tone group, analysed for rhythm and tone, provides an essential unit of analysis for work on paralanguage as far as questions of synchronicity across modalities are concerned.
Another advantage of SFL is the clear distinction it draws between paradigmatic and syntagmatic relations (system vs structure in SFL terminology). As is well-attested, there is more variation in the language structures realising systemic options than in the underlying systems themselves (Caffarel et al. 2004). This is even more true when comparing the structural realisation of systems from one modality to another. Kendon’s (2004: 186–187) well-known example of the different trajectories of the gestures accompanying “sliced the wolf’s head off” vs “sliced the wolf’s stomach open” illustrates this point. The swinging arm motions are very different structurally from the clause structures in play; but from the perspective of system, the oppositions in meaning are comparable.4 Systematically separating system from structure is crucial when comparing and contrasting modalities.
We also feel that further development of Martinec’s pioneering modelling is timely in light of theoretical and descriptive developments in SFL since his work. This has mainly to do with a clearer articulation of the stratification of language as levels of phonology, lexicogrammar and discourse semantics (e.g. Martin 2010; 2011; 2014; Martin and Rose 2007). Martinec’s work draws largely on Halliday’s lexicogrammatical systems (those proposed in Halliday 1985), the same systems which inspired Kress and van Leeuwen’s (1996) breakthrough. We have found it illuminating to further develop this work by drawing on ideational, interpersonal and textual systems at the level of discourse semantics (ideation, connexion, negotiation, appraisal, identification and periodicity). Work on appraisal (the language of evaluation) in particular (Martin and White 2005) has a number of ramifications for models of paralanguage, especially in relation to the relative marginalisation of these resources in canonical work by Calbris (Calbris 2011), Kendon (Kendon 1997) and McNeill (McNeill 2006).
In this paper we accordingly proceed as follows. In section “Language development (ontogenesis)” we briefly review SFL research on early child language development. We then move on to draw a distinction between behaviour (somasis) and meaning (semiosis), outlining our current framework in sections “Non-semiotic behaviour (somasis)" and "Embodied meaning (semiosis)”. As noted above, for this framework we adopt the term paralanguage to refer to semiosis dependent on language and realised through both sound quality and body language (including facial expression, gesture, posture and movement). We close with a brief discussion of the relations among language, paralanguage and other modalities of communication.
Language development (ontogenesis)
SFL research on language development in young children is a useful starting point for work on paralanguage in two respects. On the one hand, the emergence of the first signs (protolanguage) highlights the issue of what counts as semiosis and what does not. On the other, the realisation of these first signs is multimodal – linguistic and paralinguistic resources are not differentiated at this stage.
Matthiessen (2004, 2007, 2009) reviews the emergence of language and other semiotic systems based on SFL studies of child language development by Halliday (1975, 2003), Painter (1984, 2003) and Torr (1991). These studies show that language develops out of a protolinguistic system in which children draw on sounds, facial expressions and gestures to enact signs. With the emergence of language proper however, these resources become specialised in distinctive ways. Segmental articulation and prosody (rhythm and intonation) are marshalled as the phonology of spoken language.5 But vocal resources such as timbre, tempo, tension and loudness (explored in detail in van Leeuwen 1999) continue as expressive resources, often referred to as sound quality. And gesture, posture and facial expression develop as resources often referred to as body language. As Matthiessen points out, sound quality and body language are then coordinated with language as texts unfold: ...certain interpersonal contrasts in language are realized vocally by contrasts in tone (pitch movement) accompanied by facial contrasts involving eyebrow movements; textual contrasts in deicticity are often accompanied by pointing gestures; talking to babies may involve rounding, pouting lips – a feature that affects the sound but which is also visible; and as detailed studies have shown, there is a complex relationship between addressing somebody in language and gazing at them. (2007: 6–7).
In this paper we will follow Matthiessen (2009: 21–22) in referring to the resources of both sound quality and body language as paralanguage.
Non-semiotic behaviour (somasis)
Trained as we are as linguists and semioticians we are not ourselves in a strong position to further develop this model.9 But we have found it useful to try and compile a range of behaviours that border on semiosis and which can be interpreted by social semiotic animals as indexing purposeful activity. As Halliday and Painter have shown, early protolinguistic semiosis involves a reconstrual of some of these activities as the expression face of signs. And all of the behaviour outlined above has the potential to be used as signs – for example stamping one’s foot in frustration, coughing to remind a meeting of one’s presence, shivering to indicate one is cold, sniffing to object to an odour, kissing on the cheek as a greeting and so on. In these cases there is some degree of deliberation involved, as manifested in the fact that the behaviour will synchronise with the prosodic phonology and turn-taking structure of an interaction and will be responded to as meaningful by co-participants.10
Embodied meaning (semiosis)
Convergent verbiage/image relations in children’s picture books
Sonovergent and semovergent paralanguage
Linguistic body language
Sonovergent paralanguage (phonologically convergent)
Epilinguistic body language
Semovergent paralanguage (semantically convergent)
Sonovergent and semovergent paralinguistic systems will be introduced in turn below, drawing on examples from a Youtube video titled, ‘Let’s Talk. | Random Chatty Vlog’, used here with the presenter’s permission (https://youtu.be/YRx-zDoPbVw). A ‘vlog’ (derived from ‘blog’12) is a video in which a user recounts, or presents, some form of personal activity (e.g. a ‘day in the life’ vlog where the user shows highlights from their activity over a day). The following is the description accompanying the video posted by the vlogger:
Grab a cup of coffee and a snack. Let's just sit down and talk today. I chat about annoying people who follow me in the parking lot, my kids begging for food all summer, my hair, feet, Youtube...etc. I have no trouble rambling on. If you like this, PLEASE give this video a thumbs up so I know! I want to know what you guys like seeing. Thank you for watching! Subscribe so you don't miss another video. I post every Monday, Tuesday and Thursday at 2pm EST.
A full transcription, including intonation13 analysis, of the hair dye episode of this Vlog from which we take most of our examples is provided as Appendix 1.
Sonovergent paralinguistic systems
Tonality, Tonicity, Rhythm
Salient syllables other than the tonic syllable can be given additional prominence through various means. In the following sequence the vlogger’s pitch on the first tone group is unusually high, and contrasts with the descending lower pitch of the following tone group (a sing/song effect).
//3 hopefully / next ↑time I will
//1 get my / ↓hair colour / back //
The same sing/song effect follows on and culminates this section of the vlog, with a high pitch on the tonic syllable / now // contrasting with the low pitch on / do //. The vlogger’s eyebrows once again move up and down in tune and in sync with the contrasting pitch salience (this time on contrasting tonic syllables).
//3 [handclap] / um /but for / ↑now
//3 this will / ↓do //
These rhythmic in-tune gestures reinforce the attitudinal import of the rhythm and tonicity (cf. section “Evaluation (interpersonal semovergent paralanguage)” below).
The contribution of sonovergent paralanguage to the vlog is interrupted in tone group 15 of Appendix 1, suspended for tone groups 16–19, and resumes for tone group 20 – to allow for a somatic phase during which the vlogger uses her left hand to scratch her right arm. This phase unfolds as follows:
//3 lighter than it / was a few /days ago
//1 ^ but /yeah it’s
//1 such a / bummer and then I
//2 went to / Target
//3 ^ like / two days / later and there / was a //
It is interesting to note that the vlogger does not scratch in sync with the rhythm, tonicity and tonality of the text; the scratching lasts for two and a half tone groups, and does not match the timing of salient and tonic syllables. But the paralanguage remains in sync, stopping precisely at the tonic syllable of tone group 15 (/ days ago //), resuming with a smile precisely at the tonic syllable of tone group 18 (/ Target //) and resuming with gesture precisely at the beginning of tone group 19. This indicates that synchronicity with prosodic phonology can function as a demarcating criteria for distinguishing somatic from semiotic behaviour.
Gesture converging with meaning (semovergent paralanguage)
Converging paralinguistic and discourse semantic systems
Tonality, Tonicity, Rhythm
Representation (ideational semovergent paralanguage)
From an ideational perspective we need to take into account how spoken language combines entities, occurrences, qualities and spatiotemporal circumscriptions as figures (ideation), and how these figures are connected to one another (connexion). Semovergent paralanguage supports these resources with hand shapes, which potentially concur with entities, and hand/arm motion, which potentially concurs with occurrences; the hand/arm motion is optionally directed, potentially concurring with spatiotemporal direction (to/from there in space, to/from then in time). We say “potentially concurring” because ideational paralanguage can be used on its own, without accompany spoken language; see the discussion of mime in section "Multidimensionality (multiplying meaning)" below.
By way of illustration we now move to the next section in the vlog, which concerns a visit to the vlogger’s dermatologist (for treatment for granuloma). The sequence of figures we are interested in unfolds verbally in tone groups as follows (for the complete anecdote see Appendix B):
// and so the dermatologist um took like this needle
// and under each like bump
// and injected this like steroid
// and like it all bubbled up //
As with imagic sequences in film, animations, graphic novels, comics, cartoons and picture books, the gesture sequence does not make explicit the conjunctive relations between figures. These have to be abduced (Bateman 2007) from the sequence and concurring language. In the case of this sequence conjunctive relations of time and cause are not made explicit linguistically either; only the additive linker and, is used. A defeasible reading of the sequence is offered below.
// and so the dermatologist um took like this needle
// and under each like bump
// and injected this like steroid
// and like it all bubbled up //
// then they had a snack I
// gave them each a bowl
// like a heaping bowl full of Chex Mix and applesauce squeeze //
Evaluation (interpersonal semovergent paralanguage)
From an interpersonal perspective we need to take into account how spoken language inscribes attitudes, grades qualities and positions voices other than the speaker’s own (appraisal). We also need to account for how speakers exchange feelings, greetings, calls for attention, information and goods & services in dialogue (negotiation). Semovergent paralanguage potentially resonates with appraisal resources through facial expression, bodily stance, muscle tension hand/arm position and motion (Hood 2011, Ngo n.d. in press) and voice quality. Whereas spoken language can make explicit attitudes of different kinds (emotional reactions, judgements of character and appreciation of things), paralanguage can only enact emotion. A further interpersonal restriction (as suggested by Cleírigh), setting aside emblems (discussed in Section “Emblems” below; Kendon 2004, McNeill 2012), is that semovergent paralanguage cannot be used to distinguish move types in dialogic exchanges (although sonovergent paralanguage can of course support tone choice in relation to these moves).
// some guy was sitting there
// and there was cars behind him
// and he was like
// [mimics man’s expression]
// [mimics man’s gesture] like
// waving me out //
//^ ‘it was / some / granuloma /: / /^ / something
// I don’t know- it’s / called- it’s
// some sort of / skin thing //
Voice quality was noted in section “Sonovergent paralanguage” above in relation to the sing/song pitch (high then low) movement the vlogger uses in her last four tone groups to close down her hair dye narrative. From the perspective of appraisal the sound quality resonates with her resignation. Further work on this interpersonal aural dimension of paralanguage, drawing on van Leeuwen 1999, is beyond the scope of our current research.21
Information flow (textual semovergent paralanguage)
From a textual perspective22 we need to take into account how spoken language introduces entities and keeps track of them once there (identification) and how it composes waves of information in tone groups, clauses and beyond (periodicity). Semovergent paralanguage potentially supports these resources with pointing gestures and whole body movement and position.
This is followed by the low pitch tone group; the vlogger is winding down. Following this there is a suspension of both language and paralanguage as her the vlogger’s eyes shut and her head slumps forward (Fig. 36).
Multidimensionality (multiplying meaning)
Several examples of multiple dimensions of paralanguage converging on the same tone group were in fact presented above (for example, the combination of motion towards the future and pointing deixis in Example (19) of section “representation (ideational semovergent paralanguage)”). It is probably safe to say that whenever semovergent paralanguage is deployed, it will be coordinated with tonality, tonicity and rhythm; this is tantamount to arguing that semovergence implies sonovergence. Sonovergent paralanguage on the other hand can be deployed without semovergence, through gestures in tune with or in sync with prosodic phonology (but no more).
An important exception to these principles is what is commonly referred to as mime. In terms of our model mime is semovergent paralanguage that does not accompany language, an apparent contradiction in terms. To explore this further we will return to the miming segment in the parking lot narrative referred to above. The vlogger sets up what happened as follows:
Oh another thing that has been really annoying this summer is you know when you go to a parking lot and it’s a busy place. You get in your car and you don’t necessarily want to leave immediately. Like you might want to- I might want to have Henry test his blood sugar, give the kids snacks. Or if we were at the pool, like change or look at my phone or send a text message or whatever. It drives me crazy when a car is like sitting there following you and then they just wait for you to leave. I cannot stand that. And that has happened so many times. And I was just at the Mall of America and I got back to my car and I went into-. And I met up with a Kimmy from the Dodge family and I went to- I wanted to like Instagram a picture of us and FaceBook whatever. And as I was doing that I- I had...
This is followed by a specific parking lot incident, presented in tone groups below.
// just got in my car
// got my phone and
// as I was doing that
// some guy was sitting there and
// there was cars behind him and
// he was like
// [mimics man’s paralanguage]
// [mimics man’s paralanguage] like
// waving me out and
// I was so:: upset like
// I immediately got up
// put my phone down
// I immediately drove away a bit
// I wasn’t even thinking I
// shouldn’t have done that I
// should not have done that
// but it was just like “What!”
// There’s a guy sitting there
// waving and
// angry at me be-
// cause I was sitting in my car //
As we can see, the two miming segments are heavily co-textualised by language that makes explicit what is going on. The orientation to the narrative introduces the recurrent problem of someone following the vlogger in a parking lot and waiting for her to leave. The miming segments are themselves introduced with the incomplete tone group // he was like... //, with a missing tonic segment. The vlogger then mimes the expected information, before making it linguistically explicit in a tone group converging with the third iteration of the gesture.
Setting aside pantomime (the ‘art of silence’ Marcel Marceau referred to it), we can predict that co-textualisation of this kind is a generalisable pattern as far as semovergent paralanguage in the absence of language is concerned. What the moment of mime does not provide as far as language is concerned, the immediately preceding and following co-text does provide. So the convergent nature of semovergent paralanguage is clear.
These gestures differ from the semovergent ones illustrated thus far in critical ways (cf. McNeill 2012: 7–10). For one thing they commit very specific meanings and can be readily recognised without accompanying co-text. As part of this specificity they can enact moves in exchange structure on their own – e.g. the statements and requests noted above, alongside greetings and leave-takings (hand waving), calls (beckoning gestures), agreement (nodding head), disagreement (shaking head), challenges (upright palm facing forward for ‘stop’) and so on. For another they are much more easily called to consciousness, as the first thing that comes to mind when someone mentions gesture. And in this regard they are often commented on as culturally specific (e.g. the difference between an Anglo supine hand beckoning gesture and its Filipino prone hand equivalent). In both respects emblems contrast with common-sense dismissals of the paralanguage (introduced in sections “Sonovergent paralanguage" and "Gesture converging with meaning (semovergent paralanguage)”) as idiosyncratic (although none of us has any trouble successfully interpreting another speaker’s sonovergent and semovergent systems). From the perspective of the sign language of the deaf, emblems most strongly resemble signs; they are expression form gestures explicitly encoding meaning. Similarly, from the perspective of character based writing systems (such as those of Chinese), emblems most strongly resemble characters (but gestured rather than scribed).
In this paper we have outlined a model distinguishing behaviour from meaning (somasis vs semiosis), and within semiosis, language from paralanguage. Paralanguage itself was then divided into sonovergent and semovergent systems according to their convergence with either the expression plane or content plane of language. Sonovergent systems enact interpersonal meaning in tune with and compose textual meaning in sync with the prosodic phonology of language; semovergent systems construe ideational meaning, enact interpersonal meaning and compose textual meaning convergently with the discourse semantics of language (and its realisation through lexiogrammar).
Compared to other modalities of communication, paralanguage has a distinctive relation to language in that it is coordinated with prosodic phonology. This is obviously true, by definition, for sonovergent paralanguage. But semovergent paralanguage is also coordinated with tonality, tonicity rhythm and tone, since gestures, facial expression, bodily stance and sounds unfold in measures of time converging with units of rhythm and intonation. Even brief episodes of mime follow this principle, filling in for ‘missing’ tonic segments or tone groups as a whole. Alongside this expression form of temporal dependency, paralanguage is dependent on the content form of language because of its inherent generality. Semovergent paralanguage typically commits meaning far less specifically than spoken language can; instantiations are by and large interpretable with respect to what is said. With respect to these two dependencies, the prefix para- (understood as ‘beside’) is appropriate.
Paralanguage as expression form
This of course makes research into the relation between language and paralanguage an interesting case study as far as research into intermodality in general is concerned, possibly helping to clarify some of the theoretical and descriptive challenges posed in Martin 2011.
Our evolving work on these dependencies can be tracked through Martin et al. (2010), Hood (2011), Martin (2011), Martin, Zappavigna, Dwyer, and Cléirigh (2013) Martin and Zappavigna, 2018, Zappavigna and Martin (2018), and Hao and Hood (in press). From the perspective of SFL the most pertinent work on relations between modalities to compare with these studies is Painter et al. 2013 (on language and image in children’s picture books). Beyond these initiatives, multimodal discourse analysis research is best guided by Bateman et al. (2017).
As we stressed at the beginning of the paper building models of intermodality is facilitated if the descriptions of distinct modalities are informed by the same theoretical principles; and this is important for applications. Work in educational linguistics, for example Hood (2011) and Hao and Hood (in press), regularly has to deal with the interaction of language, paralanguage and imaging on Power Point slides. And for forensic linguistics, for example Martin and Zappavigna (2013) and Martin and Zappavigna, 2018, Zappavigna and Martin (2018), language and paralanguage interact with the semiotics of the location of the legal proceedings (which are very different for courtrooms and Youth Justice Conferences). The model of intermodal convergence (ideational concurrence, interpersonal resonance and textual synchronicity) presented in Table 2 above is far easier to operationalise when each of the modalities involved is interpreted from the perspective of SFL.
Our model of paralanguage might also prove of interest as a contribution to the growing field of interactional linguists (Ochs et al. 1996; Fox et al. 2013; Couper-Kuhlen and Selting 2001, 2018). These linguists see language structure as an emergent phenomenon which can only be understood in relation to the use of language in dialogue, and they draw heavily on Conversation Analysis (CA) in their research. This brings paralanguage and other modalities of communication into the picture as far as our understanding of language is concerned (cf. Heath and Luff 2013). SFL’s perspectives on multimodality creates an opportunity for linguistics to make a stronger contribution to this transdisciplinary exercise (Martin forthcoming).
We acknowledge here the contributions of our research group (Susan Hood, Thu Ngo, Clare Painter and Brad Smith). For an elaboration of our work see Ngo et al. forthcoming.
A comparable multifunctional perspective has been proposed (Kok 2016; Kok et al. 2016) within the framework of Functional Discourse Grammar (FDG). In contrast to SFL, FDG recognises two rather than three ‘levels’ – representational and interpersonal (the later concerned with SFL’s interpersonal and textual meaning); and within these levels DFG involves an alternative set of semantic parameters.
Kendon in fact uses this example to argue that the gestures differentiate two meanings of the verb slice. We would argue on the other hand that the gestures are converging with the meaning of two different clause structures (distinguishing two different actions): the hunter…sliced the wolf’s head off involves a phrasal verb (cf. the hunter sliced off the wolf’s head) affecting the wolf, whereas the hunter…sliced the Wolf’s stomach open involves the non-phrasal verb slice, again affecting the wolf, but this time with the effect of the action specified (as the resulting attribute open). Kendon’s misreading reflects the logocentric (word-centred) bias in much of the gesture literature.
Just as gesture and facial expression are marshalled as the expression plane of the sign language of deaf communities (cf. Johnston 2013, 2018 on the use of paralanguage alongside signing in sign language).
Cf. Martinec (e.g. Martinec 2000b) who includes non-semiotic behavior in his modeling.
Halliday and Painter set the minimum number of occasions required for recognition of a sign at 3.
For an introductory overview of Peirce’s model of signs see https://plato.stanford.edu/entries/peirce-semiotics/.
To put this another way, we are arguing that the behaviours outlined in Figure 3 can be treated as paralinguistic or not depending on whether or not they are negotiated as meaningful in interaction. We also need to acknowledge that what we are calling somatic behaviour has the potential to be imbued with cultural norms (e.g. a style of walking, norms for coughing or spitting etc.); these need to be taken into account in future work on somatic behaviour.
Cléirigh’s dimension of protolinguistic body language has been subsumed in our model as subtypes of somasis and interpersonal semovergent paralanguage. This avoids the problem of using the term protolinguistic for a paralinguistic system making meaning alongside language (protolanguage, as initial emergent semiosis, by definition cannot accompany language); and it makes room for paralinguistic systems enabled by the discourse semantic system of appraisal, a system of meaning beyond the scope of Cléirigh's study.
A ‘blog’ (truncated form of ‘weblog’) is a website comprised of posts displayed in reverse chronological order. Most often they involve personal diary-style entries composed by individuals; corporations and organisations may also incorporate blogs into their online material.
We are indebted to Brad Smith for his unfailing support for this analysis.
For wave lengths longer than a tone group whole body motion is involved.
Lateral gesture movement will be named from the perspective of the vlogger.
The term connexion has been adopted from Hao n.d.. (in press) in place of conjunction, to reinforce the differentiation of discourse semantic and lexicogrammatical terminology.
Semovergent synchronicity is concerned with the syncing of paralanguage with periodic structure composed above and beyond prosodic phonology.
The ‘out of kilter’ mouth here can be interpreted as soft focus, converging with kind of.
We also need to acknowledge that a metalanguage for facial expression, in some sense comparable in specificity to SFL work on attitude in the appraisal framework, remains to be developed.
Martinec (1998) interprets textual meaning as realized through cohesion, following Halliday and Hasan (1976); here we follow Martin (1992) who reinterprets cohesion as discourse semantics, organised metafunctionally in Martin and Rose (2007) as ideational resources (ideation, connexion), interpersonal resources (negotiation, appraisal) and textual resources (identification, periodicity).
For transcription conventions see Halliday and Greaves (2008).
An earlier abridged version of this paper was published in Chinese in the journal Contemporary Rhetoric as Martin and Zappavigna (2018).
Jointly constructed article; equal share. Both authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- Caffarel, A., J.R. Martin, and C.M.I.M. Matthiessen, eds. 2004. Language typology: A functional perspective. Amsterdam: Benjamins (Current Issues in Linguistic Theory).Google Scholar
- Couper-Kuhlen, E., and M. Selting. 2001. Studies in interactional linguistics. Amsterdam: Benjamins.Google Scholar
- Couper-Kuhlen, E., and M. Selting. 2018. Interactional linguistics studying language in social interaction. Cambridge: Cambridge University Press.Google Scholar
- Fox, B.A., S.A. Thompson, C.C. Ford, and E. Couper-Kuhlen. 2013. Conversation analysis and linguistics. In The handbook of conversation analysis, ed. J. Sidnell and T. Sirens, 726–740. London: Blackwell.Google Scholar
- Fricke, E. 2013. Towards a unified grammar of gesture and speech: A multimodal approach. In Body – language – communication: An international handbook on multimodality in human interaction (handbooks of linguistics and communication science 38.1), ed. C. Müller, A. Cienki, E. Fricke, S.H. Ladewig, D. McNeill, and S. Teßendorf, 733–754. Berlin: De Gruyter Mouton.Google Scholar
- Halliday, M.A.K. 1970. A course in spoken English: Intonation. London: Oxford University Press.Google Scholar
- Halliday, M.A.K. 1984. Language as code and language as behaviour: A systemic-functional interpretation of the nature and ontogenesis of dialogue. In R. Fawcett, M.A.K. Halliday, S.M. Lamb & A. Makkai .The Semiotics of language and culture: Vol 1: Language as social semiotic. London: Pinter. 3–35. [reprinted in Halliday 2003. 227–250].Google Scholar
- Halliday, M.A.K. 1985. An introduction to functional grammar. London: Arnold.Google Scholar
- Halliday, M.A.K. 1996. On grammar and grammatics. In Functional descriptions: Theory into practice, ed. R. Hasan, C. Cloran, and D. Butt, 1–38. Amsterdam: Benjamins.Google Scholar
- Halliday, M.A.K. 2003. The Language of early Childhood Collected works v.4. Ed. J.J. Webster. London: Continuum.Google Scholar
- Halliday, M.A.K. 2005. On matter and meaning. Linguistics and the Human Sciences 1 (1): 59–82.Google Scholar
- Halliday, M.A.K., and W.S. Greaves. 2008. Intonation in the grammar of English. London: Equinox.Google Scholar
- Halliday, M.A.K., and R. Hasan. 1976. Cohesion in English. London: Longman.Google Scholar
- Hao, J. n.d. in press. Nominalisations in scientific English: a tristratal perspective. Functions of Language.Google Scholar
- Hao, J. & S. Hood. n.d. in press Valuing science: The role of language and body language in a health science lecture. Journal of Pragmatics (Special edition, edited by D. Banks & E. Di Martino). in pressGoogle Scholar
- Heath, C., and P. Luff. 2013. Embodied action and organizational activity. In The handbook of conversation analysis, ed. J. Sidnell and T. Sirens, 283–307. London: Blackwell.Google Scholar
- Hood, S. 2011. Body language in face-to-face teaching. In Semiotic margins: Meaning in multimodalities, ed. S. Dreyfus, S. Hood, and M. Stenglin, 31–52. London: Continuum.Google Scholar
- Johnston, T. 2013. Formational and functional characteristics of pointing signs in a corpus of Auslan (Australian sign language): Are the data sufficient to posit a grammatical class of ‘pronouns’ in Auslan? Corpus Linguistics and Linguistic Theory 9 (1): 109–159.Google Scholar
- Johnston, T. 2018. A corpus-based study of the role of headshaking in negation in Auslan (Australian Sign Language): Implications for signed language typology. Linguistic Typology 22 (2): 185-231. Google Scholar
- Kendon, A. 2004. Gesture: Visible action as utterance. Cambridge University Press, Cambridge.Google Scholar
- Kok, K., K. Bergmann, A. Cienki, and S. Kopp. 2016. Mapping out the multifunctionality of speakers’ gestures. Gesture 15 (1): 37-59.Google Scholar
- Kress, G., and T. van Leeuwen. 1996. Reading images: The grammar of visual design. 1st ed. London: Routledge.Google Scholar
- Martin, J. R. 1992. English text: System and structure. Amsterdam: Benjamins.Google Scholar
- Martin, J. R. 2008. Intermodal reconciliation: Mates in arms. In New literacies and the English curriculum: Multimodal perspectives, ed. L. Unsworth, 112–148. London: Continuum.Google Scholar
- Martin, J. R. 2010. Semantic variation: Modelling system, text and affiliation in social semiosis. In New discourse on language: Functional perspectives on multimodality, identity and affiliation, ed. M. Bednarek and J.R. Martin, 1–34. London: Continuum.Google Scholar
- Martin, J. R. 2011. Multimodal semiotics: Theoretical challenges. In Semiotic margins: Meaning in multimodalities, ed. S. Dreyfus, S. Hood, and M. Stenglin, 243–270. London: Continuum.Google Scholar
- Martin, J. 2014. Evolving systemic functional linguistics: Beyond the clause. Functional Linguistics 1(3).Google Scholar
- Martin, J.R. forthcoming. Once more with feeling: Negotiating evaluation. Language, Context and Text.Google Scholar
- Martin, J. R. & D. Rose. 2007. Working with discourse: Meaning beyond the clause (2nd revised edition). London: Continuum.Google Scholar
- Martin, J. R. & P.R.R. White. 2005. The language of evaluation: Appraisal in English. London: Palgrave.Google Scholar
- Martin, J. R. & Zappavigna, M. 2013. Youth Justice Conferencing: ceremonial redress. International Journal of Law, Language and Discourse 2013: 103–142.Google Scholar
- Martin, J. R. & Zappavigna, M. 2018. Embodied meaning: a systemic functional perspective on body language. Chinese edition Contemporary Rhetoric 1 (Special issue on multimodality edited by Wang Zhenhua). Vol. 1, 2–29 詹姆斯·马丁 米歇尔·扎帕维尼娅 2018.副语言意义研究——系统功能语言学视角。《当代修辞学》.Google Scholar
- Martin, J.R., M. Zappavigna, P. Dwyer & C. Cléirigh. 2013. Users in uses of language: Embodied identity in youth justice conferencing. Text & Talk 33 (4/5): 467–496.Google Scholar
- Martinec, R. 1998. Cohesion in action. Semiotica 120 (1/2): 161–180.Google Scholar
- Martinec, R. 2000b. Types of process in action. Semiotica 130 (3/4): 243–268.Google Scholar
- Martinec, R. 2001. Interpersonal resources in action. Semiotica 135 (1/4): 117–145.Google Scholar
- Martinec, R. 2005. Topics in multimodality. In Continuing discourse on language: A functional perspective, ed. R. Hasan, C.M.I.M. Matthiessen, and J.J. Webster, vol. 1, 157–181. London: Equinox.Google Scholar
- Martinec, R. 2008. Review of Cienki & Muller. Linguistics and the Human Sciences 4 (3): 1–19.Google Scholar
- Matthiessen, C.M.I.M. 2004. The evolution of language: A systemic functional exploration of phylogenetic phases. In The development of language: Functional perspectives on species and individuals, ed. A. Lukin and G. Williams, 45–90. London: Continuum.Google Scholar
- Matthiessen, C.M.I.M. 2007. The multimodal page. In New directions in the analysis of multimodal discourse, ed. T. Royce and W. Bowcher, 1–62. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
- Matthiessen, C.M.I.M. 2009. Multisemiosis and context-based register typology. In The world told and the world shown: Multisemiotic issues, ed. E. Ventola and M. Guijarro, 11–38. London: Palgrave Macmillan.Google Scholar
- McNeill, D. 1992. Hand and mind. Chicago: Chicago University Press.Google Scholar
- McNeill, D., ed. 2000b. Language and gesture. Cambridge: Cambridge University Press.Google Scholar
- Ngo, T. n.d.(in press). Gesture as transduction of characterisation in children’s literature animation adaptation. Australian Journal of Language and Literacy.Google Scholar
- Ngo T, S Hood, JR Martin, C Painter, B Smith, M Martin & M. Zappavigna. forthcoming. Modelling paralanguage from the perspective of Systemic Functional Linguistics: theory and application. London: Routledge.Google Scholar
- O’Halloran, K.L., S. Tan, & P. Wignell n.d. (in press) SFL and Multimodal Discourse Analysis. In G. Thompson, W.L. Bowcher, L. Fontaine, & D. Schonthal (eds.), The Cambridge handbook of Systemic Functional Linguistics. Cambridge: Cambridge University Press.Google Scholar
- Ochs, E., E.A. Schegloff, and S.A. Thompson, eds. 1996. Interaction and grammar. Cambridge: Cambridge University press (studies in interactional Sociolinguistics 13).Google Scholar
- Painter, C. 1984. Into the mother tongue: A case study of early language development. London: Pinter.Google Scholar
- Painter, C., and J.R. Martin. 2012. Intermodal complementarity: Modelling affordances across verbiage and image in children’s picture books. Studies in Systemic Functional Linguistics and Discourse Analysis (III), 132–158. Beijing: Higher Education Press.Google Scholar
- Painter, C., J.R. Martin, and L. Unsworth. 2013. Reading visual narratives: Image analyses of children’s picture books. London: Equinox.Google Scholar
- Smith, B.A., and W.S. Greaves. 2015. Intonation. In The Bloomsbury companion to MAK Halliday, ed. J.J. Webster, 291–313. London: Bloomsbury.Google Scholar
- Taylor, C. 2017. Reading images: Including moving ones. In The Routledge handbook of Systemic Functional Linguistics, ed. T. Bartlett and G. O’Grady, 575–590. London: Routledge.Google Scholar
- Torr, J. 1991. From child tongue to mother tongue: A case study of language development in the first two and a half years (Monographs in Systemic Linguistics no.9). University of Nottingham: Dept. of English Studies.Google Scholar
- Tuite, K. 1993. The production of gesture. Semiotica 93 (1/2): 83–105.Google Scholar
- van Leeuwen, T. 1985. Rhythmic structure of the film text. In Discourse and communication: New approaches to the analysis of mass media discourse and communication, ed. T.A. van Dijk, 216–232. Berlin: De Gruyter.Google Scholar
- van Leeuwen, T. 1992. Rhythm and social context: Accent and juncture in the speech of professional radio announcers. In Systemic phonology, ed. P. Tench, 231–262. London: Pinter.Google Scholar
- Zappavigna, M. & Zappavigna, M. 2018. Discourse and diversionary justice: An analysis of youth justice conferencing. London: Palgrave Macmillan.Google Scholar
- Zappavigna, M., C. Cléirigh, P. Dwyer & J. R. Martin. 2010. The coupling of gesture and phonology. In M. Bednarek, & J.R. Martin (eds.), New discourse on language: Functional perspectives on multimodality, identity and affiliation. London: Continuum. 219–236.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.