Integrating mechanisms of visual guidance in naturalistic language production

Abstract

Situated language production requires the integration of visual attention and linguistic processing. Previous work has not conclusively disentangled the role of perceptual scene information and structural sentence information in guiding visual attention. In this paper, we present an eye-tracking study that demonstrates that three types of guidance, perceptual, conceptual, and structural, interact to control visual attention. In a cued language production experiment, we manipulate perceptual (scene clutter) and conceptual guidance (cue animacy) and measure structural guidance (syntactic complexity of the utterance). Analysis of the time course of language production, before and during speech, reveals that all three forms of guidance affect the complexity of visual responses, quantified in terms of the entropy of attentional landscapes and the turbulence of scan patterns, especially during speech. We find that perceptual and conceptual guidance mediate the distribution of attention in the scene, whereas structural guidance closely relates to scan pattern complexity. Furthermore, the eye–voice span of the cued object and its perceptual competitor are similar; its latency mediated by both perceptual and structural guidance. These results rule out a strict interpretation of structural guidance as the single dominant form of visual guidance in situated language production. Rather, the phase of the task and the associated demands of cross-modal cognitive processing determine the mechanisms that guide attention.

Appendices

Appendix 1: Syntactic chunking

In order to quantify syntactic guidance, the sentences produced in our experiment were decomposed into syntactic chunks. Syntactic chunks are an intermediate representation between simple parts of speech and a full parse, which gives us basic structural information about the complexity of a sentence.

Chunking was performed automatically using the TagChunk system developed by Daumé and Marcu (2005), which performs combined part-of-speech tagging and syntactic chunking. It assigns syntactic labels to a sequence of words using the BIO encoding, in which the beginning of phrase X (e.g., noun phrase or NP) is tagged B-X (e.g., B-NP), the non-beginning (inside) of the X phrase is tagged I-X (e.g., I-NP), and any word that is not in a phrase is tagged O (outside). The TagChunk systems achieves a chunking accuracy of 97.4 % on the CoNLL 2,000 data set (8,936 training sentences, and 2,012 test sentences). As an example, consider a description of the scene in Fig. 1, and its chunked version:

• (1) There is a clipboard sitting on a coffee table and another clipboard next to it on which a US army man is writing.

• (2) [B-NP There] [B-VP is] [B-NP a [I-NP clipboard]] [B-VP sitting] [B-PP on] [B-NP a [I-NP coffee [I-NP table]]] [B-O and] [B-NP another [I-NP clipboard]] [B-ADJP next] [B-PP to] [B-NP it] [B-PP on] [B-NP which] [B-NP a [I-NP US [I-NP army [I-NP man]]]] [B-VP is [I-VP writing]].

Based on the TagChunk output, we can calculate the frequency of each syntactic type and the total number of constituents (e.g., the frequency of B-NP is 7 in the above sentence), which we use as factors in the mixed models reported in the main text.

Appendix 2: Turbulence

The concept of turbulence of a sequence comes from research in population research, where the goal is to quantify the complexity of different life trajectories (e.g., Single, Married, Married with Children, sequence: S–M–MC, vs. Single only, sequence: S; Elzinga and Liefbroer 2007). Turbulence is a composite measure calculated by integrating information about: (1) number of states a sequence is composed of, (2) their relative duration, and (3) the number of unique subsequences that can be derived from it.

Suppose we have three different scan patterns (x = man-R, man-L, clipboard, window, y = man-R, clipboard, man-R, man-L, z = man-R, clipboard, man-L, man-R) each of which consists of four fixated objects. Intuitively, x is more turbulent than y and z, as in x all fixated objects are different and inspected only once. When we compare y and z instead, we find that they have the same number of uniquely fixated objects, but in z we find that more objects are fixated before looking back at man-R. Therefore, z can be considered to be more turbulent than y, as more events have occurred before man-R is inspected again.

A combinatorial implication of this reasoning is that more distinct subsequences, denoted as $\phi (x)$, can be generated from z than from y. In fact, when computing the number of distinct subsequences for the three scan patterns, we find that $\phi (x) = 16 > \phi (z) = 15 > \phi (y) = 14$.⁴ Each state of a sequence, however, is often associated with a certain duration. Let’s assume man-R is fixated for 200 ms before looking at another object. If we include fixation duration on x, we will obtain something like man-R/200, man-L/200, clipboard/200, window/200, and its turbulence increases when the variance duration between states decreases. So, x has a relative high turbulence (variance of 0 across states), compared to a scan pattern such as man-R/400, man-L/50, clipboard/75, man-L/50, where the duration is concentrated on a single state.

These intuitions can be formalized by defining the turbulence of a sequence as (Elzinga and Liefbroer 2007):

$$\begin{aligned} T(x) = \log _{2}\bigg (\phi (x) \frac{s_{t,\mathrm{max}}^2(x) + 1}{s_{t}^2 (x) + 1} \bigg ) \end{aligned}$$

(2)

where $\phi (x)$ denotes the number of distinct subsequences, $s_{t}^2$ is the variance of the state-durations and $s_{t,\mathrm{max}}^2$ the maximum of that variance given the total duration of the sequence, calculated as $s_{t,\mathrm{max}}^2 = (n - 1) (1 - \bar{t})^2$ with n denoting the number of states and $\bar{t}$ the average state duration of the sequence x. To compute the turbulence for the analyses in the main text, we utilized the R-package TraMine, a toolbox developed by Gabadinho et al. (2011) to perform categorical sequence analysis.

Appendix 3: Distance from mentioned, but not fixated, objects

Since the distance from a fixation is known to have processing implications (Nelson and Loftus 1980), we performed an additional analysis in which we looked at the closest fixation to the referent when is not mentioned, and calculate the distance of this fixation from the object centroid. We find that on average, fixations are at $11.65 \pm 5.04$ degree of visual angle from the object centroid. This indicates that peripheral vision is sufficient to identify and select the object for mentioning: for para-foveal effects, the object should be fixated within 4–5 degrees of visual angle. Moreover, a mixed model analysis reveals that the distance varies with the amount of visual clutter and the animacy of the target. In particular, an animate object tends to have a smaller visual distance (${\beta }_{{\rm Animate}} = -3.12;\,p < 0.05$), especially in a scene with minimal clutter (${\beta }_{{\rm Animate:Minimal}} = -6.25;\,p < 0.01$). This indicates that foveating the object is not an obligatory element of mentioning.