Psychological science has for the last decade been engaged in a replication crisis, prompting an increased focus on combatting questionable research practices and attempts to replicate previous results (e.g., the Open Science Collaboration) (for a review see Shrout & Rodgers, 2018). One of the important issues that has received some focus in this debate is the adequacy of the paradigms used in psychological research. If tasks and instructions fail to manipulate the psychological construct under investigation (and only that construct), findings may vary considerably. The present research introduces a novel method of examining the appropriateness of tasks and their instructions by using eye-tracking as a covert measure to evaluate the degree to which participants comply with the task instructions.
Eye movements are seen as a valid measure of attention and cognitive effort (Glaholt & Reingold, 2011; Russo, 2011), and provide information about participants’ acquisition of information, attention and natural shifts in attention through fixations and saccades (Schulte-Mecklenbeck, et al., 2011a, 2011b). Monitoring participants’ fixations within a (pre)defined area of interest (AOI) provides objective information about what participants are looking at and the direction of fixations can indicate search strategies within a display of information, as well as repeated inspections (revisits) of the same material. In addition, the length of each fixation has been used as an indirect measure of cognitive effort, with longer fixations being less common and reflecting a heavier cognitive load than short fixations (Findlay & Kapoula, 1992; Horstmann et al., 2009). As such, eye-tracking methodology provides information about several different psychological processes that are otherwise difficult to access.
To illustrate the use of covert eye-tracking, we used the sequential task paradigm, which is the most common experimental design used for investigating self-control (Carter, et al., 2015). In this paradigm, participants in the experimental condition are required to engage in an initial task requiring use of self-control, such as controlling one’s attention to a specific part of the screen while watching a video. A second, unrelated task, also taxing self-control, is then administered. Worse performance in the second task is taken to be an indicator of the ego depletion effect.
Ego depletion has been a much-debated effect, having failed to replicate in several large-scale replication attempts (Hagger, et al., 2016; Lurquin, et al., 2016) and showing weak effect sizes in meta-analyses (Carter, et al., 2015). There are several reasons why an ego depletion effect may be difficult to observe (Blazquez, et al., 2017; Cunningham & Baumeister, 2016), such as the existence of ego depletion effect itself or, if it exists, whether self-control as a depletable resource is an appropriate theoretical framework to account for the ego depletion effect (e.g., Baumeister, et al., 1998, 2007; Inzlicht & Schmeichel, 2012; Kotabe & Hofmann, 2016). Yet, another possible important issue is methodological, where the adequacy of the tasks used to elicit ego depletion has been questioned (Lee, et al., 2016). Different meta-analyses have suggested that not all manipulations or dependent variables used in the literature may work (Dang, 2018; Lurquin & Miyake, 2017). The attention-control video, for example, has been shown in a replication paper (Lurquin, et al., 2016) and a meta-analysis (Dang, 2018) to potentially be a weak task to manipulate depletion of self-control. However, less is known about why the attention-control video might be a weak task—it might not require self-control or participants might not follow the instructions.
In the attention-control video task, participants watch a six-minute silent video of a woman being interviewed by an offscreen interviewer. During the video, 36 common one-syllable words (e.g., play) appear at the bottom of the screen for 10-s each. The ego depletion manipulation supposedly involves taxing self-control resources of one group but not the other, through manipulating the video-watching instructions. In the taxing condition, participants are instructed to control their visual attention and only concentrate on the woman. In the untaxing condition, participants complete the same task without such instructions and without being informed about the presence of the words. Thus, participants in the taxing group are assumed to use self-control resources to avoid looking at the words and, thus, are predicted to become depleted in self-control compared to those in the untaxing group (Lee, et al., 2016). In other words, the changing words potentially draw participants’ attention in a bottom-up way. However, participants in the taxing condition must override such bottom-up demands, presumably using top-down control because they are instructed to not look at the words, which is taxing.
As such, participants’ compliance with the instructions is a crucial pre-condition for this self-control manipulation to impact performance. If participants in the taxing condition look at the words, they may be less depleted than assumed; conversely, if participants in the untaxing condition either try to avoid the words or try to commit the words to memory they may be more depleted than they are assumed to be. In other words, participants who fail to comply with instructions in either condition may diminish the effectiveness of the self-control manipulation intended to induce the ego depletion effect. Self-report data suggest that participants in the taxing condition may not fully comply with task instructions as they do remember some words (Henderson, et al., 2017; Lurquin, et al., 2016); however, such self-report data only provide a proxy measure for task compliance. What is needed is a more objective measure of behavior during the task.
The attention-control video can be viewed as a task where the words appearing at bottom right of the screen draw the gaze. Such bottom-up effects can be hard to ignore and requires exerting control over one’s eye movements to avoid looking. Eye movements such as fixations, revisits, and fixation durations provide such objective measures of the degree to which participants comply with task instructions and should differ based on the self-control condition participants are randomly allocated to (taxing vs. untaxing). Using (pre)defined AOIs (see Fig. 1) provides objective information about how often participants are looking at either the woman or the words (number of fixations) and how many repeated inspections (revisits) of an AOI do not follow each other in time. In other words, participants in the taxing condition who fully comply with the task instructions should have no fixations and no revisits to the Word-AOI because they do not look at the words. However, the present paradigm allows for a more fine-grained understanding of participants’ compliance as the combination of the number of fixations and revisits gives objective data of the extent of compliance.
In the present paradigm, it is also possible that the task is so easy that participants manage to not to look at the words at all. The length of each single fixation (measured in milliseconds) is often used as an indirect measure of cognitive effort. Fixation durations vary based on the activity at hand, where fixation durations longer than 500 ms often indicate a heavier cognitive load. For example, for participants calculating weighted sums in a city-size task (pre-study), the proportion of long fixation durations (> 500 ms) was 18.5% compared to 1.4% of all fixations for participants processing simple city-size tasks more intuitively (Study 1) (Horstmann, et al., 2009). As such, using self-control resources should require more effort, meaning that participants in the taxing condition should have longer fixation durations on the Woman-AOI compared to participants in the untaxing condition. Fixation durations on the woman should also potentially be longer than those on the words for both conditions because the task is to observe the woman’s body language. Taken together, eye-tracking provides several indices of participants’ attention processes/behavior, making it a useful tool to investigate the adequacy of research paradigms in psychology.
While eye-tracking is a useful method, it is not necessarily practical to include eye-tracking in every research study. The purpose of the present method is rather to understand whether participants are following instructions when they are not observed, which is how experiments are typically run. However, the Hawthorne effect suggests that knowing one’s being observed might change participants’ behavior (McCambridge, et al., 2014). As eye movements can be controlled and we wanted to investigate if participants followed the instructions not to look at the words, it was important for participants to be unaware that their eye movements are being tracked during the task completion. Overt eye-tracking would make it difficult to extrapolate to attention-control paradigms in general, thus we used covert eye-tracking. Next, we report an experiment showcasing a new approach to assess task compliance, using covert recording of eye movements.
The main aim of the paper was, thus, to shed light on whether the attention-control task failed to produce an ego depletion effect because of non-compliance with task instructions, using covert eye-tracking. As a second aim, we also tested whether the impact of the self-control manipulation on performance was moderated by the extent to which participants complied with the video-viewing task instructions. We replicated the method and materials from Schmeichel et al. (2003, Experiment 1) as closely as possible, adding eye-tracking to investigate whether participants actually engage in attentional control behavior while completing the video-viewing task by covertly monitoring gaze patterns.
Hypotheses
Following Schmeichel, et al. (2003), we used different instructions to the attention-control video as the independent variable and tasks from the analytical section of the Graduate Record Exam (GRE) as the dependent variable. H1a, thus, tests whether ego depletion from the attention-control video task occurred, as measured by Schmeichel, et al. (2003), where the main measure of performance is the number of correct responses, the proportion of correct responses indicates overall accuracy, and the number of attempted tasks is a measure of working speed and effort.
H1b and H1c test whether ego depletion occurred as measured by the attentional control exerted by participants.Footnote 1 Hypotheses H2a and H2b test whether participants followed the instructions of the attention-control video task (intended to deplete resources), via the number of fixations in the Word-AOI (H2a) and number of revisits to the Word-AOI (H2b). Finally, H3a tests the effort exerted by participants by comparing the proportion of long fixation durations (> 500 ms) in the Woman-AOI. We, therefore, set out to either support or refute the following pre-registered hypotheses:
H1a: participants in the taxing condition will perform worse on the three performance measures of the GRE, compared to the untaxing condition.
H1b: performance on the GRE will be mediated by the number of fixations in the Word-AOI.
H1c: performance on the GRE will be mediated by effort (measured as proportion of long fixation durations).
H2a: participants in the taxing condition will have fewer fixations in the Word-AOI compared to the untaxing condition.
H2b: participants in the taxing condition will have fewer revisits to the Word-AOI compared to the untaxing condition.
H3a: participants in the taxing condition will have longer fixation durations in the Woman-AOI compared to the untaxing condition.
Prior to data collection, we pre-registered the hypotheses, method and planned analyses on the Open Science Framework https://osf.io/puh6t/. We report all pre-registered analyses below, except the exploratory pupillometry analyses.Footnote 2