Although William Shakespeare supposedly said that the eyes are a window to the soul, experiments indicate that they are literally a window to evaluate brain function (Carpenter, 1994; Hong et al., 2016). The study of rapid eye movements, called saccades, has served as a valuable tool for examining brain function. There are two main types of saccadic eye movements: prosaccades and antisaccades. Prosaccades are the voluntary movement of the eyes toward a visual stimulus, and are the most frequent action an individual performs on a day-to-day basis (Robinson, 1981). In contrast, antisaccades require suppression of the automatic response to look toward a visual stimulus, in order to look opposite it instead. Antisaccades are associated with longer reaction times (RTs) and more directional errors than prosaccades (Hallett, 1978; Munoz & Everling, 2004; see Fig. 1). The behavioral “costs” of antisaccades have been linked to the top-down and executive demands of inhibiting a prepotent prosaccade and the spatial transformation of the stimulus’ coordinates. Accordingly, pro- and antisaccades provide a tool for examining how executive and nonexecutive demands influence oculomotor control in healthy individuals as well as in individuals with neurological disorders, including concussion (Ettenhofer & Barry, 2016; Hoffer et al., 2017; Ting, Schweizer, Topolovec-Vranic, & Cusimano, 2015; Ventura, Balcer, Galetta, & Rucker, 2016).

Fig. 1
figure 1

Schema describing the difference in saccade reaction times. Stimulus onset occurs at 0 s, and A and B mark the onset of eye motion for a prosaccade and an antisaccade, respectively. As illustrated, the reaction time for the antisaccade is longer than that for the prosaccade

Full size image

Eye movements can be recorded using four main techniques: search coil, electrooculography, infrared reflection devices, and video-based (Eggert, 2007). The search coil is considered to be the most accurate technique for recording eye movements, due to its high spatial and temporal resolution (Collewijn, Van der Mark, & Jansen, 1975; Robinson, 1963), but placement of scleral search coils onto the eye influences the kinematics of saccades (Frens & van der Geest, 2002). Additionally, the search coil technique is invasive as compared to the other three techniques. Video-based systems are an acceptable alternative to the search-coil technique (van der Geest & Frens, 2002), due to recent advances in high-speed video. Many studies have contrasted pro- and antisaccade metrics using laboratory-based retroreflective eye-tracking technology (Bucci & Seassau, 2014; Curtis, Calkins, & Iacono, 2001; Evdokimidis et al., 2002; Roll, Wierzbicka, & Wolf, 1996; Weiler, Mitchell, & Heath, 2014). Eye trackers typically require two separate computers to portray the visual stimuli and collect the corresponding saccade data; a control unit for output data and real-time computations; a monitor for visual feedback to the eye; and the camera itself. Precise eye position data are collected that allow for the analysis of various saccade metrics, such as velocity, acceleration, error rate, and RT. Retroreflective eye trackers are known to be sensitive to room lighting and to work best when the pupil is fully dilated in darkness; however, video tracking performance is not influenced by lighting (van der Geest & Frens, 2002). Due to the hardware setup and the requirement of a dark environment for the infrared camera to track pupil gaze, eyetracker systems are cumbersome and typically not portable.

Different models of eye trackers have different capabilities, so a standardized protocol has been proposed to ensure that consistent and quality saccadometric data are collected (Antoniades et al., 2013). Some current alternatives to lab-based eye trackers can collect saccadic error rates or track saccade accuracy. A clinical test involving a patient reacting to a clinician’s finger movements has been developed to qualitatively evaluate antisaccade directional errors (Currie, Ramsden, McArthur, & Maruff, 1991), but it lacks the precision to quantify RTs. Directional errors are likely to occur in populations such as Alzheimer’s or concussion patients (Webb, Humphreys, & Heath, 2018), but they may not be apparent with subclinical injuries such as repeated subconcussive impacts, where a metric such as RT may reveal more. Small, portable eye trackers have been developed, such as the Tobii Pro Glasses 2, ASL Eye Tracking Glasses, and others, but these are limited to 10 to 120 Hz resolution (Allison, Eizenman, & Cheung, 1996; Pelz, Canosa, Kucharczyk, & Babcock, 2000; Takemura, Takahashi, Takamatsu, & Ogasawara, 2014; Topal, Gerek, & Doǧan, 2008)—which is insufficient, relative to the recommended 200 Hz minimum when measuring saccade RTs (Andersson, Nystrom, & Holmqvist, 2010). As such, the creation of an affordable, portable eye-tracking system with sufficient temporal resolution to quantify RTs could allow for the collection of saccadometric data in a variety of settings, such as sports venues (Galetta et al., 2015; Marinides et al., 2015), hospitals, and doctors’ offices. To ensure that the quality of collected data is not compromised, such a system must follow standard data collection protocol (Antoniades et al., 2013) and be validated against an established eye-tracker system (Titz, Scholz, & Sedlmeier, 2018). In the present study, we compared pro- and antisaccade metrics captured via a “traditional” video-based and high-speed retroreflective system (Eye-Trac 6: Applied Sciences Laboratory, Bedford, MA) and captured with a portable, high-speed video-camera-based system (Exilim EX-FH20, Casio, Tokyo, Japan). Accordingly, the goal of this investigation was to evaluate whether the novel system provides a reliable measure of saccade metrics.

Method

Participants

All procedures performed in this study were in accordance with the ethical standards of Western University’s Health Sciences Research Ethics Board and with the 1964 Helsinki Declaration and its later amendments. Fifteen participants (eight male, seven female; M = 29.33 years, SD = 11.16 years) volunteered for this study, and all provided formal written informed consent. All participants had normal or corrected-to-normal vision.

Apparatus and procedure

Participants sat at a table (height 77.5 cm) with their head placed in a head–chin rest throughout data collection. The gaze location of the participant’s left iris was obtained via a video-based fixed eye-tracking system (Eye-Trac 6) sampling at 360 Hz. Prior to data collection, a nine-point calibration of the participant’s viewing space was completed. This system provided real-time point-of-gaze information, pupil diameter, and trial-by-trial saccade kinematics (displacement and velocity), which were calculated using a proprietary algorithm in the system control unit. The experimental control and stimulus presentation were programmed in MATLAB version 7.9 (MathWorks, Natick, MA, USA) using the Psychophysics Toolbox, version 3.0 (Brainard, 1997). The room lights remained on during data collection.

Visual stimuli were presented on a custom-made light board centered on the participant’s midline and located at a viewing distance of 55 cm. Light-emitting diodes (LEDs) were embedded in a 69 × 47 cm stimulus board and covered with black stereo cloth. A multicolor LED (48 cd/m2) located at participants’ midline eye level served as the fixation cross, and target LEDs located 15.5° to the left and right of the central fixation point served as the target stimuli (Weiler et al., 2014). Participants were instructed to react to the visual stimuli “as quickly and accurately as possible.” Each trial began with the presentation of the fixation LED, which alerted participants to direct their gaze to the central location. The color of the fixation LED reminded participants about the type of trial; that is, red and green fixation LEDs indicated pro- and antisaccade performance, respectively. After the participant had achieved a stable gaze (± 1.5° for 500 ms), a randomized foreperiod (1,000–2,000 ms) was initiated, after which the target LED was presented briefly (50 ms) and served as the imperative to initiate the pro- or antisaccade. The fixation LED was visible throughout the trial (no-gap paradigm). Fiber-optic cables (Simplex 1.0 mm; Industrial Fiberoptics, Tempe, AZ, USA), attached in series with the light board, were secured in the field of view of the high-speed camera in order to record the timing of the target light on the video system (see Fig. 2).

Fig. 2
figure 2

(A) Video image from the high-speed camera at the instant of left visual target onset. The overlaid green circles indicate the location of the fiber-optic cables used to indicate timing of the target light. The illuminated right cable in the picture (small bright light) represents a target presented to the left of the participant’s fixation. (B) Participant’s view of the light board with high-speed camera. The red fixation point indicates an antisaccade trial, where the participant is to direct gaze opposite from the orange stimulus light

Full size image

A block of 20 prosaccade trials and a block of 20 antisaccade trials were completed, for a total of 40 saccade trials per participant. Prosaccade trials were performed first, since participants were most familiar with a prosaccade task and accordingly could gain familiarity with the experimental protocol while executing the familiar task before progressing to the antisaccade task. Additionally, trials were completed in separate blocks, in order that saccadic responses would not be confounded with the “switching cost” (Heath, Gillen, & Samani, 2016; Weiler & Heath, 2012). This approach of not randomizing blocks is consistent with other studies’ protocols (Ettinger et al., 2003; Larson & Perry, 1999; Lenoir, Crevits, Goethals, Wildenbeest, & Musch, 2000; Morrillo, Di Russo, Pitzalis, & Spinelli, 2006; Taylor & Hutton, 2009). The target location (i.e., left, right) was pseudo-randomly ordered so as to ensure that each pro- or antisaccade block contained ten left and ten right target presentations and that participants could not anticipate the target location. Trials involving an eye blink were immediately discarded and repeated.

A high-speed digital video camera (Exilim EX-FH20, Casio, Tokyo, Japan, manufacturer’s suggested retail price ~ US $600), placed directly in front of the participant above the light board, recorded the movement of the participant’s eyes at a 210 Hz frame rate with a 480 × 360-pixel image. This sample rate complies with the recommended 200 Hz minimum when measuring saccade RTs (Andersson et al., 2010). Although the camera is capable of measuring at higher frame rates with a smaller image size, the 210 Hz option was chosen so as to maintain an adequate frame size and decrease the amount of time required for video analysis. Note that this specific camera model is no longer produced, but this approach could use any high-speed camera.

The portable high-speed camera system uses the following apparatus: a high-speed camera and tripod, a light board to present stimuli, a chin rest, and a laptop. The high-speed camera and laptop can operate via battery or plugged into a power source, depending on the surrounding environment.

Data analysis

Raw pupil location data were extracted from the Eye-Trac 6 and saved as an ASCII computer file. Horizontal gaze position data were low-pass-filtered offline using a dual-pass Butterworth filter with a cutoff frequency of 15 Hz. Residual analysis determined that this frequency was optimal (Winter, 2005). These data were used to compute instantaneous horizontal velocities and accelerations via a five-point central finite difference algorithm. Saccade onset was determined on the basis of velocity and acceleration thresholds of 30°/s and 8000°/s2, respectively. Trials with an anticipatory response (i.e., RT < 100 ms; Wenban-Smith & Findlay, 1991), a delayed response (i.e., RT > 500 ms), or missing data (e.g., no target light visible in the video recording) were excluded from subsequent analysis. The dependent variables were RT (time from target onset to eye movement onset) and directional errors (i.e., a prosaccade instead of an instructed antisaccade, or vice versa).

The video data from the high-speed camera were examined by two independent raters (J.S.B. and W.J.S.) to evaluate interrater reliability. Both raters were blinded from the RT measures made by the Eye-Trac while extracting saccade RTs from that video data. The saccade RT was determined from the number of frames from onset of the stimulus light (via optic cable) to onset of eye motion. Eye motion onset was identified when the rater first noticed horizontal displacement in the pupil after the stimulus light. The video data were analyzed using QuickTime Player (version 10.4, Apple Inc., California, USA) and were not filtered.

A Bland–Altman plot was used to evaluate the measurement agreement for the saccade RTs between the high-speed video and Eye-Trac (Bland & Altman, 1986; Zaki, Bulgiba, Ismail, & Ismail, 2012) using GraphPad Prism 7 (GraphPad, San Diego, CA, USA). The mean difference of the measurements was calculated and plotted as a horizontal line. The lower and upper limits of agreement (LOAs) were calculated as the mean difference ± 1.96 times the standard deviation.

Linear mixed-effect analyses were performed in R (R Development Core Team, 2018), using the lme4 (Bates, Mächler, Bolker, & Walker, 2015) and lmerTest (Kuznetsova, Brockhoff, & Christensen, 2017) packages to account for removed outliers and variability in participants. The prosaccade and antisaccade RT measurements were compared within each system using type of saccade as a fixed effect and an intercept for participant as a random effect. A model was created to predict Eye-Trac RTs, using high-speed camera RT and type of saccade as fixed effects (with an interaction term) and an intercept for participant as a random effect. Visual inspection of the residual plots did not reveal any obvious deviations from homoscedasticity or normality. An unstandardized regression weight (r) and y-intercept was used to estimate the correlation and offset between the RTs of both systems. A regression weight of 1 refers to perfect unity between the two systems, where an increase of one unit of the independent variable will result in the same increase in the dependent variable and the y-intercept value accounts for any offset between the independent and dependent variables. Satterthwaite’s degree-of-freedom method in the lmerTest package were used to obtain p values. A p value less than .05 was considered statistically significant.

Intraclass correlation coefficients (ICCs) and their 95% confidence intervals (CIs) were calculated to quantify the reliability between the RT measures from the high-speed camera and Eye-Trac systems, as well as the interrater reliability for the high-speed camera system RT measures. The ICC reflects both degree of correspondence and agreement among ratings (Portney & Watkins, 2000). A single-measurement, absolute-agreement, two-way mixed-effects ICC model was used to compare the RT measures from the systems, and an average-measurement, absolute-agreement, two-way random-effects ICC model was used to compare interrater reliability. ICC calculations were conducted using the SPSS statistical package, version 25 (SPSS Inc., Chicago, IL). Excellent reliability was defined as a value greater than .90, according to published guidelines (Koo & Li, 2016), and the level of significance was set at an alpha of less than .05 a priori.

Results

Antisaccade RTs measured by the Eye-Trac were longer (M = 285.29 ms, SD = 53.94 ms) than prosaccade RTs (M = 272.39 ms, SD = 58.14 ms)—a statistically significant increase of 12.91 ms, 95% CI [21.29, 4.53], t(513) = 3.02, p = .00266, d = 0.266 (see Fig. 3). Similarly, antisaccade RTs measured by the high-speed camera were longer (M = 297.14 ms, SD = 53.76 ms) than prosaccade RTs (M = 283.90 ms, SD = 56.80 ms)—a statistically significant increase of 13.24 ms, 95% CI [21.50, 4.97], t(513) = 3.14, p = .00179, d = 0.277 (see Fig. 3).

Fig. 3
figure 3

Frequency distribution of prosaccade and antisaccade reaction time trials measured by the Eye-Trac (top) and high-speed camera (bottom) systems. Antisaccade reaction times were significantly longer than prosaccade reaction times for both systems (p < .05)

Full size image

Trials displaying an anticipatory response, a delayed response, or missing data accounted for 11.8% of trials and were excluded from the analysis. A total of 529 trials were used to analyze the validity of the RT measure of the high-speed camera system as compared to the Eye-Trac system. We found an excellent degree of reliability (ICC = .959, 95% CI [.719, .985]) between the saccade RT measures from the high-speed camera and Eye-Trac. The Bland–Altman analysis revealed that the average saccade RT determined using the high-speed video camera was 11 ms longer than those based on the Eye-Trac system across all trials (bias = 11.53 ± 11.33 ms; lower LOA, – 10.69; upper LOA, 33.74; see Fig. 4).

Fig. 4
figure 4

Bland–Altman plot for the mean saccade reaction times measured by the high-speed video system and Eye-Trac. The solid line represents the mean difference between the high-speed video and the Eye-Trac, and the dashed lines represent the 95% confidence intervals around that mean difference

Full size image

There was 100% concordance between the trial direction as measured with the high-speed camera and Eye-Trac systems, resulting in perfect error rate agreement between the systems. Participants did occasionally make direction errors in both prosaccades and antisaccades (0.13% error rate for prosaccades and 0.52% error rate for antisaccades). Fewer than three outliers occurred per 100 measurements. The mixed-model analysis revealed a statistically significant strong correlation (r = .97, 95% CI [.94, .99]) between the high-speed camera and Eye-Trac RT measurements [t(493) = 72.60, p < .001], with an offset of 4.27. The 4.27 offset revealed that the average saccade RT determined using the high-speed video camera was 4.27 ms longer than the Eye-Trac system RT across all trials. The interaction term was not statistically significant [t(522) = 1.668, p = .096], suggesting no difference in the ability of the high-speed camera system to capture the RTs measured by the Eye-Trac system between different types of saccades. There was excellent interrater reliability (ICC = .995, 95% CI [.994, .996]) of the high-speed video saccade RT measures.

Discussion

The purpose of this study was to evaluate the performance of a novel system for measuring saccade RT and error rate. Saccade RT and error rate are recommended measures for evaluating the executive-based demands of the antisaccade task (Bucci & Seassau, 2014; Evdokimidis et al., 2002; Munoz, Broughton, Goldring, & Armstrong, 1998; Peltsch, Hoffman, Armstrong, Pari, & Munoz, 2008; Roll et al., 1996). Therefore, it is important for a valid and reliable system to accurately measure these parameters. The results from this study showed that the saccade RTs measured by the high-speed camera were strongly correlated (r = .97) to the measurements by the Eye-Trac. In line with previous work (Peltsch et al., 2008), antisaccades produced longer RTs and more directional errors than did their prosaccade counterparts—a finding tied to the antisaccade executive demands of response suppression and vector inversion (Evdokimidis et al., 2002). The high-speed camera system recorded saccade errors identical to those recorded by the Eye-Trac. There were fewer than three outliers per 100 measurements, illustrating that the high-speed video measures are reasonable measures of the saccade RT (Chatburn, 1996). The mixed-model analysis, which accounted for subject and saccade type variability, showed that on average the high-speed video camera recorded RTs that were 4.27 ms longer than those from the Eye-Trac system. This value may reflect limitations from the measurement accuracy error due to the equipment’s sampling frequencies (Andersson et al., 2010). The lower sampling frequency of the high-speed camera system would be expected to record a longer RT than the Eye-Trac, since we were detecting onsets, which would not appear until the subsequent measurement frame.

The excellent interrater reliability of this high-speed camera system exemplifies the validity of the measured saccade RTs. Both raters were also able to distinguish any errors in directional eye movements for both prosaccades and antisaccades.

A study examining a population of healthy young men (Evdokimidis et al., 2002) determined that the mean antisaccade RT (± SD) was 270 ± 39 ms, similar to the 285 ± 54 ms antisaccade RT measured by the Eye-Trac in this study. Another study with a large sample and a range of ages (Munoz et al., 1998) found saccade RTs similar to those measured by the high-speed camera system. The choice of a no-gap paradigm for our saccade trials resulted in longer prosaccade RTs. Similar results have been shown in other studies (mean RT = 280.01 ms; Munoz et al., 1998), with comparable prosaccade and antisaccade ranges of RTs (140–350 ms and 175–375 ms, respectively; Fischer & Weber, 1992). The time it takes for visual attention to disengage adds to the RT and accounts for the long latencies in no-gap trials, as compared with the short latencies in gap trials, where the disengagement can take place during the gap (Fischer & Weber, 1993).

Our validation of the high-speed camera system indicates that it is a useful method for quantifying reaction time for saccadic eye movements. Furthermore, since it works in a lighted environment, it can be utilized in a variety of settings, including the possibility for baseline and follow-up evaluations for a concussive event. The King–Devick test is a concussion assessment tool that requires participants to complete sequential prosaccades while reading aloud a series of numbers, and therefore provides assessments of eye movements, language, and attention control. The King–Devick test, in combination with a neuropsychological battery assessing a wider range of cognitive functions, balance, and symptoms, has a high sensitivity (92%–100%) for detecting concussion (Galetta et al., 2015; Marinides et al., 2015). However, mounting evidence demonstrates that antisaccades may provide a more sensitive measure to detect the subtle executive dysfunction that is the most salient and persistent sequela of a concussion (DiCesare, Kiefer, Nalepka, & Myer, 2017; Webb et al., 2018). The antisaccade may be a better indicator of impaired brain function in concussion than previous tests have been (Balaban et al., 2016; Heitger et al., 2004; Heitger et al., 2006; Hoffer et al., 2017). For example, persons in the acute and subacute stages of concussion elicited a marked and persistent deficit in antisaccade RTs as compared to age-matched healthy controls, whereas no such persistent deficit was identified via prosaccades (Johnson, Hallett, & Slobounov, 2015a; Johnson, Zhang, Hallett, & Slobounov, 2015b). It therefore may be the case that the executive demands associated with antisaccades provide adequate resolution to detect subtle deficits arising from a concussion and may provide a valid tool for identifying recovery of function and determining a timeline for safe return to play (Webb et al., 2018). The high-speed camera’s frame rate of 210 Hz is sufficient to detect such antisaccade deficits, and high-speed video is becoming more accessible, with current smartphones, for instance, recording 720p video at 240 Hz.

The high-speed camera system does have some limitations. Foremost are the time required for analysis and the lack of real-time assessment. Each session must be analyzed by a rater to determine the saccade RT and error rate, and this requires approximately 30 min per block of 20 trials. Although the process is laborious, it does offer the advantage of having a permanent video record of the trial, as compared to the Eye-Trac system, which does not keep a record of the raw footage. Currently, the high-speed video system does not offer real-time feedback for the participant or experimenter regarding any saccadic measurements. In addition, this system quantifies reaction time and saccade errors, which appear to be relevant for evaluating individuals with concussions (Heitger et al., 2004; Heitger et al., 2009; Kraus et al., 2007); however, it does not capture other measures, such as amplitude, velocity, and acceleration, that also appear to be relevant (Caplan et al., 2015).

Future work should focus on the efficiency of the video data analysis. At present the process is slow, in that each trial must be examined frame by frame. Incorporating the use of automated video analysis (Kassner, Patera, & Bulling, 2014), or perhaps electrooculography (Pettersson et al., 2013), would allow the whole process to be automated and analyzed in real time. Such real-time data could also provide immediate feedback to examiners and participants. As part of a sideline assessment tool, such real-time feedback would be critical in evaluating any differences in saccadic RTs or errors, reflecting a possible diagnosis of concussion (Ventura et al., 2016).