REMoDNaV: Robust Eye Movement Detection for Natural Viewing

,


Introduction
A spreading theme in cognitive neuroscience is to use dynamic and natural stimuli as opposed to isolated and distinct imagery (Matusz et al., 2019). Using dynamic stimuli promises to observe the nuances of cognition in a more natural environment. Some interesting applications include the determination of neural response to changes in facial expression (Harris et al., 2014), understanding complex social interactions by using videos (Tikka et al., 2012) and more untouched themes such as the underlying processing of music (Toiviainen et al., 2014). In such studies, an unobtrusive behavioral measurement is required to quantify the relationship between stimulus and response. Tracking the focus of participants' gaze is a suitable, well established measure that has been successfully employed in a variety of studies ranging from the understanding of visual attention (Liu and Heynderickx, 2011), memory (Hannula et al., 2010) and language comprehension (Gordon et al., 2006). Regardless of use case, the raw eye tracking data (position coordinates) provided by eye tracking devices are rarely used "as is". Instead, in order to disentangle different cognitive, occulomotor, or perceptive states associated with different types of eye movements, most research relies on the classification of eye gaze data into distinct eye movement event categories (Schutz et al., 2011). The most feasible approach for doing this lies in the application of appropriate event detection algorithms.
However, a recent comparison of algorithms found that while many readily available algorithms for eye movement classification performed well on data from static stimulation or short trial-based acquisitions with simplified moving stimuli, none worked particularly well on data from complex natural dynamic stimulation, such as video clips, when compared to human coders (Andersson et al., 2017). And indeed, when we evaluated an algorithm by Nyström and Holmqvist (2010), one of the winners in the aforementioned comparison, on data from prolonged stimulation (≈15 min) with a feature film, we found the average and median durations of labeled fixations to exceed literature reports (e.g., Holmqvist et al., 2011;Dorr et al., 2010) by up to a factor of two. Additionally, and in particular for increasing levels of noise in the data, the algorithm classified too few fixations, as also noted by Friedman et al. (2018), because it discarded poten-tial fixation events that contained artifacts such as blinks. However, robust performance on noisy data is of particular relevance in the context of "natural stimulation", as the ultimate natural stimulation is the actual natural environment, and data acquired outdoors or with mobile devices typically does not match the quality achieved in dedicated lab setups. Therefore our objective was to improve upon the available eye movement detection and classification algorithms, and develop a tool that performs robustly on data from dynamic natural stimulation, without sacrificing detection accuracy for static and simplified stimulation. Importantly, we aimed for applicability to prolonged recordings that lack any kind of trial structure, and exhibit periods of signalloss and non-stationary noise levels. In addition to the event categories fixation, saccade, and postsaccadic oscillation (PSO; sometimes termed "glissade"), the algorithm had to support the detection of smooth pursuit events, as emphasized by Andersson et al. (2017). These are slow movements of the eye during tracking of a moving target and are routinely evoked by moving visual objects during dynamic stimulation (Carl and Gellman, 1987). If this type of eye movement is not properly detected and labeled, erroneous fixation and saccade events (which smooth pursuits would be classified into instead) are introduced. Contemporary algorithms rarely provide this functionality (but see e.g., Larsson et al., 2015;Komogortsev and Karpov, 2013, for existing algorithms with smooth pursuit detection).
Here we introduce REMoDNaV (robust eye movement detection for natural viewing), a novel tool that aims to meet these objectives. It is built on the aforementioned algorithm by Nyström and Holmqvist (2010) (subsequently labeled NH) that employs an adaptive approach to velocity based eye movement event detection and classification. RE-MoDNaV enhances NH with the use of robust statistics, and a compartmentalization of prolonged time series into short, more homogeneous segments with more uniform noise levels. Furthermore, it adds support for pursuit event detection. We evaluated REMoDNaV on three different datasets from conventional paradigms, and natural stimulation (high and lower quality), and relate its performance to the algorithm comparison by Andersson et al. (2017).

Methods
Like NH, REMoDNaV is a velocity-based event detection algorithm. Compared to dispersion-based algorithms, these types of algorithms are less susceptible to noise and spatio-temporal precision, and are thus applicable to a wide range of sampling frequencies. Furthermore, any influence of biologically implausible velocities and accelerations can be prevented with the use of appropriate filters and thresholds (Holmqvist et al., 2011).
The algorithm comprises two major steps: preprocessing and event detection. A general overview and pseudo-code are shown in Figure 1. The following sections detail individual analysis steps. For each step relevant algorithm parameters are given in parenthesis.

Preprocessing
The goal of data preprocessing is to compute a time series of eye movement velocities on which the event detection algorithm can be executed, while jointly reducing non-movement-related noise in the data as much as possible.
First, implausible spikes in the coordinate time series are removed with a heuristic spike filter (Stampe, 1993) (Figure 1A, 1). This filter is standard in many eye tracking toolboxes and often used for preprocessing (e.g., Nyström and Holmqvist, 2010). Data samples around signal loss (e.g., eye blinks) can be nulled in order to remove spurious movement signals (dilate nan, min blink duration; Figure 1A, 2). Coordinate time series are temporally filtered in two different ways ( Figure 1A, 3). A relatively large median filter (median filter length) is used to emphasize long-distance saccades. This type of filtered data is later used for a coarse segmentation of a time series into shorter intervals between major saccades. Separately, data are also smoothed with a Savitzky-Golay filter (savgol {length,polyord}). All event detection beyond the localization of major saccades for time series chunking is performed on this type of filtered data.
After spike-removal and temporal filtering, movement velocities are computed ( Figure 1A, 4-5). To disregard biologically implausible measurements, a configurable maximum velocity (max vel) is enforced-any samples exceeding this threshold are replaced by this set value.

Saccade velocity threshold
Except for a few modifications, REMoDNaV employs the adaptive saccade detection algorithm proposed by Nyström and Holmqvist (2010), where saccades are initially located by thresholding the velocity time series by a critical value. Starting from an initial velocity threshold (velthresh startvelocity, termed P T 1 in NH), the critical value is determined adaptively by computing the variance of sub-threshold velocities (V ), and placing the new velocity threshold at: where F determines how many standard deviations above the average velocity the new threshold is located. This procedure is repeated until it stabilizes on a threshold velocity.
REMoDNaV alters this algorithm by using robust statistics that are more suitable for the nonnormal distribution of velocities (Friedman et al., 2018), such that the new threshold is computed by: where M AD is the median absolute deviation, and F is a scalar parameter of the algorithm.

Time series chunking
As the algorithm aims to be applicable to prolonged recordings without an inherent trial structure and inhomogeneous noise levels, the time series needs to be split into shorter chunks to prevent the negative impact of sporadic noise flares on the aforementioned adaptive velocity thresholding procedure.
REMoDNaV implements this chunking by determining a critical velocity on a median-filtered (median filter length) time series comprising the full duration of a recording ( Figure 1D). Due to potentially elevated noise levels, the resulting threshold tends to overestimate an optimal threshold.
3 . CC-BY 4.0 International license It is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.  4 . CC-BY 4.0 International license It is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint . http://dx.doi.org/10.1101/619254 doi: bioRxiv preprint first posted online Apr. 26, 2019; noise factor adaptive saccade onset threshold velocity is the median absolute deviation of velocities in the window of interest, times this factor (peak velocity threshold is twice the onset velocity); increase for noisy data to reduce false positives (Nyström and Holmqvist, 2010, equivalent: 3.0) 5 velthresh startvelocity start value for adaptive velocity threshold algorithm (Nyström and Holmqvist, 2010), should be larger than any conceivable minimum saccade velocity 300 deg/s max initial saccade freq maximum saccade frequency for initial detection of major saccades, initial data chunking is stopped if this frequency is reached (should be smaller than an expected (natural) saccade frequency in a particular context), default based on literature reports of a natural, free-viewing saccade frequency of ∼1.7 ±0.3 Hz during a movie stimulus (Amit et al., 2017) 2 Hz saccade context window length size of a window centered on any velocity peak for adaptive determination of saccade velocity thresholds (for initial data chunking only) 1 s lowpass cutoff freq cut-off frequency of a Butterworth low-pass filter applied to determine drift velocities in a pursuit event candidate 4 Hz pursuit velthresh fixed drift velocity threshold to distinguish periods of pursuit from periods of fixation; higher than natural ocular drift velocities during fixations (e.g., Goltz et al., 1997;Cherici et al., 2012) 2 deg/s 5 . CC-BY 4.0 International license It is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint .
Consequently, only periods of fastest eye movements will exceed this threshold. All such periods of consecutive above-threshold velocities are weighted by the sum of these velocities. Boundaries of time series chunks are determined by selecting such events sequentially (starting with the largest sums), until a maximum average frequency across the whole time series is reached (max initial saccade freq). The resulting chunks represent data intervals between saccades of maximum magnitude in the respective data.

Detection of saccades and post-saccadic oscillations
Detection of these event types is identical to the NH algorithm, only the data context and metrics for determining the velocity thresholds differ. For saccades that also represent time series chunk boundaries (event label SACC), a context of 1 s (saccade context window length) centered on the peak velocity is used by default, for any other saccade (event label ISAC) the entire time series chunk represents that context ( Figure 1E).
Peak velocity threshold and on/offset velocity threshold are then determined by equation 3 with F set to 2 × noise factor and noise factor, respectively. Starting from a velocity peak, the immediately preceding and the following velocity minima that do not exceed the on/offset threshold are located and used as event boundaries.
Qualifying events are rejected if they do not exceed a configurable minimum duration or violate the set saccade maximum proximity criterion (min saccade duration, min intersaccade duration).
As in NH, post-saccadic oscillations are events that immediately follow a saccade, where the velocity exceeds the saccade velocity threshold within a short time window (max pso duration). REMoD-NaV distinguishes low-velocity (event label LPSO for chunk boundary event, ILPS otherwise) and high-velocity oscillations (event label HPSO or IHPS), where the velocity exceeds the saccade onset or peak velocity threshold, respectively.

Pursuit and fixation detection
For all remaining, unlabeled time series segments that are longer than a minimum duration (min fixation duration), velocities are low-pass filtered (Butterworth, lowpass cutoff freq). Any segments exceeding a minimum velocity threshold (pursuit velthresh) are classified as pursuit (event label PURS). Pursuit on/offset detection uses the same approach as that for saccades: search for local minima preceding and following the above threshold velocities. Any remaining segment that does not qualify as a pursuit event is classified as a fixation (event label FIXA).

Operation
REMoDNaV is free and open-source software, written in the Python language and released under the terms of the MIT license. In addition to the Python standard library it requires the Python packages NumPy (Oliphant, 2006), Matplotlib (Hunter, 2007), statsmodels (Seabold and Perktold, 2010), and SciPy (Jones et al., 2001-) as software dependencies. Furthermore, DataLad (Halchenko et al., 2013-), and Pandas (McKinney et al., 2010) have to be available to run the test battery. REMoDNaV itself, and all software dependencies are available on all major operating systems. There are no particular hardware requirements for running the software other than sufficient memory to load and process the data.
A typical program invocation looks like remodnav <inputfile> <outputfile> \ <px2deg> <samplingrate> where <inputfile> is the name of a tab-separatedvalue (TSV) text file with one gaze coordinate sample per line. An input file can have any number of columns, only the first two columns are read and interpreted as X and Y coordinates. The second argument <outputfile> is the file name of a BIDScompliant (Gorgolewski et al., 2016) TSV text file that will contain a report on one detected eye movement event per line, with onset and offset time, onset and offset coordinates, amplitude, peak velocity, median velocity and average velocity. The remaining arguments are the only two mandatory parameters: the conversion factor from pixels to visual degrees, i.e., the visual angle of a single (square) pixel (<px2deg> in deg), and the temporal sampling rate (<sampling rate> in Hz). All additionally supported parameters (sorted by algorithm step) with their description and default 6 . CC-BY 4.0 International license It is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint . http://dx.doi.org/10.1101/619254 doi: bioRxiv preprint first posted online Apr. 26, 2019; value, are listed in Table 1. While the required user input is kept minimal, the number of configurable parameters is purposefully large to facilitate optimal parameterization for data with specific properties. Besides the list of detected events, a visualization of the detection results, together with a time course of horizontal and vertical gaze position, and velocities is provided for illustration and initial quality assessment of algorithm performance on each input data file.

Validation analyses
The selection of datasets and analyses for validating algorithm performance was guided by three objectives: 1) compare to other existing solutions; 2) demonstrate plausible results on data from prolonged gaze coordinate recordings during natural viewing; and 3) illustrate result robustness on lowerquality data. The following three sections each introduce a dataset and present the validation results for these objectives. All analysis presented here are performed using default parameters (Table 1), with no dataset-specific tuning other than the built-in adaptive behavior.

Algorithm comparison
Presently, Andersson et al. (2017) represents the most comprehensive comparative study on eye movement detection algorithms. Moreover, the dataset employed in that study was made publicly available. Consequently, evaluating REMoDNaV performance on these data and using their metrics offers a straightforward approach to relate this new development to alternative solutions.
The dataset provided by Andersson et al. (2017) 1 consists of monocular eye gaze data produced from viewing stimuli from three distinct categoriesimages, moving dots and videos. The data release contains gaze coordinate time series (500 Hz sampling rate), and metadata on stimulus size and viewing distance. Importantly, each time point was manually classified by two expert human raters as one of six event categories: fixation, saccade, PSO, smooth pursuit, blink and undefined (a sample that did not fit any other category). A minor labeling mistake 1 github.com/richardandersson/EyeMovementDetector Evaluation reported in Zemblys et al. (2018) was fixed prior to this validation analysis.
For each stimulus category, we computed the proportion of misclassifications per event type, comparing REMoDNaV to each of the human coders, and, as a baseline measure, the human coders against each other. A time point was counted as misclassified if the two compared classifications did not assign the same label. We limited this analysis to all time points that were labeled as fixation, saccade, PSO, or pursuit by any method, hence ignoring the rarely used NaN/blinks or "undefined" category. For a direct comparison with the results in Andersson et al. (2017), the analysis was repeated while also excluding samples labeled as pursuit. Table 2 shows the misclassification rates for all pairwise comparisons, in all stimulus types. In comparison to the NH algorithm, after which the proposed work was modelled, REMoDNaVperformed consistently better (32/93/70% average misclassification for NH, vs. 6.5/10.8/ 9.1% worst misclassification for REMoDNaV in categories images, dots, and videos). Compared to all ten algorithms evaluated in Andersson et al. (2017), REMoDNaV exhibits the lowest misclassification rates across all stimulus categories. When taking smooth pursuit events into account, the misclassification rate naturally increases, but remains comparably low. Importantly, it still exceeds the performance of all algorithms tested in Andersson et al. (2017) in the dots and video category, and performs among the best in the images category. Additionally, both with and without smooth pursuit, REMoDNaVs performance exceeds also that of a recent deep neural network trained specifically on video clips (Startsev et al., 2018, compare Table 7: 34% misclassification versus 31.5% for REMoDNaV). Figure 2 shows confusion patterns for a comparison of algorithm classifications with human labeling. While REMoDNaV does not achieve a labeling similarity that reaches the human inter-rater agreement, it still performs well. In particular, the relative magnitude of agreement with each individual human coder for fixations, saccades, and PSOs, resembles the agreement between the human coders. Classification of smooth pursuits is consistent with human labels for the categories moving dots, and videos. However, there is a substantial confusion of fixation and pursuit for the static images. In a realworld application of REMoDNaV, pursuit detection 7 . CC-BY 4.0 International license It is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint . could be disabled (by setting a high pursuit velocity threshold) for data from static images, if the occurrence of pursuit events can be ruled out a priori. For this evaluation, however, no such intervention was made.
In order to further rank the performance of the proposed algorithm with respect to the ten algorithms studied in Andersson et al. (2017), we followed their approach to compute root mean square deviations (RMSD) from human labels for event duration distribution characteristics (mean and standard deviation of durations, plus number of events) for each stimulus category (images, dots, videos) and event type (fixations, saccades, PSOs, pursuits). This measure represents a scalar distribution dissimilarity score that can be used as an additional comparison metric of algorithm performance that focuses on overall number and durations of detected events, instead of sample-by-sample misclassification. The RMSD measure has a lower bound of 0.0 (identical to the average of both human raters), with higher values indicating larger differences (for detail information on the calculation of this metric see Andersson et al., 2017). Table 3 reproduces Andersson et al. (2017, Tables 3-6), and the RMSD calculation for the added rows on REMoDNaV is based on the scores for the human raters given in these original tables. As acknowledged by the authors, the absolute value of the RMSD scores is not informative due to scaling with respect to the respective maximum value of each characteristic. Therefore, we converted RMSDs for each algorithm and event type into zero-based ranks (lower is more human-like).
The LNS algorithm (Larsson et al., 2013) was found to have the most human-like performance for saccade and PSO detection in Andersson et al. (2017). REMoDNaV performs comparable to LNS for both event types (saccades: 2.0 vs. 3.3; PSOs: 2.3 vs. 2.0, mean rank across stimulus categories for LNS and REMoDNaV, respectively).
Depending on the stimulus type, different algorithms performed best for fixation detection. NH performed best for images and videos, but worst for moving dots. REMoDNaV outperforms all other algorithms in the dots category, and achieves rank 5 and 6 (middle range) for videos and images, respectively. Across all stimulus and event categories, REMoDNaV achieves a mean ranking of 2.9, and a mean ranking of 3.2 when not taking smooth pursuit  Andersson et al., 2017) and the REMoDNaV algorithm (AL) for gaze recordings from stimulation with static images (left column), moving dots (middle column), and video clips (right column). All matrices present gaze sample based Jaccard indices (JI; Jaccard, 1901). Consequently, the diagonals depict the fraction of time points labeled congruently by both raters in relation to the number of timepoints assigned to a particular event category by any rater.

9
. CC-BY 4.0 International license It is made available under a (which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
Taken together, REMoDNaV yields classification results that are, on average, more human-like than any other algorithm tested on the dataset and metrics put forth by Andersson et al. (2017). In particular, its performance largely equals or exceeds that of the original NH algorithm. NH outperforms it only for fixation detection in the image and video category, but in these categories REMoDNaV also classifies comparatively well. These results are an indication that the changes to the NH algorithm proposed here to improve upon its robustness are not detrimental to its performance on data from conventional paradigms and stimuli.

Prolonged natural viewing
Given that REMoDNaV yielded plausible results for the "video" stimulus category data in the Andersson et al. (2017) dataset ( Figure 2, and Table 3, right columns), we determined whether it is capable of analyzing data from dynamic stimulation without a trial structure.
As a test dataset we used publicly available eye tracking data from the studyforrest.org project, where 15 participants were recorded watching a feature-length (≈2 h) movie in a laboratory setting (Hanke et al., 2016). Eye movements were measured by an Eyelink 1000 with a standard desktop mount (software version 4.51; SR Research Ltd., Mississauga, Ontario, Canada) and a sampling rate of 1000 Hz. The movie stimulus was presented on a 522 × 294 mm LCD monitor at a resolution of 1920×1280 px and a viewing distance of 85 cm. Participants watched the movie in eight approximately 15 min long segments, with measurement recalibration before every segment.
As no manual eye movement event labeling exists for these data, algorithm evaluation was limited to a comparison of marginal distributions and well-known properties, such as the log-log-linear relationship of saccade amplitude and saccade peak velocity (Bahill et al., 1975). Figure 3 (top row) depicts this main sequence relationship. Additionally, Figure 4 (top row) shows duration histograms for all four event types across all participants. Shapes and locations of these distributions match previous reports in the literature, such as a strong bias towards short (less than 500 ms) fixations for dynamic stimuli (Dorr et al., 2010, Fig. 3), peak number of PSOs with durations between 10-20 ms (Nyström and Holmqvist, 2010, Fig. 11), and a non-Gaussian saccade duration distribution located below 100 ms (Nyström and Holmqvist, 2010, Fig. 8, albeit for static scene perception).
Overall, the presented summary statistics suggest that REMoDNaV is capable of detecting eye movements with plausible characteristics, in prolonged gaze recordings without a trial structure. A visualization of such a detection result is depicted in Figure 5 (top row).

Lower-quality data
An explicit goal for REMoDNaV development was robust performance on lower-quality data. While lack of quality cannot be arbitrarily compensated and will inevitably lead to misses in eye movement detection, it is beneficial for any further analysis if operation on noisy data does not introduce unexpected event property biases.

Images
The copyright holder for this preprint .   N = 15), and the entire duration of the same feature-length movie stimulus. All histograms depict absolute number of events. Visible differences are limited to an overall lower number of events, and fewer long saccades for the MRI sample. These are attributable to a higher noise level and more signal loss (compare Hanke et al., 2016, Fig. 4b) in the MRI sample, and to stimulus size differences (23.75°MRI vs. 34°lab). Figure 5: Exemplary eye movement detection results for the same 10 s excerpt of a movie stimulus for a single participant in the high quality lab sample (top), and in the lower quality MRI sample (bottom). The plots show filtered gaze coordinates (black), computed velocity time series (gray) overlayed on the eye movement event segmentation with periods of fixation (green), pursuit (beige), saccades (blue), and high/low-velocity post-saccadic oscillations (dark/light purple). The variable noise level, and prolonged signal loss (white) visible in the MRI sample represent a challenge for algorithms. REMoDNaV uses an adaptive approach that determines major saccade events first, and subsequently tunes the velocity threshold to short time windows between these events. Figures like this accompany the program output to facilitate quality control and discovery of inappropriate preprocessing and detection parameterization.
We performed the identical analysis as before, in order to compare performance between a high and lower-quality data acquisition. Figures 3-5 depict the results for the lab-quality dataset, and the MRIscanner dataset in the top and bottom rows, respectively.
Overall, the detection results exhibit strong similarity, despite the potential behavioral impact of watching a movie while lying on their back and looking upwards on the participants, or the well known effect of increasing fatigue during a two-hour session in an MRI-scanner. In particular, saccade amplitude and peak velocity exhibit a clear mainsequence relationship that resembles that found for the lab acquisition ( Figure 3). Duration distributions for fixations, PSOs, and pursuits are strikingly similar between the two datasets (Figure 4), except for a generally lower number of detected events for the MRI experiment, which could be explained by the higher noise level and fraction of signal loss. There is a notable difference regarding the saccade duration distributions, with a bias towards shorter saccades in the MRI dataset. This effect may be attributable to the differences in stimulus size (30% smaller in the MRI environment).

Conclusion
Based on the adaptive, velocity-based algorithm for fixation, saccade, and PSO detection by Nyström and Holmqvist (2010), we have developed an improved algorithm that, in contrast to the original, performs robustly on prolonged recordings with dynamic stimulation, without a trial structure and variable noise levels, and also supports the detection of smooth pursuit events. Through a series of validation analyses we have shown that its performance is comparable to or better than ten other contemporary detection algorithms, and that plausible detection results are achieved on high and lower quality data. These aspects of algorithm capabilities and performance suggest that REMoDNaV is a stateof-the-art tool for eye movement detection with particular relevance for emerging complex, naturalistic data collections paradigms, such as mobile or outdoor aquisition, or the combination of eye tracking and functional MRI in simultaneous measurements.
The proposed algorithm is rule-based, hence can be applied to data without prior training, apart from the adaptive estimation of velocity thresholds. This aspect distinguishes it from other recent developments based on deep neural networks (Startsev et al., 2018), and machine-learning in general (Zem-blys et al., 2018). Such algorithms tend to require substantial amount of (labeled) training data, which can be a critical limitation in the context of a research study. However, in its present form REMoD-NaV cannot be used for real-time data analysis, as its approach for time series chunking is based on an initial sorting of major saccade events across the entire time series.
Just as Andersson et al. (2017), we considered human raters as a gold standard reference for event detection when evaluating algorithms. The implications of the results presented herein are hence only valid if this assumption is warranted. Some authors voice concerns (e.g., Komogortsev et al., 2010), regarding potential biases that may limit generalizability. Nevertheless, human-made event labels are a critical component of algorithm validation, as pointed out by Hooge et al. (2018).
REMoDNaV aims to be a readily usable tool, available as cross platform compatible, free and open source software, with a simple command line interface and carefully chosen default settings. However, as evident from numerous algorithm evaluations (e.g., Andersson et al., 2017;Larsson et al., 2013;Zemblys et al., 2018;Komogortsev et al., 2010) different underlying stimulation, and data characteristics can make certain algorithms or parameterizations more suitable than others for particular applications. The provided implementation of the REMoDNaV algorithm (Hanke et al., 2019) acknowledges this fact by exposing a range of parameters through its user interface that can be altered in order to tune the detection for a particular use case.
The latest version of REMoDNaV can be installed from PyPi 2 via pip install remodnav. The source code of the software can be found on Github 3 . All reports on defects and enhancement can be submitted there. The analysis code underlying all results and figures presented in this paper, as well as the L A T E X sources, are located in another Github repository 4 . All required input data, from Andersson et al. (2017) and the studyforrest.org project, are referenced in this repository at precise versions as DataLad 5 subdatasets, and can be ob-