CancellationTools: All-in-one software for administration and analysis of cancellation tasks
In a cancellation task, a participant is required to search for and cross out (“cancel”) targets, which are usually embedded among distractor stimuli. The number of cancelled targets and their location can be used to diagnose the neglect syndrome after stroke. In addition, the organization of search provides a potentially useful way to measure executive control over multitarget search. Although many useful cancellation measures have been introduced, most fail to make their way into research studies and clinical practice due to the practical difficulty of acquiring such parameters from traditional pen-and-paper measures. Here we present new, open-source software that is freely available to all. It allows researchers and clinicians to flexibly administer computerized cancellation tasks using stimuli of their choice, and to directly analyze the data in a convenient manner. The automated analysis suite provides output that includes almost all of the currently existing measures, as well as several new ones introduced here. All tasks can be performed using either a computer mouse or a touchscreen as an input device, and an online version of the task runtime is available for tablet devices. A summary of the results is produced in a single A4-sized PDF document, including high quality data visualizations. For research purposes, batch analysis of large datasets is possible. In sum, CancellationTools allows users to employ a flexible, computerized cancellation task, which provides extensive benefits and ease of use.
KeywordsSpatial attention Visual search Neglect syndrome Cancellation tasks Computerized testing
Almost half of all stroke patients initially suffer from impaired attention (Lesniak et al., 2008). One of the most severe stroke-induced attention deficits is hemispatial neglect, a syndrome where patients disregard what happens towards contralesional space. It occurs in 25–50 % of stroke victims (Appelros et al., 2002; Buxbaum et al., 2004; Nijboer et al., 2013a), predominantly after damage to the right hemisphere (Ringman et al., 2004). Stroke patients suffering from neglect are hospitalized longer and face profound problems in daily life (Nijboer et al., 2013b; Nys et al., 2005). Although spontaneous recovery occurs, about 30–40 % of individuals with neglect still suffer from the syndrome after a year (Nijboer et al., 2013a, b). Importantly, neglect is associated with many negative factors, for example it appears to have a suppressive effect on upper-limb motor recovery (both synergism and strength) especially over the first ten weeks post-stroke (Nijboer et al., 2014).
Because of its severity, it is important that good tools are available to diagnose the neglect syndrome, and to support research on potential rehabilitation methods. One type of test that is widely used for assessment measures multitarget visual search. Such cancellation tasks require participants to cross out (“cancel”) all stimuli of a certain type, often while ignoring stimuli of all other types (distractors). These search tasks have gained immense popularity in cognitive neuropsychology, and have proven their worth both in clinical and research environments.
Cancellation performance is not only a measure of interest in patient groups, but in other sets of participants as well. For example, a recent study on a wide age range of healthy adults described search patterns on cancellation tasks in a qualitative manner (e.g., “horizontal left-to-right”), and concluded that no significant differences exist between different age groups (Warren et al., 2008). However, this investigation lacked more sensitive measures of search organization that have been shown to improve with age in children (Woods et al., 2013). Healthy elderly people tested two years before dementia require significantly more time to complete a cancellation task than elderly individuals who did not develop dementia (Fabrigoule et al., 1998). Differences in performance within demented patients became apparent when tests of a higher attentional load were deployed: patients with Alzheimer's disease performed as accurately as patients with multi-infarct dementia on a low-load cancellation task, but were both less accurate and faster on a cancellation task that required more selective and divided attention (Gainotti et al., 2001). Principal component analysis of a range of neuropsychological tests, including cancellation, indicates there might be a common factor underlying performance deterioration for in the pre-clinical stage of Alzheimer's disease, perhaps associated with a general ability to control cognitive processes (Fabrigoule et al., 1998).
All of the findings summarized above could profitably be extended with more sensitive measures of cancellation performance and search organisation. When diagnosing neglect, the primary measures of cancellation tasks are usually the amount and spatial spread of omissions (non-cancelled targets). However, there is emerging evidence that the neglect syndrome constitutes more than just lateralized deficits (Husain & Rorden, 2003), and deficits of spatial working memory or sustained attention might contribute, for which additional indices of cancellation performance might be helpful.
Numerous measures of general performance, timing, and search strategy that can be derived from cancellation tasks have been suggested in the literature (for an overview, see the section Supported Measures). However, data collection for these measures is often performed using labor-intensive and perhaps suboptimal procedures, e.g., frame-by-frame video analysis (Mark et al., 2004; Woods & Mark, 2007), monitoring of “verbal cancellation” (Samuelsson et al., 2002), “observing and recording the predominant search pattern” during a task by a human observer (Warren et al., 2008), or asking patients to change the color of their pencil every 10–15 cancellations (Weintraub & Mesulam, 1988). A more efficient way of analyzing search patterns would be to use a computerized cancellation task, with which cancellation positions and order can be recorded without the risk of human error.
Although the first reports of computerized cancellation software date back 15 years (Donnelly et al., 1999), the currently available packages are very limited in either the number of supported tasks (Donnelly et al., 1999; Wang et al., 2006), or the supported measures (Rorden & Karnath, 2010; Wang et al., 2006), and none of them provide both task presentation and data analysis (CACTS by Wang et al. is reported to be able to do both, but is not available for download). Therefore, most laboratories use custom software and most clinicians still prefer pen-and-paper tests.
Due to the lack of practically useful software, the field is currently in a situation in which ample theoretically valid measures exists (Donnelly et al., 1999; Hills & Geldmacher, 1998; Malhotra et al., 2006; Mark et al., 2004; Rorden & Karnath, 2010; Samuelsson et al., 2002; Warren et al., 2008; Weintraub & Mesulam, 1988), of which most are validated on a small scale in research studies, but very few can be applied on a large scale in clinical practice or research studies due to the aforementioned practical issues.
In the current paper, we present a potential solution: CancellationTools, a package that combines the administration and the analysis of cancellation tasks, supporting almost all types of cancellation tests, and outputting almost all of the currently available research measures. The software is designed to be as user-friendly as possible, by using a very straightforward interface, and the option to import a scanned task that allows users to use their preferred cancellation task type. Additionally, CancellationTools supports touchscreen input, which is very comparable to pen-and-paper cancellation, for example in the sense that it allows bedside testing. Our package is open source, and is available to download for free. An online version of the task software is available to provide support for tablet devices.
CancellationTools has been written completely in Python (Van Rossum & Drake, 2011), using as few dependencies as possible. The graphical user interface (GUI) has been written from scratch using the PyGame toolbox, and the software to analyze and visualize data has been written using the NumPy (Oliphant, 2007) and Matplotlib (Hunter, 2007) packages. All of these are open-source projects that are maintained by a large community of volunteers.
The software can be downloaded for free from www.cancellationtools.org>. It is released under the GNU General Public License version 3 (Free Software Foundation, 2007), which ensures that it can be used, shared, and modified by anyone. The source code is publicly available and managed via GitHub, which stimulates programming with frequent feedback, version control, and collaboration on a large scale – all according to the best practices for scientific computing as formulated by Wilson et al. (2014).
A simplified version of the application can be used online. Due to copyright issues, we cannot allow users to upload their own tasks to our website. We do provide different versions of the Landolt C cancellation task. After online completion of a task, a raw data file can be downloaded, which can later be analyzed via the offline version. No data will be permanently stored or accessed by the authors of CancellationTools, or any third party. An advantage of the online runtime is that it can be accessed from computers that do not allow installation of new software (e.g., in most hospitals), or via tablet devices (e.g., Apple's iPad) that are gaining increasing popularity in neuropsychological testing.
Currently, the standalone version of CancellationTools is only available on Windows. Users of other operating systems can choose between running the application from source via Python, or using the online runtime to test participants and a PC for data analysis. We are currently working on standalone versions for other platforms, e.g., Macintosh OS X and Android, and will release these in the future.
Landolt C cancellation task
Importing scanned tasks
For researchers and clinicians who prefer to work with a different cancellation task, CancellationTools has an option to import scanned tasks. If users select this option, they are asked to provide an image file. The image is automatically scaled to the display resolution, and a user can proceed to manually indicate where the targets and distractors are. The task is then saved, and is available for future use in task administration and analysis.
We have attempted to include all of the currently existing measures that can be derived from cancellation tasks, which can be broadly divided into three categories: measures of biases in spatial attention, of search organization, and of general performance. Furthermore, to complement or improve on existing measures, we have devised a few of our own (e.g., the standardized angle, see below). We have not included qualitative descriptions of cancellation path structure (Samuelsson et al., 2002; Warren et al., 2008; Weintraub & Mesulam, 1988), or an algorithm to categorize search organization (Huang & Wang, 2008). In our view, these do not provide much further insight into cancellation performance than the included qualitative measures and visualizations.
CancellationTools reports the total number of omissions and the omissions per half of the search array, which have traditionally been used to diagnose neglect. These values are to be interpreted using standardized scores, depending on what task is employed. Traditionally, a relatively large number of omissions has been used as one index of neglect, but the left:right omissions ratio is potentially more informative and has been used widely. For example, a recent study on a particularly large sample (55 neglect patients, 138 non-neglect patients, and 119 controls) by Rabuffetti et al. (2012) reported that neglect patients show a large directional (left vs. right) imbalance in omissions, compared to healthy controls and patients with left or right lesions without neglect.
A revisit is a cancellation of a previously cancelled target. Some authors refer to this kind of response in the cancellation literature as 'perseveration'. However, perseverations are often used as a term associated with a (frontal) lack of ability to inhibit. In neglect research, there is evidence that while some patients might have a problem with the ability to inhibit re-cancelling a previously visited item, others re-cancel because of a deficit in spatial working memory (Mannan et al., 2005). Therefore, we prefer to use the empirically descriptive term 'revisit'.
Revisits can occur immediately, when a participant cancels the same target twice in a row – analogous perhaps to perseveration. A delayed revisit occurs when a participant goes back to a previously cancelled target, after cancelling other targets (Mannan et al., 2005). The number of revisits correlates with measures of disorganized search, such as the best R (see below), the inter-cancellation distance, and the number of cancellation path intersections (Mark et al., 2004). Parton et al. (2006) reported that neglect patients demonstrated a higher number of revisits than non-neglect patients, an effect that was especially apparent when no cancellation marks were visible, i.e., when patients had to remember which targets they had previously visited. In this touch screen study, the median number of intervening targets was 8. The authors argued that a possible underlying mechanism for such revisiting behaviour might therefore be a deficit in spatial working memory. Our software provides the option of using an invisible cancellation condition, should users wish to use this type of search display which can provide a more sensitive measure of left:right biases in neglect, and allows investigation of the role of spatial working memory in cancellation tasks (Wojciulik et al., 2004).
Standardized inter-cancellation distance
Center of cancellation
The center of cancellation (CoC), introduced by Binder et al. (1992) and popularized by Rorden and Karnath (2010), is the average horizontal position of all cancelled targets, standardized so that a value of −1 corresponds with the leftmost, and 1 with the rightmost target. The CoC is a very elegant measure of neglect severity, as it captures an attentional gradient rather than a bimodal decision (i.e., left field is or is not impaired). Additional to the horizontal CoC., CancellationTools provides the vertical CoC, where −1 corresponds with the topmost target, and 1 with the target that is closest to the bottom of the task.
The total amount of time a participant spends on a cancellation task might be an indication of the participant's sustained attention for the task. Primary reports indicate that this measure is potentially influenced by pharmacological intervention (Malhotra et al., 2006), and could therefore be used in diagnostics and rehabilitation. The average inter-cancellation time (sometimes dubbed latency index) differs between healthy controls and brain-damaged patients, but also between neglect and non-neglect patients (Rabuffetti et al., 2012). It could hypothetically serve as a measure of executive functioning, as it reflects how much processing time a participant needs to find and cancel a new target.
n is the number of cancellations
s is the distance between two consecutive cancellations
t is the time between two consecutive cancellations
Quality of search (Q) score
Ncor is the number of cancelled targets (correct responses)
Ntar is the total number of targets
ttot is the total time spent on the task
(X1 , y1) is the starting coordinate of the line between two consecutive cancellations (cancellation n)
(X2 , y2) is the ending coordinate of the line between two consecutive cancellations (cancellation n+1)
(Px , Py) is the coordinate of the intersection between two inter-cancellation lines
n is the number of inter-cancellation lines (not to be confused with the number of cancellations)
Rhor is the Pearson correlation coefficient of the horizontal position of all cancellations and their rank numbers
Rver is the Pearson correlation coefficient of the vertical position of all cancellations and their rank numbers
γ is the angle between two consecutive cancellations
Δy is the vertical distance between two consecutive cancellations
d is the Euclidean distance between two consecutive cancellations
n is the total amount of inter-cancellation angles between consecutive cancellations that are not immediate revisits
Age has a significant influence on measures of search organisation. Specifically, the mean inter-cancellation distance and the amount of intersections decrease as age increases in children, while the best R increases, demonstrating an improvement in search organisation over time (Woods et al., 2013). Another index that increases with age is the likelihood of the first cancellation to be in the top-left quadrant of the search array. CancellationTools provides the location of the first marking in standardized space, so that the top left of the search array is (0,0) and the bottom right (1,1). These standardized locations are comparable between different task types and sizes. A qualitative description (e.g., “top-left”) of the quadrant in which the first cancellation happened is also available.
Averages, standard deviations (between round brackets), and 95 % confidence intervals (between square brackets) of a healthy sample and a neglect patient sample, collected using 1280×1024 pixels Landolt C cancellation tasks with 64 targets and 128 distractors, invisible cancellation markings, and a time limit of 2 min
Healthy sample (N=10)
Neglect sample (N=10)
Omissions in left half
Omissions in right half
Horizontal center of cancellation
Vertical center of cancellation
Mean inter-cancellation time
Mean inter-cancellation distance
Standardized inter-cancellation distance
(cancellations per second)
Mean inter-cancellation angle
Standardized inter-cancellation angle
First cancellation x-coordinate
First cancellation y-coordinate
Theoretically task-independent measures (provided there is a relatively equal spread of targets over the search array) are left:right omission ratio, standardized inter-cancellation distance, center of cancellation, average inter-cancellation speed, intersections rate, and location of the first cancellation in standardized space. Whether this theoretical task-independency holds up in practice, might be determined in future research.
Several plots are created by each CancellationTools analysis. These give further insight into the performance of single participants, and can be used in addition to the measures described above. These plots include the aforementioned cancellation path and heatmap. The cancellation path (Fig. 7a-b) gives a clear view of a participant's cancellation behavior, e.g., to help with the interpretation of measures of disorganized search. A plot of the relation between the cancellation rank number and either the horizontal or vertical position of the cancelled target (Fig. 5d-f) gives an indication of how organized a participants search was (Mark et al., 2004; Woods & Mark, 2007).
For the cancellation and omission a Gaussian kernel is added to the location of each cancelled or missed target. The resulting field is then scaled to the heatmap that would result from an optimal performance on the cancellation task in question, which means that heatmaps are comparable between individuals and tasks. Heatmaps for individual data from a healthy individual and a neglect patient are displayed in Fig. 7c-d. Averaged heatmaps of a healthy and a neglect sample are shown in Fig. 8, and show an even spread of cancellations across the search array in healthy people (Fig. 8a), whereas neglect patients show a rightward bias (Fig. 8b). Neglect patients also display a leftward bias of omissions (Fig. 8d), whereas our healthy sample shows a lack of omissions (Fig. 8c).
There is a need to quantify multitarget visual search performance on cancellation tasks. We made an effort to summarize all of the currently available measures that can be derived from cancellation task data. In the new software introduced here, we included all relevant measures from the currently available literature in an application that can be used to administer a computerized cancellation task, and to analyze the resulting data with the click of a button. We have aimed to make this software as flexible as possible, e.g., by allowing users to incorporate their own scanned tasks into the software, whilst keeping an eye on simplicity. The result is a user-friendly interface that can be employed both in clinical and research settings. Our software is open source, and free to download and use by anyone.
We have introduced two new measures of search organisation: the standardized inter-cancellation distance and angle. The former is an improvement of the existing mean inter-cancellation distance, which takes into account the distances between targets within a search array, therefore allowing comparisons of cancellation performance on different tasks. The standardized inter-cancellation angle can be viewed as complimentary to the best R, as it is robust to situations where the best R does not reflect search organisation optimally (Fig. 5c). Even though the best R and standardized inter-cancellation angle seem to differentiate between our small test groups, a much larger difference between healthy people and leftward neglect patients is observed in the intersections rate, suggesting that this might be the clearest measure of search organisation.
CancellationTools is already useful to clinicians, as it provides quantitative data on established measures of neglect (e.g., number of omissions), as well as qualitative data that provides better insight in patient behavior than pen-and-paper cancellation tests (e.g., cancellation path plots). However, for the majority of the measures summarized above, there are currently no norm scores to compare individual test results to. The value ranges that we provide based on our pilot testing (Table 1) serve as a preliminary indication of how neglect patients and healthy controls differ on different measures, and should not be treated as a clinical directive.
Apart from our newly introduced standardized angle measure, all of the indices we report have been validated on a small scale in the articles in which they were coined. A few have been validated on a larger scale in the study of Rabuffetti et al. (2012), but it is arguable whether this provides enough data to base norm scores on. We aim to facilitate the fast testing of the summarized measures by providing a unified tool that helps to gather cancellation task data as easy as possible. Our hope is that this will help to establish norm scores for the measures that proof to have diagnostic value.
By making CancellationTools publicly available, we hope to inspire large-scale international collaborations to pool data, from healthy people and patient groups, on all of the measures we summarize in the current article. By removing practical boundaries that previously prevented large-scale testing, our software opens up exciting new research possibilities. The availability of CancellationTools creates a situation in which analysis of cancellation task data can be performed at a high level across different clinical and research settings.
E.S.D. is supported through a European Union FP7 Marie Curie ITN grant (606901). M.H. is supported by the Welcome Trust. T.C.W.N. is supported by the NWO (Netherlands organization for Scientific Research; grant 451-10-013).
- Bays, P. M., Singh-Curry, V., Gorgoraptis, N., Driver, J., & Husain, M. (2010). Integration of Goal- and Stimulus-Related Visual Signals Revealed by Damage to Human Parietal Cortex. Journal of Neuroscience, 30(17), 5968–5978. doi:10.1523/JNEUROSCI.0997-10.2010 PubMedCentralCrossRefPubMedGoogle Scholar
- Buxbaum, L. J., Ferraro, M. K., Veramonti, T., Farne, A., Whyte, J., Ladavas, E., … Coslett, H. B. (2004). Hemispatial neglect: Subtypes, neuroanatomy, and disability. Neurology, 62(5), 749–756. doi:10.1212/01.WNL.0000113730.73031.F4
- Fabrigoule, C., Rouch, I., Taberly, A., Letenneur, L., Commenges, D., Mazaux, J. M., … Dartigues, J. F. (1998). Cognitive process in preclinical phase of dementia. Brain, 121(1), 135–141. doi:10.1093/brain/121.1.135
- Free Software Foundation. (2007). GNU General Public License. Retrieved from https://gnu.org/licenses/gpl.html
- Mannan, S. K., Mort, D. J., Hodgson, T. L., Driver, J., Kennard, C., & Husain, M. (2005). Revisiting Previously Searched Locations in Visual Neglect: Role of Right Parietal and Frontal Lesions in Misjudging Old Locations as New. Journal of Cognitive Neuroscience, 17(2), 340–354. doi:10.1162/0898929053124983 CrossRefPubMedGoogle Scholar
- Nijboer, T. C. W., Van de Port, I., Schepers, V., Post, M., & Visser-Meily, A. (2013b). Predicting Functional Outcome after Stroke: The Influence of Neglect on Basic Activities in Daily Living. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00182
- Rabuffetti, M., Farina, E., Alberoni, M., Pellegatta, D., Appollonio, I., Affanni, P., … Ferrarin, M. (2012). Spatio-Temporal Features of Visual Exploration in Unilaterally Brain-Damaged Subjects with or without Neglect: Results from a Touchscreen Test. PLoS ONE, 7(2), e31511. doi:10.1371/journal.pone.0031511
- Samuelsson, H., Hjelmquist, E. K. E., Jensen, C., & Blomstrand, C. (2002). Search pattern in a verbally reported visual scanning test in patients showing spatial neglect. Journal of the International Neuropsychological Society, 8(03), 382–394. doi:10.1017/S1355617702813194 CrossRefPubMedGoogle Scholar
- Van Rossum, G., & Drake, F. L. (2011). Python Language reference manual. Bristol, UK: Network Theory Ltd.Google Scholar
- Wilson, G., Aruliah, D. A., Brown, C. T., Chue Hong, N. P., Davis, M., Guy, R. T., … Wilson, P. (2014). Best Practices for Scientific Computing. PLoS Biology, 12(1), e1001745. doi:10.1371/journal.pbio.1001745
- Wojciulik, E., Rorden, C., Clarke, K., Husain, M., & Driver, J. (2004). Group study of an “undercover” test for visuospatial neglect: invisible cancellation can reveal more neglect than standard cancellation. Journal of Neurology, Neurosurgery & Psychiatry, 75(9), 1356–1358. doi:10.1136/jnnp.2003.021931 CrossRefGoogle Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.