Alternative-based thresholding with application to presurgical fMRI

Abstract

Functional magnetic reasonance imaging (fMRI) plays an important role in pre-surgical planning for patients with resectable brain lesions such as tumors. With appropriately designed tasks, the results of fMRI studies can guide resection, thereby preserving vital brain tissue. The mass univariate approach to fMRI data analysis consists of performing a statistical test in each voxel, which is used to classify voxels as either active or inactive—that is, related, or not, to the task of interest. In cognitive neuroscience, the focus is on controlling the rate of false positives while accounting for the severe multiple testing problem of searching the brain for activations. However, stringent control of false positives is accompanied by a risk of false negatives, which can be detrimental, particularly in clinical settings where false negatives may lead to surgical resection of vital brain tissue. Consequently, for clinical applications, we argue for a testing procedure with a stronger focus on preventing false negatives. We present a thresholding procedure that incorporates information on false positives and false negatives. We combine two measures of significance for each voxel: a classical p-value, which reflects evidence against the null hypothesis of no activation, and an alternative p-value, which reflects evidence against activation of a prespecified size. This results in a layered statistical map for the brain. One layer marks voxels exhibiting strong evidence against the traditional null hypothesis, while a second layer marks voxels where activation cannot be confidently excluded. The third layer marks voxels where the presence of activation can be rejected.

Introduction

A common treatment for patients suffering from a brain tumor is surgical resection of the tumor. In order to minimize the risk of resecting brain tissue involved in essential brain functions, such as speech or language comprehension, these patients often undergo presurgical functional magnetic resonance imaging (fMRI). This is a technique that shows subject-specific neural activity changes in the brain. The resulting fMRI data can assist the surgeon in performing the tumor resection while preserving the brain tissue involved in important cognitive and sensorimotor functions (Bartsch, Homola, Biller, Solymosi, & Bendszus, 2006) and can even be used to predict the outcome of postoperative cognitive functioning (Richardson et al., 2004).

To analyze fMRI data, a huge number of statistical tests are performed simultaneously. In cognitive neuroscience, this technique is used to link neurological and neuropsychological functions with their respective location in the brain, supporting different theories of brain function. To be confident that a brain area is associated with a task, it is essential to account for the multiple testing problem. This can be done using corrections for either the familywise error rate (Friston, Frith, Liddle, & Frackowiak, 1991; Worsley et al., 1996) or the false discovery rate (Genovese, Lazar, & Nichols, 2002). These multiple testing corrections result in a more stringent control of the null hypothesis of no activation, and consequently, the probability of a false negative increases (Lieberman & Cunningham, 2009; Logan & Rowe, 2004). In cognitive neuroscience, a false positive means fallacious support for a given cognitive theory. While false positives can often be discovered by unsuccessfully trying to replicate the study, much time, effort, and money can be expended. As a result, the scientific discipline generally deems stringent control of false positives necessary, accepting the concomitant sacrifices in sensitivity.

In a clinical setting such as presurgical fMRI, however, a loss in power means that true activation is not discovered, and this might result in the resection of vital brain tissue (Haller & Bartsch, 2009). Inversely, false positives have a less negative impact on the surgical result (Gorgolewski, Storkey, Bastin, & Pernet, 2012). The goal of classical hypothesis testing is to prevent the null hypothesis from being rejected by considering voxels as being active only when enough evidence against the null of no activation is found. This asymmetrical way of penalizing errors in statistical inference is undesirable in this context (Johnson, Liu, Bartsch, & Nichols, 2012), and instead, the focus should be on protecting the alternative hypothesis: one wants to exclude activation only when enough evidence against activation is found. We therefore present a new hypothesis thresholding procedure that incorporates information on both false positives and false negatives and, thus, is ideally suited for presurgical fMRI.

In classical hypothesis testing, the evidence against null hypothesis is measured with the p-value, the null hypothesis probability of data as or more extreme than that observed. Thresholding a p-value at α produces a statistical test that controls the false positive rate at α. To allow direct control of false negative risk, we present a symmetrical measure that quantifies evidence against the alternative hypothesis (Moerkerke, Goetghebeur, De Riek, & Roldan-Ruiz, 2006). Correspondingly, thresholding this probability measure at β ensures control of the false negative rate at β.

By combining thresholds on the classical and alternative p-values, we use information on the probability of false positives and false negatives. We show that thresholding both error measurements results in a layered statistical map for the brain, each layer marking voxels with evidence (or lack thereof) against the null and/or alternative hypothesis. One layer consists of voxels exhibiting strong evidence against the null of no activation, while a second layer is formed by voxels for which activation cannot be confidently excluded. The third level then consists of voxels for which the presence of activation can be rejected.

fMRI data can be analyzed in different ways. The most popular method is a confirmatory mass-univariate general linear model (GLM) analysis, where the measured time series in each voxel is regressed onto the design of the experiment, resulting in an estimate of the effect, for which a T-statistic with a corresponding classical p-value can be computed for each voxel. This method has been shown to be very effective and robust, but its downside is the mass-univariate character. While many attempts have been made to take into account the spatial character of the data with data smoothing and peak- and cluster-thresholding, the GLM fails to recognize patterns of activation or noise. In this light, statistical techniques for multivariate data have been successfully applied to fMRI data. Independent component analysis (ICA; Beckmann & Smith, 2004) is an exploratory method used to find hidden source signals, modeling the observed data as a (unobserved) linear mixture of (unobserved) sources. ICA therefore allows one to discover spatially and temporally structured noise. Given the popularity of the GLM and the upcoming interest for ICA, especially in a clinical context, we will introduce the thresholding procedure for both techniques. We show how the ideas can be translated to different statistical techniques.

In the Method section, we introduce and combine quantities to measure significance when testing for activation. To this end, we start with a simple setting in which test statistics are assumed to be Gaussian distributed and take the general form of the ratio of an observed effect and its standard error. These settings directly translate to the case of univariate linear modeling that makes use of T-distributions. We further demonstrate how to use the principle for ICA. In the Results section, we present the results of the procedure applied to presurgical fMRI data.

Method

Measures of evidence against the null and alternative

At each voxel i, i = 1, …, I, we assume that a linear model is fit and produces ${\widehat{\varDelta}}_i$, an unbiased estimate of the BOLD effect of interest Δ _i , and an estimate of the standard deviation of ${\widehat{\varDelta}}_i$, its “standard error” $\mathrm{SE}\left({\widehat{\varDelta}}_i\right)$. We henceforth suppress the voxel subscript unless needed for clarity. We assume that the degrees of freedom are sufficiently large so that $\mathrm{SE}\left(\widehat{\varDelta}\right)$ has negligible variability, as is the case for fMRI time series. We further assume that the data, model, and contrast have been scaled appropriately so that $\widehat{\varDelta}$ has units of percent BOLD change (or at least approximately, as when global brain intensity is scaled to 100¹).

The null and the alternative hypotheses

The null hypothesis H ₀:Δ = 0 states that the true effect magnitude is zero and an underlying difference between conditions Δ is equal to 0. Classical statistical inference involves computing a test statistic, converted to a p-value, that measures the evidence against this null hypothesis. The decision procedure to reject H ₀ is calibrated to maintain the type I error at α. However, failing to reject H ₀ does not allow one to conclude that H ₀ is true. The reason is that the probability calculation of the p-value is based on the assumption that the null hypothesis is true. It is a logical fallacy, “affirming the consequent” or “reasoning to a forgone conclusion,” to begin by assuming something and then, eventually, conclude that the initial thing is true. More concretely, when we fail to reject H ₀, it could simply be because there are only subtle deviations from H ₀ that are not detected or because the precision on the observed effect is too small to reach statistical significance. Scientists frequently make this mistake, and there have been various guidelines for reporting study results (see, e.g., Meehl, 1978; Schmidt & Hunter, 2002), all of which stress the importance of complementing p-values with effect sizes.

Our procedure considers an “alternative hypothesis” p-value, p ₁, that measures the evidence against H _a :Δ = Δ₁, the nonzero effect magnitude expected under activation. Often, fMRI studies are preceded by power analyses for sample size calculations, which also require the specification of Δ₁. In literature, different approaches to choosing a meaningful Δ₁ have been presented (Desmond & Glover, 2002; Hayasaka, Pfeiffer, Hugenschmidt, & Laurienti, 2007; Mumford & Nichols, 2008; Zarahn & Slifstein, 2001). Alternatively, in presurgical fMRI, one can estimate Δ₁ on the basis of data in previous patients.

Measures of significance

At a given voxel, we have a test statistic T with observed value

$$ t=\frac{\widehat{\varDelta}}{\mathrm{SE}\left(\widehat{\varDelta}\right)}. $$

(1)

We assume that T has a known distribution under H ₀ (e.g., Student’s t with given degrees of freedom or Gaussian), so that we can compute the classical p-value:

$$ {p}_0=P\left(T\ge t\left|{H}_0\right.\right). $$

(2)

That is, p ₀ quantifies the evidence against the null hypothesis H ₀ of no task-related activation.

In a symmetrical fashion, the alternative p-value is defined as in Moerkerke et al., 2006):

$$ {p}_1=P\left(T\le t\left|{H}_a\right.\right). $$

(3)

Correspondingly, p ₁ measures the evidence against H _a and corresponds to the classical p-value for testing a “null” H ₁ versus an “alternative” H ₀. In general, as the evidence in favor of H ₁ grows, p ₀ becomes smaller and p ₁ becomes larger.

In order to compute p ₁, we need the distribution of T under H _a , which requires specification of Δ₁. However, we expect not a single magnitude of true activation, but a distribution of different true values (Desmond & Glover, 2002). Therefore, in a Bayesian spirit, we specify a distribution of likely values of Δ₁ instead of a fixed value:

$$ {\varDelta}_1\sim \mathcal{N}\left(\mu, {\tau}^2\right), $$

(4)

where μ is the expected magnitude of effect under true activation while acknowledging variation among voxels—specifically, Gaussian variation with standard deviation τ.

Assuming that T also follows a Gaussian distribution, it has the following distribution under H _a at voxel i:

$$ {T}_i\sim \mathcal{N}\left(\frac{\mu }{\mathrm{SE}\left({{\displaystyle \widehat{\varDelta}}}_i\right)},\frac{\mathrm{SE}{\left({{\displaystyle \widehat{\varDelta}}}_i\right)}^2+{\tau}^2}{\mathrm{SE}{\left({{\displaystyle \widehat{\varDelta}}}_i\right)}^2}\right), $$

(5)

where voxel subscripts are used to emphasize that the values of μ and τ are fixed for the entire brain and based on prior knowledge or other experiments, while $\mathrm{SE}\left({\widehat{\varDelta}}_i\right)$ is from each individual voxel. With this distribution, we can compute p ₁ at each voxel. An illustration of both measures of significance can be seen in Fig. 1. Since the alternative distribution depends on the voxel-specific standard error, the distance between the null and alternative distributions will be voxel specific. In particular, a large standard error results in a large overlap between H ₀ and H _a , while small standard errors lead to a large distance and little overlap between H ₀ and H _a .

Fig. 1

The distributions of an effect under H ₀ and H _a are displayed for an observed effect of t = 1.5, $\mathrm{SE}\left(\widehat{\varDelta}\right)=1$, Δ₁ = 2, and τ = 1. Note that H _a has a wider distribution than H ₀ due to the uncertainty on Δ₁

Combining measures of significance

In classical null hypothesis significance testing, a threshold α on p ₀ can be translated into a threshold t _α for the test statistic in Eq. 1. In parallel, a threshold β on p ₁ can be translated into a test statistic threshold t _β . While t _α is determined by α (and degrees of freedom, if not using a Gaussian), t _β further depends on β, μ, τ, and $\mathrm{SE}\left({\widehat{\varDelta}}_i\right)$. Thus, t _β varies over the brain depending on the (estimated) standard error.

Figure 2 shows the possible results of this testing procedure, with α and β relatively small. In what is expected to be the typical scenario, with a standard error that is large relative to the true effect magnitude, t _β < t _α and three possible outcomes can be distinguished.

Fig. 2

When thresholding p ₀ and p ₁ at significance levels α and β, two possibilities arise: t _β < t _α (upper panel) or t _α < t _β (lower panel)

One outcome is when voxels exhibit evidence against H ₀ and, at the same time, are consistent with H _a (p ₀ < α and p ₁ > β; red in Fig. 2). This is the most compelling case for the presence of true activation (Δ > 0). The opposite outcome is a large p ₀ and a small p ₁ (p ₀ > α and p ₁ < β; gray in Fig. 2). Here, the data are consistent with the null, and there is evidence to reject the alternative; this is the most compelling case for true absence of activation (Δ = 0). The third outcome is when the data are compatible with both the null and the alternative and neither can be excluded (p ₀ > α and p ₁ > β; yellow in Fig. 2).

A less frequent, albeit possible scenario appears when the standard error is small relative to the true effect magnitude, t _α < t _β , and H ₀ and H _a can be clearly distinguished. Voxels with no effect or strong effects will be identified as before (p ₀ > α and p ₁ < β, no activation; p ₀ < α and p ₁ > β, activation). However, for certain data, there is evidence against both H ₀ and H _a (p ₀ < α and p ₁ < β; orange in Fig. 2). It indicates a case where the effect is so small as to lack practical significance.

For presurgical fMRI, this procedure provides information on which areas are confidently safe to be resected (gray areas), which areas should absolutely be avoided when resecting brain tissue (red areas), and in which areas the surgeon should take care because neither hypotheses can be rejected (yellow areas). When the fourth type of voxel is found, meaning both hypotheses can be rejected (orange areas), an abundance of caution suggests that again care be taken, since rejection of H ₀ does suggest some association with the task, just at a possibly very small magnitude. The specific application to real data is shown in the Results section.

Alternative thresholding of independent component analysis

Above, we described the classical and alternative p-value for a traditional setting, where a test statistic is the ratio of an observed effect and its standard error, as is the case for T-statistics when the GLM is used. Here, we demonstrate that the technique is also applicable in more general settings—in particular, with maps from independent component analysis. Exact implementation details of ICA methods differ; our development here follows the FSL² software’s implementation, MELODIC (Beckmann & Smith, 2004), but should be readily applicable to other ICA software.

ICA is a technique for multivariate data-driven analysis of fMRI data. It does not require the specification of the experimental design and produces spatiotemporal patterns that explain the variability in the data. ICA transforms the four-dimensional fMRI data into K pairs of spatial and temporal modes. Each spatial mode, or independent component (IC) image, is associated with one IC time series. The variation explained by each component is the IC time series scaled by the weights at each voxel in the IC image; equivalently, it is the spatial pattern in an IC image scaled by each value of the IC time series. Stated simply, the weights represent the association between the temporal activation pattern observed in the voxel and the temporal pattern in the K different components.

Let Y represent the J × I data matrix, where J is the number of time points and I is the number of voxels. We assume that the data at each voxel have been mean-centered; that is, the column means of Y are zero. ICA decomposes the data as per

$$ \boldsymbol{Y}\approx \boldsymbol{M}\kern0.5em {\boldsymbol{S}}_1\kern0.5em \boldsymbol{C}\kern0.5em {\boldsymbol{S}}_2{\boldsymbol{S}}_0, $$

(6)

where M is a J × K matrix with one temporal mode in each column and C is a K × I matrix with one spatial mode in each row; S ₁ (K × K) is a diagonal scaling matrix that ensures that the temporal modes have unit variance, and S ₀ and S ₂ (both I × I) are diagonal scaling matrices that ensure that background noise in the spatial modes have unit variance (see Appendix 1 for detailed definitions of these scaling factors).

In the presentation and interpretation of ICA results, each of the K spatial modes in C are visualized and explored. Since they have been noise-normalized, they are often treated as z-score images and thresholded to control a nominal false positive rate. The end result is an inference that quantifies the relation between the corresponding temporal mode in M and Y. We seek to apply our alternative hypothesis thresholding procedure to these maps, but first we need to define a meaningful effect size in percentage of BOLD change and transform this to the scale of C.

Meaningful BOLD effect sizes with ICA

Consider a particular IC of interest, k ∊ {1, …, K}, and a particular voxel i ∊ {1, …, I} of interest in the spatial mode. Specifically, consider the contribution of the kth IC to the time series at voxel i:

$$ {\boldsymbol{m}}_k\kern0.5em {s}_{1,k}\kern0.5em {c}_{ki}\kern0.5em {s}_{2,i}\kern0.5em {s}_{0,i}, $$

(7)

where m _k is the kth column of M, c _ki = (C) _ki , and s _1,k , s _2,i , and s _0,i are the indicated diagonal elements of the scaling matrices.

As was previously mentioned, the rows of C are normalized to have noise variance of 1, so c _ki has z-score (and not BOLD data) units. We need to compute a meaningful percentage of BOLD change effect. We will first compute this for a fixed Δ₁, in the units of c _ki , and will later impose a distribution on the effect size. Equation 7 shows that the temporal variation from IC k is determined not only by c _ki and the scaling factors, but also by m _k . But m _k is scaled to unit variance and will not induce a unit BOLD change in the data. We propose scaling m _k so that it (roughly) expresses a unit BOLD effect and, as a result, preserves the units of the other terms. Specifically, we introduce h _k :

$$ {\boldsymbol{m}}_k\kern0.5em {h}_k{h}_k^{-1}\kern0.5em {s}_{1,k}\kern0.5em {c}_{ki}\kern0.5em {s}_{2,i}\kern0.5em {s}_{0,i}., $$

(8)

so that m _k h _k expresses a unit BOLD effect in the data. One way to set the factor h _k is so that m _k h _k has a baseline-to-peak range of 1. Another way is to regress m _k on a covariate d that expresses the anticipated (unit) experimental effect; setting h _k to the inverse of the regression coefficient will ensure that m _k h _k corresponds to an approximate unit BOLD effect.

Finally, we correct for the attenuation of the hypothesized effect based on the mismatch between m _k and d. That is, even if we choose h _k well, m _k h _k may only be weakly correlated with d. As a result, we scale the expected BOLD effect of IC k at voxel i by ${\rho}_{{\boldsymbol{m}}_k\boldsymbol{d}}$, the correlation between m _k and d (see Appendix 2 for details).

Now we can relate the expected (attenuated) percentage of BOLD change, ${\rho}_{{\boldsymbol{m}}_k\boldsymbol{d}}{\varDelta}_1$, to the units of IC temporal mode. Let Δ ^*₁ be the expected alternative mean effect in the z-score statistic c _ki ; then,

$$ {\rho}_{{\boldsymbol{m}}_k\boldsymbol{d}}\kern0.5em {\varDelta}_1\approx {h}_k^{-1}\kern0.5em {s}_{1,k}\kern0.5em {\varDelta}_1^{*}\kern0.5em {s}_{2,i}\kern0.5em {s}_{0,i}, $$

(9)

and thus we can translate BOLD units into c _ik units with Δ ^*₁  ≈ s ^* _ik  Δ₁, where

$$ {s}_{ki}^{*}={\rho}_{{\boldsymbol{m}}_k\boldsymbol{d}}\kern0.5em {h}_k\kern0.5em {s}_{1,k}^{-1}\kern0.5em {s}_{2,i}^{-1}\kern0.5em {s}_{0,i}^{-1}. $$

(10)

Finally, this implies that our distribution of alternative effects in c _ki units is

$$ {\varDelta}_1^{*}\sim \mathcal{N}\left({s}_{ki}^{*}\kern0.5em \mu, \kern0.5em {s}_{ki}^{*2}\kern0.5em {\tau}^2\right) $$

(11)

(cf Eq. 4).

Significance procedure

Since c _ki has unit noise variance, with an assumption of Gaussianity, the null distribution is given by

$$ {c}_{ki}\left|{H}_0\sim \mathcal{N}\left(0,1\right)\right.. $$

(12)

Under the alternative, we consider the addition of effect Δ ^*₁ to c _ki yielding the alternative distribution

$$ {c}_{ki}\left|{H}_a\sim \mathcal{N}\right.\left({s}_{ki}^{*}\kern0.5em \mu, \kern0.5em 1+{s}_{ki}^{*2}\kern0.33em {\tau}^2\right). $$

(13)

Data

We consider data from a patient suffering from a left prefrontal brain tumor. The study design was a boxcar design, where the patient was asked to alternate between recitation of tongue-twisters and quiescence. Figure 3 shows a sagittal slice of the T2 image, with the tumor visible in the inferior prefrontal frontal cortex. For the application to mass univariate linear modeling, the data were analyzed with FEAT in FSL 4.1 (Smith et al., 2004). The application to independent component analysis was performed using MELODIC in FSL 4.1 (Beckmann & Smith, 2004).

Fig. 3

Anatomical scan of the patient. The tumor can be clearly seen in the prefrontal cortex

Results

Univariate linear modeling

We applied these techniques to the data described in the Data section. We derived the expected effect magnitude for Δ₁ and the variability of that effect τ from 5 patients who underwent the same fMRI paradigm. We threshold the image of each individual using an FDR control at 0.05 and look at the average percent BOLD change units in each individual. The results are shown in Table 1. Therefore, we specify the expected effect magnitude for Δ₁ of μ = 0.73 percent BOLD change units and variability of that effect as $\tau =\sqrt{{{\displaystyle \widehat{\tau}}}^2}=0.21$ percent BOLD change. These results are consistent with others in the literature (see, e.g., Desmond & Glover, 2002, Fig. 7A).

Table 1 Average effect sizes in 5 previously tested patients in percent BOLD change units

Results are shown in Fig. 4 with thresholds α = 0.001 and β = 0.20. In other words, we specified a p ₀ threshold for declaring an activation when there is none at 1-in-1,000; and we set the p ₁ threshold for declaring the absence of activation when, in fact, the specified activation magnitude is present at 1-in-5. The red and the (scant) orange voxels show where H ₀ can be confidently rejected, and, if presurgical planning was done only on the basis of classical null hypothesis testing, all other tissue would be regarded as “safe.” Considering information on the alternative, we have the red voxels where, specifically, H ₀ can be rejected and H _a cannot be rejected; that is, the red voxels are incompatible with the null and compatible with the alternative and, thus, are strong evidence for the effect. The yellow areas are areas where neither H ₀ nor H _a can be rejected; here, the data are compatible with both the null and alternative and suggest a lack of confidence in ruling out activation. Finally, for voxels with no coloration, the H ₀ cannot be rejected, but H _a can; the data are compatible with the null and incompatible with the alternative and, thus, have good evidence for a lack of activation and suggest that these brain regions can be safely resected. This shows the key strength of the procedure: Among voxels traditionally classified as “nonactive”—that is, those with insufficiently small p ₀s, it distinguishes between voxels where there is compelling evidence for nonactivation (not colored) and those voxels where we cannot rule out the possibility of activation (yellow).

Fig. 4

Sagittal slice of “layered” activation inference overlaying grayscale T2* reference image, threshold values of α = 0.001 and β = 0.20. Red areas show areas of high confidence of activation (H ₀ rejected, H _a not rejected), while yellow areas show areas where activation cannot be ruled out (neither H ₀ nor H _a rejected); uncolored areas have high confidence of no activation (H ₀ not rejected, H _a rejected), while the few orange voxels indicate voxels with significant but surprisingly small BOLD response magnitude (H ₀ and H _a rejected)

The orange voxels represent voxels for which the observed effect size is between the null hypothesis of no activation and the expected effect size. In these voxels, both the null and the alternative hypotheses are rejected, which corresponds to very low residual noise in the GLM.

Independent components analysis results

We applied these techniques to the data described in the Data section. We used the same effect size and uncertainty as in the Univariate linear modeling section—that is, μ = 0.73 and τ = 0.18 percent BOLD change units.

MELODIC’s automated dimensionality estimation method in MELODIC found 52 components. We chose one IC whose time series corresponded to the design matrix, shown in Fig. 5. Regressing this temporal mode on the design gives a coefficient of $\widehat{\beta}=1.48$, and thus $h={\widehat{\beta}}^{-1}=0.677$ is the scaling factor used to have the temporal mode express a unit-BOLD effect (see Eq. 8). The pointwise correlation between the design and the chosen component is ρ _md = 0.63, which is used to attenuate the expected effect magnitude (see Eq. 9).

Fig. 5

The time series of a selected IC, with a least squares fit of regressing the series on the design shown in gray. The estimated response height is used to normalize the component to have unit BOLD effect, and the pointwise correlation between the design and the selected IC is used to attenuate the expected BOLD response magnitude

The layered thresholding procedure for this IC is shown in Fig. 6, for α = 0.001 and β = 0.20. There is a set of voxels with strong evidence (red, H ₀ rejected; H _a , accepted) but also additional voxels where both hypotheses are rejected (orange). As was mentioned above, in the setting of presurgical planning, these orange regions are best regarded as regions of possible activation and, thus, excluded from resection.

Fig. 6

Results of the alternative thresholding procedure when using ICA. Sagittal slice of “layered” activation inference overlaying grayscale T2* reference image, threshold values of α = 0.001 and β = 0.20. Red areas show areas of high confidence of activation (H ₀ rejected, H _a not rejected), orange areas show voxels with significant but surprisingly small BOLD response magnitude (H ₀ and H _a rejected); uncolored areas have high confidence of no activation (H ₀ not rejected, H _a rejected)

This result is quite different from the GLM results and is a reflection of the dramatically lower voxel-wise variance in the IC spatial mode relative to the GLM statistic image. The explanation is that the GLM result accounts for all noise variance, while the IC spatial map reflects only the noise in the subspace corresponding to the IC temporal mode (Beckmann & Smith, 2004).

Crucially, we stress that our thresholding procedure seeks only to improve the interpretability of the ICA result and does not produce confirmatory inferences; IC selection is intrincally post hoc and subsequent inferences circular, and all we attempt to do here is improve the thresholding of a selected IC spatial map.

Discussion

Statistical thresholding in the context of multiple tests is generally driven by the need to limit false positives. These stringent testing procedures in fMRI research lead to an abundance of false negatives (Lieberman & Cunningham, 2009) and are, therefore, less useful in the context of presurgical fMRI, where a false negative can have dire consequences. While many attempts have been made to propose more liberal testing criteria—for example, by controlling the FDR instead of the FWER (Genovese et al., 2002)—the focus is still on protecting the type I error rate. The unilateral focus on preventing false positives leads to a bias toward large obvious effects and against complex cognitive and affective effects (Lieberman & Cunningham, 2009). We therefore propose a measure that quantifies the evidence against the alternative hypothesis as introduced in Moerkerke et al. (2006). We use this quantity p ₁ in addition to the classical p ₀ value in a procedure that results in a thresholding procedure with multiple layers of significance. One layer consists of voxels exhibiting strong evidence of activation (red in Figs. 4 and 6), while another layer shows voxels with ambiguous evidence (yellow and orange), and a final layer then consists of voxels for which the presence of activation can be confidently rejected (an absence of overlaid statistic values). Thereby, we offer a more symmetrical interest toward both false positives and false negatives.

We have chosen to focus on voxel-wise inference instead of other topological features, such as peaks (Chumbley, Worsley, Flandin, & Friston, 2010) or clusters (Chumbley & Friston, 2009). These topological inference methods have reduced spatial specificity relative to voxel-wise inference and are, therefore, less suitable for presurgical fMRI, where maximal spatial precision is needed.

To use the procedure described in this article, an expected effect size and its variance need to be defined on a BOLD scale. This is an arbitrary choice, however many possibilities are available. Desmond and Glover (2002) showed, for a specific experimental paradigm, the distribution of percentage of signal change with its distribution. They showed, on average, a BOLD effect size of 0.48 percent BOLD change. Another possibility for estimating the expected effect size can be based on previous research. Since, in presurgical fMRI, the same experiment is repeated over most patients, the effect size can be derived from patients who already underwent the experiment and surgery. The degree to which brain activation in patients is representative for the particular setting for which estimates are needed highly depends on the context and should be carefully judged. It can be expected that different methods will affect the estimates for μ and τ; however, we found that the estimates we obtained by averaging over voxels is close to the effect sizes that can be found in literature.

The two different analytical approaches we used, the GLM and ICA, showed somewhat different results. While both GLM and ICA analyses found similar sets of voxels that were confidently activated (H ₀ rejected, H _a not), in the GLM analysis many voxels were found that did not show evidence against the null or against the alternative (yellow in Fig. 4). The explanation for this outcome is the high level of noise present in the data and, thus, confusion about the veracity of either H ₀ or H _a . In contrast, in the ICA analysis, almost no voxels have this ambiguity, and instead, we find voxels that have evidence against both the null and the alternative. Since ICA is a good tool for identifying structured noise in a data-driven manner, it can be expected that the residual voxel-wise variance will be smaller. Low variances result in a large distance between the null and the alternative distribution functions. Whereas the difference between ICA and the GLM seem contradictory at first, we argue that the differences in our approach reflect real differences between the two analysis tools.

The quantity p ₁ shows a relationship with the voxel-based statistical power defined by Van Horn, Ellmore, Esposito, and Berman (1998). The voxel-wise power in Van Horn et al. translates to the complement of the alternative p-value, p ₁, in our study. However, the use of the quantity is fundamentally different. Whereas Van Horn et al. used the voxel-wise power to visualize and interpret the results of a certain study, we explicitly threshold the quantity. Moreover, the interpretation of both quantities is not so straightforward. When a high power is encountered in a certain voxel, with the method of Van Horn et al., it is interpreted as follows: “If the observed effect in the voxel is used as a cutoff when testing from H ₀, we have a high probability of rejecting H ₀ when H ₀ is indeed false.” However, a large voxel-wise power translates to a small p ₁ and is, in our study, interpreted as follows: “When the alternative hypothesis is true, there is a small probability of observing this effect,” and we will interpret this effect as evidence against the alternative hypothesis. This interpretation is much more straightforward and usable.

This procedure has been developed in light of presurgical fMRI, since false negatives can have harmful consequences for the patient. However, the lack of power is omnipresent in fMRI analyses (Lieberman & Cunningham, 2009), and therefore, this procedure is also very useful in all branches of cognitive neuroscience. For example, negative results (i.e., voxels that are not significantly related to the task) are sometimes regarded as evidence against activation. However, these conclusions are not provided by null hypothesis significance testing. The presented procedure, on the other hand, quantifies the evidence for no activation at each voxel and is, therefore, perfectly suited to interpreting negative results.

We would like to stress that this procedure does not abandon null hypothesis significance testing. The classical significance testing framework is still included in the procedure, represented by one layer of significance. The method is merely an extension of the thresholded statistical parametric map, thereby providing a new layer with information on type II error rate control. Mixture modeling is similar in spirit to this method, in that null and the alternative distribution are used; however, mixture model applications usually focus on only controlling type I errors. With a fitted mixture model, you could also apply our method and find p ₀ and p ₁ values; however, we take pains to estimate alternative effect magnitudes a priori, from separate data, to remove any circularity.

In this procedure, control of false positives remains possible, but our procedure also takes into account information on the false negative rate. We do not assert that our method alleviates all concerns with multiplicity, and one possible direction of future work is a multiplicity correction that adjusts both null and alternative hypothesis inferences for the number of tests.

Appendices

Appendix 1. Scaling steps in ICA

Let Y be the J × I data matrix for time points j = 1, …, J and voxels i = 1, …, I. Then the normalized data matrix Y ^* can be expressed as

$$ {\boldsymbol{Y}}^{*}=\boldsymbol{Y}{\boldsymbol{S}}_0^{-1}, $$

(14)

where S ₀ the I × I diagonal matrix with voxel-wise robust variance estimates on the diagonal. Then the independent component analysis results in the following decomposition

$$ {\boldsymbol{Y}}^{*}\approx \boldsymbol{M}{\boldsymbol{C}}^{*}, $$

(15)

where M represents the J × K mixing matrix, where K is the number of components. When K < J, Eq. 15 is only an approximation. C ^* is the K × I matrix with the original image component loadings.

In the probabilistic ICA framework (Beckmann & Smith, 2004), C ^* is Gaussian distributed and can, therefore, be used as a test statistic. To normalize C ^* to a standard Gaussian distribution,

$$ \boldsymbol{C}={\boldsymbol{S}}_1^{-1}{\boldsymbol{C}}^{*}{\boldsymbol{S}}_2^{-1}, $$

(16)

where the diagonal matrix S ₁ scales the components (over voxels) and the diagonal matrix S ₂ scales the voxels (over components):

$$ {\boldsymbol{S}}_1^2= diag\left\{{\boldsymbol{M}}^{-1}\left({\boldsymbol{M}}^{-1}\right)\prime \right\} $$

(17)

$$ {\boldsymbol{S}}_2= diag\left\{\mathrm{SD}\left({\boldsymbol{Y}}^{*}-\boldsymbol{M}{\boldsymbol{C}}^{*}\right)\right\}\frac{\sqrt{I-K}}{\sqrt{I-1}}, $$

(18)

where SD(Y ^*−MC ^*) is the column-wise standard deviation of the residuals of the ICA approximation. Consequently, we can approximate Y as MS ₁ CS ₂ S ₀ in Eq. 6.

Appendix 2. Attenuation of anticipated BOLD effect for a given IC

A basic result in psychometrics (Spearman, 1904) holds that the correlation between unreliable measures is attenuated by the test–retest reliability of each measure. For example, if measure A imperfectly measures variable A ^* and B imperfectly measures B ^*, then the correlation of A and B is attenuated relative to the uncorrupted measures:

$$ \mathrm{Corr}\left(A,B\right)=\mathrm{Corr}\left({A}^{*},{B}^{*}\right)\kern0.5em \sqrt{\rho_{AA}}\kern0.33em \sqrt{\rho_{BB}}, $$

(19)

where ρ _AA and ρ _BB are the test–retest correlations of A and B.

In our setting, let A be the BOLD response and B be the IC time course m. The reproducibility of BOLD (ρ _AA ) is respectable, with Vul, Harris, Winkielman, and Pashler (2009) reporting reliabilities between .66 and .94; however, most procedures for fMRI data analysis do not take this into account, and hence we only consider ρ _AA = 1.

We are interested in the “reproducibility” of an IC time course relative to the experimental design d. Of course, obtaining two replicates of an IC time course that both equally reflect d is not feasible, but we can indirectly estimate ρ _BB as follows.

Consider a general test–retest setting, where measurement B ₁ is made at one time and, later, a “retest” gives measurement B ₂. Assuming additive error, we can relate the “corrupted” measures to the “uncorrupted” measures as

$$ {B}_1={B}^{*}+{\upepsilon}_1 $$

(20)

$$ {B}_2={B}^{*}+{\upepsilon}_2, $$

(21)

where ε _j , j = 1, 2 are the measurement-specific errors, and we assume Var(B ^*) = σ ² is the variance of the perfectly reproducible measure and Var(ε _j ), is the variance of the corrupting noise. The test–retest correlation is then

$$ {\rho}_{BB}=\mathrm{Corr}\left({B}_1,{B}_2\right)=\frac{\sigma^2}{\sigma^2+{\tau}^2}. $$

(22)

If, instead, one corrupted measure is correlated with the uncorrupted “true” measure, you find

$$ \mathrm{Corr}\left({B}_1,{B}^{*}\right)=\frac{\sigma }{\sqrt{\sigma^2+{\tau}^2}}=\sqrt{\rho_{BB}}. $$

(23)

Or, equivalently, ρ _BB = Corr(B ₁, B ^*)². In short, these results show that we can estimate the “reproducibility” of m _k as a noisy sample of the true d as ${\rho}_{{\boldsymbol{m}}_k\boldsymbol{d}}^2$.

Finally, from Eq. 19, taking ρ _AA = 1 and ${\rho}_{BB}={\rho}_{{\boldsymbol{m}}_k\boldsymbol{d}}^2$, we see that the attenuation factor needed to account for the mismatch between m _k and d is just ${\rho}_{{\boldsymbol{m}}_k\boldsymbol{d}}$.