A critical reexamination of doing arithmetic nonconsciously
 948 Downloads
 4 Citations
Abstract
A recent study claimed to have obtained evidence that participants can solve invisible multistep arithmetic equations (Sklar et al., 2012). The authors used a priming paradigm in which reaction times to targets congruent with the equation’s solution were responded to faster compared with incongruent ones. We critically reanalyzed the data set of Sklar et al. and show that the claims being made in the article are not fully supported by the alternative analyses that we applied. A Bayesian reanalysis of the data accounting for the random variability of the target stimuli in addition to the subjects shows that the evidence for priming effects is less strong than initially claimed. That is, although Bayes factors revealed evidence for the presence of a priming effect, it was generally weak. Second, the claim that unconscious arithmetic occurs for subtraction but not for addition is not supported when the critical interaction is tested. Third, the data do not show wellestablished features of numerosity priming as derived from Vshaped response time curves for primetarget distances. Fourth, we show that it is impossible to classify reaction times as resulting from congruent or incongruent primetarget relationships, which should be expected if their results imply that participants genuinely solve the equations on each trial. We conclude that the claims being made in the original article are not fully supported by the analyses that we apply. Together with a recent failure to replicate the original results and a critique of the analysis based on regression to the mean, we argue that the current evidence for unconscious arithmetic is inconclusive. We argue that strong claims require strong evidence and stress that cumulative research strategies are needed to provide such evidence.
Keywords
Interocular suppression Unconscious processing ReproducibilityIntroduction
In their article, Sklar et al. (2012) claimed to have shown that participants can solve complex arithmetic equations nonconsciously, i.e., in the absence of consciously perceiving the equations. Specifically, they examined whether the presentation of multistep additions and subtractions with three singledigit operands (e.g., “9 − 3 − 4 =”, “3 + 1 + 4 =”) and without the result (2 and 8, respectively) would bias the verbal enumeration of a subsequently presented, visible number. Thus, the experiments by Sklar et al. were designed to test for “priming” effects, in which the exposure to a stimulus (or, prime) influences the response to a second stimulus (or target).^{1} Following the seminal work by Meyer and Schvaneveldt (1971), priming has been widely used in the fields of cognitive psychology and neuroscience to infer the structure of semantic representations, including the representation of numerical values (Dehaene, Molko, Cohen, & Wilson, 2004; Knops, 2016). In the case of Sklar et al., target numbers were either congruent or incongruent with the result of the prime equation. For example, the target number “2” is congruent with the result of the equation “9 − 3 − 4 =”, while numbers “3” or “5” are incongruent. Repeated measures analysis of variance (rmANOVA) revealed significantly shorter response times (RTs) for congruent compared with incongruent priming trials. Rather surprisingly, this congruency priming effect was significant for subtractions, but not for additions.^{2} The authors concluded from these data “that uniquely human cultural products, such as […] solving arithmetic equations, do not require consciousness” (p. 19617).
For the following reasons, we believe that it is crucial to reexamine whether the claims made by Sklar et al. (2012) are fully supported by the available data. From a theoretical standpoint, the claim of “doing arithmetic nonconsciously” is a strong claim and, hence, demands strong evidence. Most cognitive scientists would agree that the complex nature of the underlying cognitive processes renders it implausible, rather than plausible, that effortful arithmetic operations may be performed without consciousness. Specifically, multistep additions and subtractions as used by Sklar et al. cannot be solved by declarative fact retrieval from long term memory alone. Successful performance would necessitate that arithmetic rules can be initiated and followed unconsciously, and that the unconscious intermediary results are stored in working memory. So far, there exists only a single study on unconscious addition making the former claim (Ric & Muller, 2012). Furthermore, considering the technical setup adds to the a priori implausibility of the effect reported by Sklar et al. They used continuous flash suppression (CFS) to render the prime equations invisible for up to 2 seconds. Following the introduction of this interocular suppression method (Tsuchiya & Koch, 2005), a very heterogeneous picture has emerged about the extent to which highlevel unconscious visual processing is possible under CFS (Ludwig & Hesselmann, 2015; Sterzer, Stein, Ludwig, Rothkirch, & Hesselmann, 2014; Yang, Brascamp, Kang, & Blake, 2014). To the best of our knowledge, there is no evidence for Sklar and colleagues’ premise that CFS allows for more unconscious processing, because “it gives unconscious processes ample time to engage with and operate on subliminal stimuli” (p. 19614). On the contrary, it may rather be that long suppression durations (Experiment 6: 1,700 ms and 2,000 ms; Experiment 7: 1,000 ms and 1,300 ms) are associated with a particularly deep suppression of visual processing under CFS (Tsuchiya, Koch, Gilroy, & Blake, 2006), and extended periods of invisible stimulation have been shown to lead to negative priming influences (Barbot & Kouider, 2011).
The goal of the current article is to provide a critical reexamination of the claims made by Sklar et al. (2012) based on the data they collected. We do so by approaching the original data set from different angles, and we assess whether the conclusions based on the original results still hold when taking the results of these new analyses into account. In the next sections, we provide five reanalyses of the data obtained by Sklar et al. First, we verified the repeatability of the reported analyses. This was a crucial first step that guaranteed we were analyzing the same data set as the original study. Second, we analyzed the data using a Bayesian linear mixedeffects models with crossed random effects for participants and stimuli, relying on Bayes factors to quantify how strongly the data support the predictions made by one statistical model compared to another. Indeed, throughout our reanalyses we are explicitly interested in quantifying the degree to which the data provide evidence for the claims that were made in the original study. It has been argued that classical significance testing approaches are not explicitly connected to statistical evidence, whereas the Bayes factor provides the possibility to quantify evidence in a coherent framework (Morey, Romeijn, & Rouder, 2016). This motivated us to rely on the Bayes factor throughout our reanalyses (except for the last one, see further). Third, Sklar et al. (2012) claimed that the congruency effect was observed for subtraction equations but not for addition equations. However, the interaction between congruency and operation was never tested although it is critical to ensure the congruency effect is different for subtraction compared to addition. Fourth, if the congruency effect observed for the subtraction equations was genuinely due to number processing, one would predict a distance effect to be present in the data. That is, as primetarget distance increases, response times should increase as well. Therefore, we assessed whether the data showed such distancedependent priming effects. Fifth, Sklar et al. (2012) interpreted that the congruency effect indicated that participants unconsciously solved the equations. It has recently been argued that such a claim is only warranted if reaction times are predictive of primetarget congruency. That is, accurate classification of the primetarget relationship should be possible from the reaction time distributions.
For the sake of brevity, we report all reanalyses for Experiment 6 only in the main text of this article. We refer to the Supplementary Materials for the results of the reanalysis of Experiment 7, which were qualitatively the same. An extensive overview of all calculations is also available in the Supplementary Materials, including the code used to process the data and conduct the analyses.
Methods
Data preparation
We obtained the data from Sklar et al. (Sklar et al., 2012). All data were processed and analyzed in R 3.3.2, a statistical programming language, and RStudio 1.0.44 (R Core Team, 2014; RStudio Team, 2015). A complete overview of this analysis can be found in the R markdown file in the Supplementary Materials (https://doi.org/10.6084/m9.figshare.4888391.v2). All data were visualized with the yarrr package version 0.1.2 (Phillips, 2016).
Reanalysis #1: Repeatability of the reported analyses
We followed all data processing steps reported in Sklar et al. to compute mean response times for each participant – condition combination. We used the afex package version 0.161 to recalculate the repeated measures ANOVA by using the aov_car() function (Singmann, Bolker, Westfall, & Aust, 2016). Type III sums of squares were used, as these are default in many commercially available statistical packages and because one of these (SPSS) was used by Sklar et al. to analyze the data.
Reanalysis #2, #3, and #4: Bayesian linear mixedeffects models with crossed random effects
In itself, the Bayes Factor can be interpreted as a relative measure of evidence for one statistical model compared to another. That is, the value of the Bayes Factor has no absolute meaning, and should always be interpreted relative to the statistical models under consideration. As Etz and Vandekerckhove (2016, p.4) put it: “The Bayes factor is most conveniently interpreted as the degree to which the data sway our belief from one to the other hypothesis.” Although the Bayes factor is inherently continuous, its values are sometimes partitioned into categories indicating different grades of evidence. For example, a Bayes factor of 3 or more often is associated with moderate evidence for one model, whereas Bayes factors larger than 10 are deemed strong evidence for that model. Bayes factors between 1/3 and 3 often are interpreted as providing equal support for both models, or anecdotal evidence for either model. We took these categories as guidelines, but we do not wish to fall prey to traditional accept/reject classifications, such as those that are standard in classical null hypothesis significance testing.
We used the generalTestBF() function to calculate the Bayes factors associated with the full model (i.e., including all fixed and random effects of interest) and most reduced versions of the full model (the whichModels argument was set to “withmain,” such that interaction effects were only included if the respective main effects also were included in the model) (Rouder, Engelhardt, McCabe, & Morey, 2016). With respect to the random effects, random intercepts were included for both participants and target stimuli. Initial analyses also were performed, including random slopes for participants and target stimuli for the congruency effect. However, models including random slopes were never favored over models, including random intercepts only. Therefore, we decided to drop random slopes altogether in the analyses reported. All default prior settings were used (i.e., a “medium” prior scale for the fixed effects (r = 0.5) and a “nuisance” prior scale for the random effects (r = 1)). Our general strategy of reporting the Bayes factors is as follows. We extracted the model with the highest Bayes factor compared to an empty model (i.e., an interceptonly model) and considered this to be the model that predicted the data best (in the following referred to as “best model”). We then recalculated all Bayes factors such that they are compared to this best model. In all tables, this yields an overview of the best model (Bayes factor = 1), and how strongly the data support the predictions made by this model compared to all other models. Because prior settings influence the Bayes factor, we also report on sensitivity analyses in the Supplementary Materials by varying the value of the prior scale of the fixed effects (which are of most interest here). We always included two models in the sensitivity analysis (i.e., yielding a single Bayes factor for each value of the prior scale). One of those was the best model, and in the other one the most important variable for the current reanalysis was included or excluded (depending on its inclusion in the best model). For example, if the best model contained only a main effect of primetarget congruency in addition to the random effects, the sensitivity analysis would be conducted for this model and the model including random effects only.
Reanalysis #5: A significant difference does not imply accurate classification
For this analysis, we used the R code that was used in Franz and von Luxburg (Franz & von Luxburg, 2015), which is publicly available (https://osf.io/7825t/). The classification analysis can be summarized as follows. The goal is to determine a threshold RT that can be used to classify RTs as either congruent or incongruent. In the case of the median classifier, the median RT is used as a threshold. In the case of the trained classifier, the data set is split into two halves: a training and a test set. For the training set, the threshold value is determined that leads to the fewest number of misclassifications, and this threshold is then applied to the test set. This was then repeated 10 times, and the average classification accuracy was taken as classification performance for the trained classifier.
Results
Reanalysis #1: Repeatability of the reported analyses
Reanalysis #2: (Bayesian) linear mixedeffects modeling with crossed random effects
Bayes factor analysis of the subtraction data (Experiment 6)
Model  BF  Error 

1. Congruency + Subject + Target  1  0 
2. Congruency + Presentation duration + Subject + Target  1.16  0.07 
3. Subject + Target  2.15  0.03 
4. Presentation duration + Subject + Target  2.65  0.03 
5. Congruency * Presentation duration + Subject + Target  10.87  0.05 
All other models  >100  NA 
Reanalysis #3: Analysis of the claimed “congruency x operation” interaction
Sklar et al. (2012) interpret the results of Experiments 6 and 7 as follows: “The results so far show that subtraction equations are solved nonconsciously and hence are sufficient to confirm our hypothesis that complex arithmetic can be performed unconsciously. However, why did not we find evidence for nonconscious solution of the easiertosolve addition equations?” (p. 19616).
Bayes Factor analysis for the full data set (Experiment 6)
Model  BF  Error 

1. Target + Subject  1  0 
2. Presentation duration + Target + Subject  1.30  0.02 
3. Operation + Target + Subject  4.57  0.01 
4. Congruency + Target + Subject  5.28  0.02 
5. Presentation duration + Operation + Target + Subject  5.83  0.07 
6. Congruent + Presentation duration + Target + Subject  7.21  0.02 
All other models  >100  NA 
Reanalysis #4: The effect of numerical primetarget distance on response times
Bayes factor analysis of the primetarget distance (Experiment 6)
Model  BF  Error 

1. Subject + Target  1  0 
2. Presentation duration + Subject + Target  1.34  0.01 
3. Operation + Subject + Target  4.40  0.04 
4. Presentation duration + Operation + Subject + Target  6.03  0.02 
5. Presentation duration*Operation + Subject + Target  9.12  0.06 
6. Distance + Subject + Target  14.67  0.01 
All other models  >100  NA 
Reanalysis #5: A significant difference does not imply accurate classification
Classification performance for the subtraction data (Experiment 6)
Classifier  Mean  SD  SEM  Median 

Median classifier  53.10  4.43  1.07  53.52 
Trained classifier  50.88  5.76  1.40  51.89 
Discussion
In this article, we critically reanalyzed the data reported in Sklar et al. (2012). We first established that all analyses were repeatable without any discrepancies (Reanalysis #1). For making their data available to us and their research transparent, the authors should be applauded. Indeed, recent empirical evaluations have shown that the published biomedical literature generally lacks transparency, including public access to raw data and code (Goodman, Fanelli, & Ioannidis, 2016; Iqbal, Wallach, Khoury, Schully, & Ioannidis, 2016; Leek & Jager, 2016). Furthermore, a recent series of studies has indicated that half of the published psychology papers include at least one statistical inconsistency, and one in eight even a gross inconsistency (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2015). Importantly, the full repeatability ensured that our following reanalyses were based on exactly the same data set as used for the original publication.

When applying a statistical model that provides better control for the Type I error rate for the experimental design at hand, we showed that the evidence in favor of the presence of a congruency effect was not as strong as would be derived from the analyses that were reported in the original article, albeit the fact that the best model did include an effect of primetarget congruency (Reanalysis #2). Thus, on purely statistical grounds, this result shows that merely accounting for item variability substantially attenuates the strength of the evidence for the reported priming effect. In essence, this does not contradict the result reported by Sklar et al. Nevertheless, we argue that the strength of the evidence provides an important nuance to the interpretation of these results.

The data do not strongly support the claim that unconscious arithmetic can happen for subtraction equations, but not for addition equations. That is, none of the models reported in Reanalysis #3 included an interaction between primetarget congruency and operation. Moreover, all BF analyses also indicated that the data were more consistent with statistical models not including an effect of primetarget congruency. Thus, an analysis based on the full data set rather than different subsets of the data revealed that no strong evidence for main effects of or interactions with primetarget congruency was observed.

No characteristic patterns of number processing, which have repeatedly and robustly been reported in the literature, are present in the current data set (Reanalysis #4). This indicates that one should be very cautious to invoke mechanisms related to number processing to explain these results.

Even if the priming effect is taken at face value after the results of the three previous reanalyses, the data set does not provide evidence that participants unconsciously solved the equations that were presented subliminally. That is, the classification of the primetarget congruency based on the reaction times is nearly at chance, calling into question the assertion that people can solve equations nonconsciously (Reanalysis #5). Although the median classifier performed slightly above chance, its performance was still considerably lower than the performance that was taken to be the cutoff for establishing invisibility of the prime equations (60%).
Taken together, we conclude that the converging nature of all four reanalyses indicates that the data used for invoking the existence of unconscious arithmetic contain little evidential value for those claims (i.e., evidential value in terms of the Bayes factors obtained in the reanalyses). Within the conceptual framework proposed by Goodman et al. (2016), our reanalyses therefore suggest low inferential reproducibility of the study by Sklar et al.
A critical reviewer suggested that, based on the results of our reanalyses, one would expect that the original findings would not replicate easily. In this context, a direct replication of the study by Sklar et al., using the same experimental setup and exactly the same stimulus material, would be very informative. This was the goal of the recent study by Karpinski, Yale, and Briggs (2016). The authors used exactly the same materials as in Sklar et al. and aimed at replicating the original effect in a larger sample (n = 94). Interestingly, they obtained evidence for unconscious addition, but not subtraction (i.e., opposite findings compared to Sklar et al.). As this data set would be very informative for our reanalysis, we contacted the authors of this replication study. Upon reanalyzing the data set together with the authors, it became apparent, however, that a coding error led to an incorrect calculation of the mean RTs. A corrected analysis of the data did not reveal any priming effects for unconscious additions or subtractions (Karpinski & Briggs, personal communication). That is, the critical paired comparisons for assessing priming effects for addition and subtraction both no longer passed the threshold for statistical significance (addition: t(93) = 0.11, p = 0.92; subtraction: t(93) = 0.23, p = 0.82). The paper has now been retracted. Thus, the single published replication study of the unconscious addition and subtraction effects reported in Sklar et al. actually failed to replicate the original pattern of results. Together with the results of our reanalyses, we argue that the results of this nonreplication calls for caution when interpreting the original results.
Exploring the scope and limits of nonconscious processing is essential for the formulation of theories of consciousness (Dehaene, Charles, King, & Marti, 2014). Since the results reported in Sklar et al. (2012) might have important implications for theories of (un)conscious processing (Dehaene et al., 2014; Koch, Massimini, Boly, & Tononi, 2016; Soto & Silvanto, 2014), we were motivated to conduct this critical reanalysis. If the reported effect is true, it can indeed be considered as an extraordinary case of subliminal perception and, as Sklar et al. argue, it might even “call for a significant update of our view of conscious and unconscious processes” (p.19614). In line with this notion, the senior author of this study recently suggested that “unconscious processes can carry out every fundamental highlevel function that conscious processes can perform” (p. 195) (Hassin, 2013). Nonconscious arithmetic would be the most recent culmination point in a decadeslong debate among cognitive scientists about the existence and potency of subliminal perception (Doyen, Klein, Simons, & Cleeremans, 2014). This debate has been characterized by a repeating cycle of provocative claims followed by methodological criticism, primarily aimed at the psychophysical and statistical methods used to establish the absence of conscious perception (Hesselmann & Moors, 2015). Of note, for the purpose of this reexamination, we solely relied on the data that were used to claim the existence of nonconscious arithmetic. For example, we simply took at face value that the post hoc selected sample of participants, whose data were submitted to statistical analysis, did not see the arithmetic equations; the crucial aspect of post hoc data selection and its implications has been treated elsewhere (Shanks, 2016).
The scientific study of consciousness has traditionally sought to assemble an exhaustive inventory of the psychological processes that can proceed unconsciously to isolate those that are exclusively restricted to conscious cognition (Naccache, 2009). During the course of the last decades, a large body of empirical evidence has been accumulated by applying this strategy, in particular within the domain of visual perception. Vision research provides a wide range of paradigms designed to transiently suppress visual stimuli from conscious perception, i.e., render a physically present target stimulus invisible for neurologically intact observers (Bachmann, Breitmeyer, & Ogmen, 2007). These paradigms differ with respect to what types of visual stimuli can be suppressed from awareness, and how effective the suppression is in terms of duration and controllability of onset and offset (Kim & Blake, 2005). Along another dimension, the available paradigms may be placed within a functional hierarchy of unconscious processing, according to the extent to which features of visual stimuli are processed on an unconscious level and still induce effects on behaviour, e.g., in priming experiments (Breitmeyer, 2015). The results of our reanalysis can be framed into an emerging series of results that indicate that unconscious processing associated with CFS is not as highlevel as previously thought (Hedger, Adams, & Garner, 2015; Hesselmann, Darcy, Sterzer, & Knops, 2015; Hesselmann & Knops, 2014; Moors, Boelens, et al., 2016) and that neural activity related to stimuli suppressed by CFS is considerably reduced already in early visual areas (Fogelson, Kohler, Miller, Granger, & Tse, 2014; YuvalGreenberg & Heeger, 2013). Importantly, building such a functional hierarchy should eventually allow to formulate predictions on the level of unconscious processing that can be expected in a specific experimental setup. In the absence of prior assumptions on the depth of visual suppression associated with a specific paradigm, every new report of highlevel unconscious processing seems equally plausible, and the boundaries of nonconscious processing are ultimately pushed further and further.
In sum, as extraordinary claims require extraordinary evidence, we were motivated to reanalyze the data obtained in Sklar et al. based on statistical, methodological, and theoretical considerations, and within a framework that allowed us to quantify the evidence for statistical models that reflected theoretical claims (i.e., unconscious arithmetic revealed through priming effects). Together with the recent nonreplication of the original results (Karpinski et al., 2016) and a recent critique of the posthoc selection of unaware participants that was used in the original study (Shanks, 2016), we argue that our results indicate that the evidence for the existence of unconscious arithmetic is inconclusive. This current state of affairs can only be overcome by cumulative research strategies, explicitly aimed at assessing the robustness of the findings and quantifying the strength of the evidence for the theoretical claims.
Supporting Information
All Supplementary Materials can be accessed as a HTML or R Markdown file at: https://doi.org/10.6084/m9.figshare.4888391.v2
Footnotes
 1.
Please note that the study by Sklar et al. also included experiments on “nonconscious reading” that will not be addressed here. Instead of priming, Experiments 15 used a variant of interocular suppression (breaking CFS) in which the time to target detection is the dependent variable. The extent to which this paradigm can provide evidence for unconscious processing has been called into question, however (Stein & Sterzer, 2014).
 2.
Priming effects for additions were only observed when Sklar et al. modified the experimental design and used a different dependent measure. In Experiment 9, equations with two singledigit operands (e.g., “8+7 =”) were unconsciously presented, and participants had to report whether a subsequently presented visible addition equation with two singledigit operands and result (e.g., “9+6=15”) was correct or not. The results showed that participants made significantly fewer mistakes in compatible trials (3.2%) than in incompatible trials (4.4%). We do not address this weaker finding in our article.
Notes
Acknowledgements
P.M. was supported by the Research Fund Flanders (FWO Vlaanderen) through a doctoral fellowship. GH is supported by the German Research Foundation (grant HE 6244/12).
Author contributions
P.M. and G.H. designed research; P.M. analyzed data; P.M. and G.H. wrote the paper.
References
 Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixedeffects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. doi: 10.1016/j.jml.2007.12.005 CrossRefGoogle Scholar
 Bachmann, T., Breitmeyer, B., & Ogmen, H. (2007). The experimental phenomena of consciousness: A brief dictionary. Oxford: Oxford University Press.Google Scholar
 Barbot, A., & Kouider, S. (2011). Longer is not better: Nonconscious overstimulation reverses priming influences under interocular suppression. Attention, Perception & Psychophysics, 74, 174–184. doi: 10.3758/s1341401102263 CrossRefGoogle Scholar
 Breitmeyer, B. G. (2015). Psychophysical “blinding” methods reveal a functional hierarchy of unconscious visual processing. Consciousness and Cognition, 35, 234–250. doi: 10.1016/j.concog.2015.01.012 CrossRefPubMedGoogle Scholar
 Clark, H. H. (1973). The languageasfixedeffect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335–359. doi: 10.1016/S00225371(73)800143 CrossRefGoogle Scholar
 Core Team, R. (2014). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved from http://www.Rproject.org/ Google Scholar
 Dehaene, S., Molko, N., Cohen, L., & Wilson, A. J. (2004). Arithmetic and the brain. Current Opinion in Neurobiology, 14(2), 218–224. doi: 10.1016/j.conb.2004.03.008
 Dehaene, S., Charles, L., King, J.R., & Marti, S. (2014). Toward a computational theory of conscious processing. Current Opinion in Neurobiology, 25, 76–84. doi: 10.1016/j.conb.2013.12.005 CrossRefPubMedGoogle Scholar
 Doyen, S., Klein, O., Simons, D. J., & Cleeremans, A. (2014). On the other side of the mirror: Priming in cognitive and social psychology. Social Cognition, 32(Supplement), 12–32. doi: 10.1521/soco.2014.32.supp.12 CrossRefGoogle Scholar
 Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: Psychology. PLoS ONE, 11(2), e0149794. doi: 10.1371/journal.pone.0149794 CrossRefPubMedPubMedCentralGoogle Scholar
 Fogelson, S. V., Kohler, P. J., Miller, K. J., Granger, R., & Tse, P. U. (2014). Unconscious neural processing differs with method used to render stimuli invisible. Frontiers in Psychology, 5, 601. doi: 10.3389/fpsyg.2014.00601 CrossRefPubMedPubMedCentralGoogle Scholar
 Franz, V. H., & von Luxburg, U. (2014). Unconscious lie detection as an example of a widespread fallacy in the neurosciences. arXiv:1407.4240. Retrieved from http://arxiv.org/abs/1407.4240
 Franz, V. H., & von Luxburg, U. (2015). No evidence for unconscious lie detection: A significant difference does not imply accurate classification. Psychological Science. doi: 10.1177/0956797615597333 PubMedGoogle Scholar
 Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328–331. doi: 10.1198/000313006X152649 CrossRefGoogle Scholar
 Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science Translational Medicine, 8(341). doi: 10.1126/scitranslmed.aaf5027
 Hassin, R. R. (2013). Yes it can: On the functional abilities of the human unconscious. Perspectives on Psychological Science, 8(2), 195–207. doi: 10.1177/1745691612460684 CrossRefPubMedGoogle Scholar
 Hedger, N., Adams, W. J., & Garner, M. (2015). Fearful faces have a sensory advantage in the competition for awareness. Journal of Experimental Psychology: Human Perception and Performance. doi: 10.1037/xhp0000127 Google Scholar
 Hesselmann, G., & Knops, A. (2014). No conclusive evidence for numerical priming under interocular suppression. Psychological Science. doi: 10.1177/0956797614548876
 Hesselmann, G., & Moors, P. (2015). Definitely maybe: Can unconscious processes perform the same functions as conscious processes?. Frontiers in Psychology, 6(584). doi: 10.3389/fpsyg.2015.00584
 Hesselmann, G., Darcy, N., Sterzer, P., & Knops, A. (2015). Exploring the boundary conditions of unconscious numerical priming effects with continuous flash suppression. Consciousness and Cognition, 31, 60–72. doi: 10.1016/j.concog.2014.10.009 CrossRefPubMedGoogle Scholar
 Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., & van Noort, V. (2009). Repeatability of published microarray gene expression analyses. Nature Genetics, 41(2), 149–155. doi: 10.1038/ng.295 CrossRefPubMedGoogle Scholar
 Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016). Reproducible research practices and transparency across the biomedical literature. PLoS Biology, 14(1), e1002333. doi: 10.1371/journal.pbio.1002333 CrossRefPubMedPubMedCentralGoogle Scholar
 Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103(1), 54–69. doi: 10.1037/a0028347 CrossRefPubMedGoogle Scholar
 Karpinski, A., Yale, M., & Briggs, J. C. (2016). Unconscious arithmetic processing: A direct replication. European Journal of Social Psychology. doi: 10.1002/ejsp.2175 Google Scholar
 Kim, C.Y., & Blake, R. (2005). Psychophysical magic: Rendering the visible “invisible.”. Trends in Cognitive Sciences, 9(8), 381–388. doi: 10.1016/j.tics.2005.06.012 CrossRefPubMedGoogle Scholar
 Knops, A. (2016). Probing the neural correlates of number processing. The Neuroscientist. doi: 10.1177/1073858416650153 PubMedGoogle Scholar
 Koch, C., Massimini, M., Boly, M., & Tononi, G. (2016). Neural correlates of consciousness: Progress and problems. Nature Reviews Neuroscience, 17(5), 307–321. doi: 10.1038/nrn.2016.22 CrossRefPubMedGoogle Scholar
 Leek, J. T., & Jager, L. R. (2016). Is most published research really false? bioRxiv, 50575. doi: 10.1101/050575
 Ludwig, K., & Hesselmann, G. (2015). Weighing the evidence for a dorsal processing bias under continuous flash suppression. Consciousness and Cognition, 35, 251–259. doi: 10.1016/j.concog.2014.12.010 CrossRefPubMedGoogle Scholar
 Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90(2), 227–234.CrossRefPubMedGoogle Scholar
 Moors, P., Boelens, D., van Overwalle, J., & Wagemans, J. (2016). Scene integration without awareness: No conclusive evidence for processing scene congruency during continuous flash suppression. Psychological Science, 27(7), 945–956. doi: 10.1177/0956797616642525 CrossRefPubMedGoogle Scholar
 Moors, P., Wagemans, J., & deWit, L. (2016). Faces in commonly experienced configurations enter awareness faster due to their curvature relative to fixation. PeerJ, 4. doi: 10.7717/peerj.1565
 Morey, R. D., Rouder, J. N., Love, J., & Marwick, B. (2015). BayesFactor: 0.9.122 CRAN [Zenodo]. Retrieved from doi: 10.5281/zenodo.31202
 Morey, R. D., Romeijn, J.W., & Rouder, J. N. (2016). The philosophy of Bayes factors and the quantification of statistical evidence. Journal of Mathematical Psychology, 72, 6–18. doi: 10.1016/j.jmp.2015.11.001 CrossRefGoogle Scholar
 Naccache, L. (2009). Priming. In T. Bayne, A. Cleeremans, & P. Wilken (Eds.), The Oxford companion to consciousness (pp. 533–536). Oxford: Oxford University Press.Google Scholar
 Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14(9), 1105–1107. doi: 10.1038/nn.2886 CrossRefPubMedGoogle Scholar
 Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L., Epskamp, S., & Wicherts, J. M. (2015). The prevalence of statistical reporting errors in psychology (19852013). Behavior Research Methods. doi: 10.3758/s1342801506642 PubMedPubMedCentralGoogle Scholar
 Phillips, N. (2016). yarrr: A companion to the ebook YaRrr!: The Pirate’s Guide to R. R package version 0.0.5. Retrieved from www.thepiratesguidetor.com
 Reynvoet, B., Brysbaert, M., & Fias, W. (2002). Semantic priming in number naming. The Quarterly Journal of Experimental Psychology, 55(4), 1127–1139. doi: 10.1080/02724980244000116 CrossRefPubMedGoogle Scholar
 Ric, F., & Muller, D. (2012). Unconscious addition: When we unconsciously initiate and follow arithmetic rules. Journal of Experimental Psychology: General, 141(2), 222–226. doi: 10.1037/a0024608 CrossRefGoogle Scholar
 Roggeman, C., Verguts, T., & Fias, W. (2007). Priming reveals differential coding of symbolic and nonsymbolic quantities. Cognition, 105(2), 380–394. doi: 10.1016/j.cognition.2006.10.004 CrossRefPubMedGoogle Scholar
 Rouder, J. N., Engelhardt, C. R., McCabe, S., & Morey, R. D. (2016). Model comparison in ANOVA. Psychonomic Bulletin & Review, 23(6), 1779–1786. doi: 10.3758/s1342301610265 CrossRefGoogle Scholar
 RStudio Team. (2015). RStudio: Integrated development environment for R (Version 0.99.441). Boston: RStudio, Inc.Google Scholar
 Shanks, D. R. (2016). Regressive research: The pitfalls of post hoc data selection in the study of unconscious mental processes. Psychonomic Bulletin & Review, in press.Google Scholar
 Singmann, H., Bolker, B., Westfall, J., & Aust, F. (2016). afex: Analysis of Factorial Experiments. R package version 0.161. Retrieved from http://CRAN.Rproject.org/package=afex
 Sklar, A. Y., Levy, N., Goldstein, A., Mandel, R., Maril, A., & Hassin, R. R. (2012). Reading and doing arithmetic nonconsciously. Proceedings of the National Academy of Sciences, 109(48), 19614–19619. doi: 10.1073/pnas.1211645109 CrossRefGoogle Scholar
 Soto, D., & Silvanto, J. (2014). Reappraising the relationship between working memory and conscious awareness. Trends in Cognitive Sciences, 18(10), 520–525. 10.1016/j.tics.2014.06.005.CrossRefPubMedGoogle Scholar
 Stein, T., & Sterzer, P. (2014). Unconscious processing under interocular suppression: Getting the right measure. Frontiers in Psychology, 5, 387. doi: 10.3389/fpsyg.2014.00387
 Stein, T., Kaiser, D., & Peelen, M. V. (2015). Interobject grouping facilitates visual awareness. Journal of Vision, 15(8), 10. doi: 10.1167/15.8.10 CrossRefPubMedGoogle Scholar
 Sterzer, P., Stein, T., Ludwig, K., Rothkirch, M., & Hesselmann, G. (2014). Neural processing of visual information under interocular suppression: A critical review. Frontiers in Psychology, 5, 453. doi: 10.3389/fpsyg.2014.00453 CrossRefPubMedPubMedCentralGoogle Scholar
 Tsuchiya, N., & Koch, C. (2005). Continuous flash suppression reduces negative afterimages. Nature Neuroscience, 8(8), 1096–1101. doi: 10.1038/nn1500 CrossRefPubMedGoogle Scholar
 Tsuchiya, N., Koch, C., Gilroy, L. A., & Blake, R. (2006). Depth of interocular suppression associated with continuous flash suppression, flash suppression, and binocular rivalry. Journal of Vision, 6(10), 1068–1078. doi: 10.1167/6.10.6 CrossRefPubMedGoogle Scholar
 Van Opstal, F., Gevers, W., De Moor, W., & Verguts, T. (2008). Dissecting the symbolic distance effect: Comparison and priming effects in numerical and nonnumerical orders. Psychonomic Bulletin & Review, 15(2), 419–425.CrossRefGoogle Scholar
 Wolsiefer, K., Westfall, J., & Judd, C. M. (2016). Modeling stimulus variation in three common implicit attitude tasks. Behavior Research Methods Google Scholar
 Yang, E., Brascamp, J., Kang, M.S., & Blake, R. (2014). On the use of continuous flash suppression for the study of visual processing outside of awareness. Frontiers in Psychology, 5, 724. doi: 10.3389/fpsyg.2014.00724 PubMedPubMedCentralGoogle Scholar
 YuvalGreenberg, S., & Heeger, D. J. (2013). Continuous flash suppression modulates cortical activity in early visual cortex. The Journal of Neuroscience, 33(23), 9635–9643. doi: 10.1523/JNEUROSCI.461212.2013 CrossRefPubMedPubMedCentralGoogle Scholar