Abstract
To study consciousness, scientists need to determine when participants are conscious and when they are not. They do so with consciousness detection procedures. A recurring skeptical argument against those procedures is that they cannot be calibrated: there is no way to make sure that detection outcomes are accurate. In this article, I address two main skeptical arguments purporting to show that consciousness scientists cannot calibrate detection procedures. I conclude that there is nothing wrong with calibration in consciousness science.
This is a preview of subscription content, log in via an institution to check access.
Source: Fleming and Dolan (2010). Post-decision wagering procedure. The subject performs the Type 1 task (here, identifying whether a series of letters is a word or not). She then has to bet either 5 lb or 50 pennies on the correctness of her decision. Depending on the accuracy of her decision, she either earns or loses 5 lb or 50 pennies
Similar content being viewed by others
Notes
A subject is conscious of a stimulus if there’s “something it’s like” for her to perceive it (Nagel 1974). She perceives a stimulus unconsciously if she mentally represents that there is a stimulus, but doing so does not feel any different from not representing that there’s a stimulus.
Consciousness scientists and philosophers alike often talk about “measures” of consciousness (Irvine 2013; Sandberg et al. 2010; Seth et al. 2008; Timmermans and Cleeremans 2015; Spener, forthcoming). I prefer to talk about the “detection” of consciousness. Measurement is supposed to apply to quantitative properties (Joint Committee for Guides in Metrology 2012; Michell 1999). Using the term “measurement” assumes that consciousness is a quantitative property, and that the outcomes of procedures used by consciousness scientists attribute degrees of consciousness of stimuli to the subjects. In most experiments, consciousness scientists simply try to assess whether or not subjects are conscious of stimuli, that is, they attempt to detect consciousness.
Skepticism about the calibration of consciousness detection procedures comes in degrees. For instance, while Spener (2015) accepts some of the skeptics’ arguments, she argues that we can identify conditions in which introspective judgments tend to be reliable (see also Bayne and Spener 2010). Irvine (2012a, 2012b, 2019) is a skeptic in the most straightforward sense, and I mainly focus on her arguments here.
We should distinguish between metacognition and introspection. As I use the terms, metacognitive representations are about cognitive or perceptual states. Representations that result from introspection, on the other hand, are about the subject herself. In a slogan: metacognition is cognition about cognition, introspection is cognition about the cognizer. While introspection is by definition a metacognitive process, not all metacognitive processes are introspective. For instance, a poker player might monitor her own states of uncertainty, and make use of those states, without having to consciously self-attribute those states (Carruthers and Ritchie 2012). In this article, I assume that Type-2 tasks are generally introspective in nature because they require subjects to self-attribute perceptual or confidence states.
Philosophers and scientists often draw the distinction in terms of “objective” and “subjective” procedures. The latter are probably called “subjective” because they rely on “subjective” reports. But those reports consist of publicly observable behaviors like pressing buttons, as observed by Piccinini (2009). It is unclear what is “subjective” in pressing buttons, or even in verbal reports. In any case, “Type-1-based” and “Type-2-based” procedures is a more neutral way of putting the distinction.
I will not discuss the statistical tools that scientists use to find those correlations (for reviews, see Fleming and Lau, 2014; Maniscalco and Lau 2014). The gold-standard is currently to compute an index called Meta-d’, which is then used to quantify how much of the information used during the Type-1 task was available for the subject to perform the Type-2 task (Maniscalco & Lau, 2012).
This correlation could be driven by a third factor influencing both Type-1 and Type-2 decisions in the same way, instead of being explained by the influence of visual information used for the Type-1 task over the Type-2 decision. At this stage, this ‘third factor’ hypothesis cannot be ruled out, although it is unclear what that third factor would be. The hypothesis that participants take Type-2 decisions based on what they consciously see during the Type-1 task seems to be a simple and straightforward explanation of the correlation between Type-1 and Type-2 performance.
Those two criteria are complementary. There’s a big difference between not seeing something and seeing something unconsciously. I don’t see what’s behind my head consciously, but I don’t see it unconsciously either. The zero-correlation criterion allows experimenters to decide whether or not a large proportion of stimuli were consciously perceived during the task. The guessing criterion allows them to decide whether the stimuli that were not perceived consciously were perceived unconsciously or not perceived at all.
Attempts to calibrate procedures relying on ‘subjective’ or ‘introspective’ judgments are not new. For instance, Titchener’s Experimental Psychology: A Manual of Laboratory Practice (1905) can be considered as an ‘introspective training manual’ (Schwitzgebel 2011; Chapter 5), allowing psychologists to improve introspective ‘observations’ (See Kroker, 2003). With respect to calibration, an important difference with contemporary consciousness science is that calibration in Titchener’s practice was focused on the introspector, not on the whole detection process, including experimental settings, as well as methods of statistical analyses. As (Lyons 1986) puts it, calibration was more concerned with ‘the attitude of the introspector’ than ‘the laboratory conditions themselves’ (p. 19).
This distinction is inspired from Tal’s distinction between black-box and white-box calibration of measuring instruments (Tal 2017).
I assume that concordance-calibration can provide a good indication of accuracy, if not a sufficient condition for establishing accuracy. This point has been debated (Basso 2017). Most researchers recognize the value of comparing various measurement procedures, while disagreeing on the reason why doing so is valuable. Concordance-calibration could be useful because it allows one to compare procedures that do not share the same sources of error, namely, independent procedures (Kuorikoski et al. 2012). An alternative source of value for concordance-calibration is to be found, not in the independence of measurement procedures, but in their complementarity: different procedures have different strengths and weaknesses. As Basso (2017) puts it: “Since each procedure can fail to realize the definition (and hence to measure the quantity of interest) in different ways, the convergence of their results can be taken as a sign that these sources of uncertainty do not lead the results completely astray and, therefore, that the procedures measure the quantity as defined with sufficient reliability.” (p. 8).
If you’re not convinced, try to do a discrimination task when stimuli are presented behind your head, and then do the same when stimuli are presented in front of you. I bet that d’ will be higher in the latter case, and that, in that case, you will be conscious of the stimuli more often. If not, well done, you’ve just found a proof that you’re gifted with extra-sensory perception, and that most of what we believe about perception is false. This puts you in a good position to become a superhero and win a Nobel Prize!
Take, for instance, Regnault and De Luc’s long and tedious work of concordance-calibration of thermometers (Chang 2004, p. 76–84). Here’s just a small, non-exhaustive list of the potential factors that could influence measurement outcomes in this case: Were air thermometers more accurate than mercury thermometers, or alcohol thermometers? How did the type of alcohol influence the outcome: were alcohol thermometers with old Languedoc wine more reliable than thermometers with Brandy, or mixtures of alcohol? How did the type of glass of the thermometer influence the outcomes: was Swedish glass better than green glass, or ‘Choisy-le-Roi’ crystal? Would air thermometers with different air densities provide different outcomes? What about thermometers with different gases, such as hydrogen, carbon dioxide or sulfuric acid gas? These questions were solved in large part through a long work of concordance-calibration involving systematic comparisons between different thermometers (Chang 2004).
In practice, there are two main ways in which consciousness scientists attempt to mitigate the effects of response biases. The first is to use statistical tools inspired from Signal Detection Theory such as meta-d’ (Maniscalco and Lau 2012), or the area under the Type-2 ROC curve (AUROC2) (Fleming and Lau 2014). The second approach is to design “bias-free” tasks, such as two-interval forced choice tasks (de Gardelle and Mamassian 2014; Knotts et al. 2018; Peters and Lau 2015; Peters et al. 2017).
For instance, in a study by Stolyarova et al. (2019) rats were trained to discriminate the orientation of Gabor patches. After their decisions, they could either directly initiate a new trial, or wait for a sugar pellet reward, provided only if their decision was correct. The basic idea is that if rats are unsure about their decisions they should directly initiate a new trial instead of wasting time waiting for a sugar pellet that they probably won’t get.
Recent literature in social and consumer psychology casts doubt on the existence of loss aversion as a systematic bias (for a review, see Gal and Rucker 2017). In general, the effect of loss aversion might be smaller than usually assumed (Walasek et al. 2018). Instead, it could be that the low bet is perceived by subjects as maintaining the status quo, and subjects have a status quo bias instead of loss aversion (Gal 2006).
References
Basso, A. (2017). The appeal to robustness in measurement practice. Studies in History and Philosophy of Science Part A, 65–66, 57–66.
Bayne, T., & Spener, M. (2010). Introspective Humility. Philosophical Issues, 20, 1–22.
Carruthers, P., & Ritchie, J. B. (2012). The emergence of metacognition: Affect and uncertainty in animals. In M. J. Beran, J. L. Brandl, J. Perner, & J. Proust (Eds.), Foundations of metacognition (pp. 76–93). Oxford: Oxford University Press.
Chang, H. (2004). Inventing Temperature: Measurement and scientific progress. New York: Oxford University Press.
Cheesman, J., & Merikle, P. M. (1986). Distinguishing conscious from unconscious perceptual processes. Canadian Journal of Psychology, 40(4), 343–367.
Chirimuuta, M. (2014). Psychophysical methods and the evasion of introspection. Philosophy of Science, 81(5), 914–926.
Clifford, C. W. G., Arabzadeh, E., & Harris, J. A. (2008). Getting technical about awareness. Trends in Cognitive Sciences, 12(2), 54–58.
Cowey, A. (2010). The blindsight saga. Experimental Brain Research, 200(1), 3–24.
de Gardelle, V., & Mamassian, P. (2014). Does confidence use a common currency across two visual tasks? Psychological Science, 25, 1286–1288.
Dienes, Z., Altmann, G. T. M., Kwan, L., & Goode, A. (1995). Unconscious knowledge of artificial grammars is applied strategically. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(5), 1322–1338.
Dienes, Z., & Perner, J. (2004). Assumptions of a subjective measure of consciousness: Three mappings. In R. Gennaro (Ed.), Higher order theories of consciousness (pp. 173–199). Amsterdam: John Benjamins Publishers.
Dienes, Z., & Seth, A. (2010). Gambling on the unconscious: A comparison of wagering and confidence ratings as measures of awareness in an artificial grammar task. Consciousness and Cognition, 19(2), 674–681. https://doi.org/10.1016/j.concog.2009.09.009.
Elgin, C. (2017). True Enough. Cambridge, MA: MIT Press.
Feest, U. (2014). Phenomenal experiences, first-person methods, and the artificiality of experimental data. Philosophy of Science, 81(5), 927–939.
Feest, U. (2012). Introspection as a method and introspection as a feature of consciousness. Inquiry, 55(1), 1–16.
Fleming, S. M., & Daw, N. D. (2017). Self-evaluation of decision-making: A general Bayesian framework for metacognitive computation. Psychological Review, 124(1), 91–114.
Fleming, S. M., & Dolan, R. J. (2010). Effects of loss aversion on post-decision wagering: Implications for measures of awareness. Consciousness and Cognition, 19(1), 352–363.
Fleming, S. M., & Lau, H. C. (2014). How to measure metacognition. Frontiers in Human Neuroscience, 8, 443.
Gal, D. (2006). A psychological law of inertia and the illusion of loss aversion. Judgment and Decision Making, 1(1), 23–32.
Gal, D., & Rucker, D. D. (2017). The Loss of Loss Aversion: Will It Loom Larger Than Its Gain? Journal of Consumer Psychology, 28(3), 497–516.
Galvin, S. J., Podd, J. V., Drga, V., & Whitmore, J. (2003). Type 2 tasks in the theory of signal detectability : Discrimination between correct and incorrect decisions. Psychonomic Bulletin & Review, 10(4), 843–876.
Goldman, A. (2002). Pathways to Knowledge: Private and Public. Oxford: Oxford University Press.
Goldman, A. I. (2004). Epistemology and the evidential status of introspective reports. Journal of Consciousness Studies, 11(7–8), 1–16.
Green, D., & Swets, S. (1966). Signal Detection Theory and Psychophysics. New York: Wiley.
Hacking, I. (1981). Do we see through a microscope? Pacific Philosophical Quarterly, 62(4), 305–322.
Holender, D. (1986). Semantic activation without conscious identification in dichotic-listening, parafoveal vision, and visual masking: A survey and appraisal. Behavioral and Brain Sciences, 9, 1–23.
Irvine, E. (2012a). Old problems with new measures in the science of consciousness. British Journal for the Philosophy of Science, 63(3), 627–648.
Irvine, E. (2012b). Consciousness as a Scientific Concept . Dordrecht: Springer.
Irvine, E. (2013). Measures of consciousness. Philosophy Compass 8(3), 285–297.
Irvine, E. (2019). Developing Dark Pessimism Towards the Justificatory Role of Introspective Reports. Erkenntnis. https://doi.org/10.1007/s10670-019-00156-9.
Joint Committee for Guides in Metrology (JCGM). (2012). International Vocabulary of Metrology (VIM), 3rd edition. http://www.bipm.org/en/publications/guides/vim.html. Accessed 8 April 2021.
Kahneman, D. (2011). Thinking, fast and slow. London: Penguin Books.
Knotts, J. D., Lau, H., & Peters, M. A. K. (2018). Continuous flash suppression and monocular pattern masking impact subjective awareness similarly. Attention, Perception, & Psychophysics, 80(8), 1974–1987.
Kroker, K. (2003). The progress of introspection in America, 1896–1938. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 34(1), 77–108.
Kuorikoski, J., Lehtinen, A., & Marchionni, C. (2012). Robustness analysis disclaimer: Please read the manual before use! Biology and Philosophy, 27, 891–902.
Lau, H. (2008). Are we studying consciousness yet? In M. Davis & L. Weiskrantz (Eds.), Frontiers of Consciousness (pp. 245–269). Oxford: Oxford University Press.
LeDoux, J. (2019). The Deep History of Ourselves: The Four-Billion-Year Story of How We Got Conscious Brains. New York: Viking.
Li, Q., Hill, Z., & He, B. J. (2014). Spatiotemporal dissociation of brain activity underlying subjective awareness, objective performance and confidence. The Journal of neuroscience, 34(12), 4382–4395.
Lyons, W. (1986). The disappearance of introspection. Cambridge, MA: MIT Press.
Macmillan, N. A., & Creelman, C. D. (2005). Detection Theory: A User’s Guide. Taylor & Francis.
Maniscalco, B., & Lau, H. (2012). A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Consciousness and Cognition, 21(1), 422–430.
Maniscalco, B., & Lau, H. (2014). Signal Detection Theory Analysis of Type 1 and Type 2 Data : Meta- d’, Response- Specific Meta-d’, and the Unequal Variance SDT Model. In S. M. Fleming & C. D. Frith (Eds.), The Cognitive Neuroscience of Metacognition (pp. 25–66). New York: Springer.
Mealor, A., & Dienes, Z. (2012). No-loss gambling shows the speed of the unconscious. Consciousness and Cognition, 21(1), 228–237.
Michel, M. (2019). The Mismeasure of Consciousness: A problem of coordination for the Perceptual Awareness Scale. Philosophy of Science, 86(5), 1239–1249.
Michel, M. (2020). Consciousness Science Underdetermined: A brief history of endless debates. Ergo, 6(28).
Michell, J. (1999). Measurement in Psychology: A critical history of a methodological concept. Cambridge: Cambridge University Press.
Morales, J., Chiang, J., & Lau, H. C. (2015). Controlling for performance capacity confounds in neuroimaging studies of conscious awareness. Neuroscience of Consciousness, 2015(1), 1–11. https://doi.org/10.1093/nc/niv008.
Nagel, T. (1974). What is it like to be a bat? The Philosophical Review, 83(4), 435–450.
Norman, E., & Price, M. C. (2015). Measuring consciousness with confidence ratings. In M. Overgaard (Ed.), Behavioral Method in Consciousness Research. Oxford University Press, Oxford.
Overgaard, M., Rote, J., Mouridsen, K., & Ramsoy, T. Z. (2006). Is conscious perception gradual or dichotomous? A comparison of report methodologies during a visual task. Consciousness and Cognition, 15(4), 700–708.
Persaud, N., McLeod, P., & Cowey, A. (2007). Post-decision wagering objectively measures awareness. Nature Neuroscience, 10(2), 257–261.
Peters, M. A. K., & Lau, H. (2015). Human observers have optimal introspective access to perceptual processes even for visually masked stimuli. ELife, 4, 1–30.
Peters, M. A. K., Fesi, J., Amendi, N., Knotts, J. D., Lau, H., & Ro, T. (2017). Transcranial magnetic stimulation to visual cortex induces suboptimal introspection. Cortex, 97, 119–132.
Piccinini, G. (2009). First-person data, publicity, and self-measurement. Philosophers’ Imprint, 9(9), 1–16.
Rahnev, D., & Denison, R. N. (2018). Suboptimality in Perceptual Decision Making. Behavioral and Brain Sciences, 41, 1–107.
Rahnev, D., & Fleming, S. M. (2019). How experimental procedures influence estimates of metacognitive ability. Neuroscience of Consciousness, 5, 1–42.
Rausch, M., & Zehetleitner, M. (2014). A comparison between a visual analogue scale and a four point scale as measures of conscious experience of motion. Consciousness and Cognition, 28, 126–140.
Rausch, M., Müller, H. J., & Zehetleitner, M. (2015). Metacognitive sensitivity of subjective reports of decisional confidence and visual experience. Consciousness and Cognition, 35, 192–205.
Rausch, M., Hellmann, S., & Zehetleitner, M. (2018). Confidence in masked orientation judgments is informed by both evidence and visibility. Attention, perception & psychophysics, 80(1), 134–154.
Rounis, E., Maniscalco, B., Rothwell, J. C., Passingham, R. E., & Lau, H. (2010). Theta-burst transcranial magnetic stimulation to the prefrontal cortex impairs metacognitive visual awareness. Cognitive Neuroscience, 1(3), 165–175.
Rosenthal, D. (2018). Consciousness and confidence. Neuropsychologia, 128, 255–265.
Sandberg, K., & Overgaard, M. (2015). Using the perceptual awareness scale (PAS). In M. Overgaard (Ed.), Behavioral Methods in Consciousness Research. Oxford: Oxford University Press.
Sandberg, K., Timmermans, B., Overgaard, M., & Cleeremans, A. (2010). Measuring consciousness: Is one measure better than the other? Consciousness and Cognition, 19(4), 1069–1078.
Seth, A. K., Dienes, Z., Cleeremans, A., Overgaard, M., & Pessoa, L. (2008). Measuring consciousness: relating behavioural and neurophysiological approaches. Trends in Cognitive Sciences, 12(8), 314–321.
Schmidt, T. (2015). Invisible stimuli, implicit thresholds: Why invisibility judgments cannot be interpreted in isolation. Advances in Cognitive Psychology, 11(2), 31–41. https://doi.org/10.5709/acp-0169-3.
Schwitzgebel, E. (2011). Perplexities of consciousness. Cambridge, MA: MIT Press.
Schwitzgebel, E. (2012). Introspection, what? In D. Smithies & D. Stoljar (Eds.), Introspection and consciousness (pp. 29–48). Oxford: Oxford University Press.
Snodgrass, M., Bernat, E., & Shevrin, H. (2004). Unconscious perception: A model-based approach to method and evidence. Perception and Psychophysics, 66(5), 846–867.
Spener, M. (2013). Moderate scepticism about introspection. Philosophical Studies, 165(3), 1187.
Spener, M. (2015). Calibrating introspection. Philosophical. Issues, 25(1), 300–321.
Spener, M. (forthcoming). Consciousness, introspection, and subjective measures. In U. Kriegel (Ed.), Handbook of the Philosophy of Consciousness. Oxford: Oxford University Press.
Stolyarova, A., Rakhshan, M., Hart, E. E., O’Dell, T. J., Peters, M. A. K., Lau, H., Soltani, A., & Izquierdo, A. (2019). Contributions of anterior cingulate cortex and basolateral amygdala to decision confidence and learning under uncertainty. Nature Communications, 10(1), 1–14. https://doi.org/10.1038/s41467-019-12725-1.
Szczepanowski, R., Traczyk, J., Wierzchoń, M., & Cleeremans, A. (2013). The perception of visual emotion: Comparing different measures of awareness. Consciousness and Cognition, 22(1), 212–220.
Tal, E. (2014). Making time: A study in the epistemology of measurement. British Journal for the Philosophy of Science, 67(1), 297–335.
Tal, E. (2017). Calibration: Modelling the measurement process. Studies in History and Philosophy of Science Part A, 65–66, 33–45.
Timmermans, B., & Cleeremans, A. (2015). How can we measure awareness? an overview of current methods. In M. Overgaard (Ed.), Behavioral Methods in Consciousness Research. Oxford: Oxford University Press.
Titchener, E. B. (1905). Experimental psychology: A manual of laboratory practice. London: The Macmillan Company.
Walasek, L., Mullett, T. L., & Stewart, N. (2018). A meta-analysis of loss aversion in risky contexts. Available at SSRN: https://ssrn.com/abstract=3189088.
Weiskrantz, L. (1998). Consciousness and Commentaries. International Journal of Psychology, 33, 227–233.
Weiskrantz, L. (2009). Blindsight: A case study spanning 35 years and new developments. Oxford: Oxford University Press.
Wierzchoń, M., Paulewicz, B., Asanowicz, D., Timmermans, B., & Cleeremans, A. (2014). Different subjective awareness measures demonstrate the influence of visual identification on perceptual awareness ratings. Consciousness and Cognition, 27(1), 109–120.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Michel, M. Calibration in Consciousness Science. Erkenn 88, 829–850 (2023). https://doi.org/10.1007/s10670-021-00383-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10670-021-00383-z