Abstract
When experimental philosophers carry out studies on thought experiments, some participants are excluded based on certain exclusion criteria, mirroring standard social science vignette methodology. This involves excluding people that do not pay attention or who miscomprehend the scenario presented in thought experiments. However, experimental philosophy studies sometimes exclude an alarmingly high number of participants. We argue that this threatens the external and internal validity of the conclusions being drawn and we show how a simple visualization of thought experiments can reduce exclusion rates significantly. Furthermore, we argue that focus should not merely be on how many are excluded, but also why they are excluded, and we highlight the role of comprehension questions in this regard. Philosophical thought experiments often rely on the acceptance of certain key premises that may be regarded contestable, and asking comprehension questions involving such key assumptions could be problematic as that may result in some participants being inadvertently excluded from the study, potentially creating a selection bias.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Interestingly Starmans and Friedman (2012) point out that their results are qualitatively the same regardless of whether they include or exclude those subjects having failed comprehension questions. Philosophers cannot however ignore comprehension questions simply on those grounds. For studies including thought experiments to be methodologically sound, comprehension questions must be included, but the observation made by Starmans et al. suggests that we need to look further into the role of comprehension questions and how they are posed. Who do they exclude and why?
An obvious reason for this high number is that the study is carried out in subcontinent India and subjects are reading a case presented in English while selfreporting different levels of language skills.
A simple calculation shows that in a small sample (N < 25) even a 30% failure rate would be statistically indistinguishable from a random answer to a binary question, assuming a 0.05 p value cut-off.
Turri (2013) is arguably doing something to this effect by way of structuring thought experiment presentation so as to help people better understand the structure of the situation presented, but his focus is not to improve comprehension rates.
We should also add that by replicating Colaço et al. (2014), we do not thereby endorse what may be characterised as an instance of ‘negative’ experimental philosophy (branch of experimental philosophy on a quest to eradicate any assumed reliance on intuitive judgments)—even if Colaço et al. so qualifies. Our project is merely to contribute to the continued improvement of the methodology of experimental philosophy and (here, at least) we take no stance on whether experimental philosophy should be positive or negative in nature.
All reported results are unchanged when including respondents with a philosophy degree or excluding respondents report to they have heard about the Gettier problem before.
We chose to rely on several comprehension questions (and an attention check). Another approach to ensure actual comprehension is to require different types of answers, e.g. both negative and positive answers, making it less likely that people pass a test by chance.
73 participants are over 50 years of age and the result is thus not driven by relatively few old participants. We also note that none of the 18 participants aged 60 or above failed the comprehension test.
We thank an anonymous reviewer for this observation.
References
Alexander, C. S., & Becker, H. J. (1978). The use of vignettes in survey research. Public Opinion Quarterly, 42, 93–104.
Buckwalter, W. (2013). Gettier made ESEE. Philosophical Psychology, 27(3), 368–383.
Chisholm, R. (1966). Theory of knowledge. N.J.: Englewood Cliffs.
Clark, C. J., Winegard, B. M., & Baumeister, R. F. (2019). Forget the folk: Moral responsibility preservation motives and other conditions for compatibilism. Frontiers in Psychology, 10, 215.
Cohen, M. (1999). 101 philosophy problems. London: Routledge.
Colaço, D., Buckwalter, W., Stich, S., & Machery, E. (2014). Epistemic intuitions in fake-barn thought experiments. Episteme, 11(02), 199–212.
Douglas, K., Otto, R., Desmarais, S., & Borum, R. (2001). Clinical forensic psychology. In I. B. Weiner, J. A. Schinka, & W. F. Velicer (Eds.), Handbook of psychology, research methods in psychology. New York: Wiley.
Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25–42.
Horvath, J. (2010). How (not) to react to experimental philosophy. Philosophical Psychology, 23, 447–480.
Jackson, F. (1986). What Mary didn’t know. Journal of Philosophy, 83(5), 291–295.
Kim, M., & Yuan, Y. (2014). No cross-cultural differences in the Gettier car case intuition: A replication study. Episteme, 12(3), 355–361.
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr., Alper, S., et al. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443–490. https://doi.org/10.1177/2515245918810225.
Levi, W. H., & Lentz, R. (1982). Effects of text illustrations: A review of research. Educational Communication and Technology Journal, 30(4), 195–232.
Little, R. J. A., & Rubin, D. B. (2014). Statistical analysis with missing data. New York: Wiley.
Machery, E., Stich, S., Rose, D., Chatterjee, A., Karasawa, K., Struchiner, N., et al. (2015). Gettier across cultures. Noûs, 51(3), 645–664.
Maniaci, M., & Rogge, R. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48(1), 61–83.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
Murray, D., & Nahmias, E. (2014). Explaining away incompatibilist intuitions. Philosophy and Phenomenological Research, 88(2), 434–467.
Nahmias, E., Coates, D. J., & Kvaran, T. (2007). Free will, moral responsibility, and mechanism: Experiments on folk intuitions. Midwest Studies in Philosophy, 31(1), 214–242.
Nahmias, E., Morris, S., Nadelhoffer, T., & Turner, J. (2005). Surveying freedom: Folk intuitions about free will and moral responsibility. Philosophical Psychology, 18(5), 561–584.
Nichols, S., & Bruno, M. (2010). Intuitions about personal identity: An empirical study. Philosophical Psychology, 23, 293–312.
Nichols, S., & Knobe, J. (2007). Moral responsibility and determinism: The cognitive science of folk intuitions. Nous, 41, 663–685.
Oppenheimer, D., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.
Philips, J. & Cushman, F. (2017) Replication of Nahmias, Coates, & Kavarn (2007) Free will, moral responsibility, and mechanism: Experiments on folk intuitions. https://osf.io/n4573/.
Polonioli, A., Vega-Mendoza, M., Blankinship, B., & Carmel, D. (2018). Reporting in experimental philosophy: Current standards and recommendations for future practice. Review of Philosophy and Psychology. https://doi.org/10.1007/s13164-018-0414-3.
Searle, J. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3(3), 417–457.
Seyedsayamdost, H. (2015). On normativity and epistemic intuitions: Failure of replication. Episteme, 12(1), 95–116.
Sorensen, R. (1992). Thought experiments. New York: Oxford University Press.
Sosa, E. (2009). A defense of the use of intuitions in philosophy. In M. Bishop & D. Murphy (Eds.), Stich and his critics (pp. 101–112). Oxford: Blackwell.
Starmans, C., & Friedman, O. (2012). The folk conception of knowledge. Cognition, 124, 272–283.
Thomas, K., & Clifford, S. (2017). Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior, 77, 184–197.
Turri, J. (2013). A conspicuous art: Putting Gettier to the test. Philosophers’ Imprint, 13, 1–16.
van’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.
Weinberg, J. M., Nichols, S., & Stich, S. (2001). Normativity and epistemic intuitions. Philosophical topics, 49(1&2), 429–460.
Williams, B. (1973). Problems of the self. Cambridge: Cambridge University Press.
Williamson, T. (2011). Philosophical expertise and the burden of proof. Metaphilosophy, 42, 215–229.
Acknowledgements
The authors wish to thank Vilius Dranseika, Oana Vuculescu and two anonymous reviewers for valuable comments and suggestions. Research for this publication was supported by grants from the Carlsberg Foundation (CF15-0943) and the Independent Research Fund Denmark (DFF-4180-00071).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Bergenholtz, C., Busch, J. & Praëm, S.K. Exclusion Criteria in Experimental Philosophy. Erkenn 86, 1531–1545 (2021). https://doi.org/10.1007/s10670-019-00168-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10670-019-00168-5